this pink blush right here. See this pink blush right here? Look at that pink blush. We're going to be putting that on our under eyes and we're going to test out whether or not this viral TikTok tutorial really works on mature under eyes or any under eyes for that matter. We're going to see whether or not this is the latest and greatest thing or whether or not it just won't work for us. Let's do this video right now. Hi friends. So I actually did not originally hear about this on TikTok. I actually heard about this from Allure magazine. I subscribed to their emails and read their emails all the time and one of their editors was talking about doing this and how she would never look back and I'm like okay I haven't even heard about this. So then I had to go research it find out what's going on. What it is is everybody is saying they are putting a pink cream blush. In this case it's liquid. You could use I'm sure any cream blush that you had that was a little bit more on the pink side. I would choose something that was more kind of a neutral pink. That's what I've kind of chosen today. This is happy from the Rare Beauty line. That's what I chose to use today. I'm using it on both sides but I'm actually trying a drug store the NYX Serum Concealer and the COSIS my beloved COSIS on the other side to try and test out and see if there's any difference on there. We're going to test this completely out. We're going to see how it might work better. Might not. I'm definitely going to let you know what I think of it. I also want to let you know that I do have a numbering system where I will number each product that I'm talking about as I'm talking about them one through however many products there are. You don't have to worry about the name of the product just drop jot down the number that is next to the product and then you can open up the description box and once you do that you will see everything numbered and you can just go to the number and then click on the link if you'd like to shop that way and please remember that those are affiliate links and I do make a small commission when you do that that helps to support my channel and I do appreciate it. So I'm going to have my eyes and my eyebrows done but the rest of my makeup won't be done but without any further ado let's get into seeing whether or not pink blush works on our under eyes with our concealer. Let's figure this out right now together. So we're going to see how we do with a high end on one side and then a drug store on another but I did choose to use the same blush liquid blush and that's from Rar Beauty and this one is called Happy. It's quite pink.  If I was to tap that on my eye it would get way too much too fast so I'm going to put it on my hand first and get my mirror out that might help some too. So there's the color of it very bright and then I'm just going to put a little bit on my hand right here and I'm going to go and stick. Oh my goodness okay that was a lot. I would say that since this is such a pigmented blush that you need way less than you think you do. So this is the first time me doing this with you guys. I'm hoping that this works out. I just put some in there. It feels really strange to have that there. It's very emollient though. I love that about the liquid blushes. Very emollient. So I'm going to just leave that there for a second. Then I'm going to go and I'm going to put on my this is the NYX Bare With Me Serum Concealer. I don't know what color this is. I think this is beige. Anyway I'm going to put some of that on my hand right here a little bit and I don't know. I don't use this very much so I don't know how much I need. I'm going to put some right here and I'm just going to lightly kind of just lightly go over this and just leave it to set up because I know from my past experience that if I go in and I just immediately start blending everything's going to look weird anyway so I don't want to do that. And then I'm using my Beloved Cosas over on this side over here and I'm going to just lightly go across that blush and we're going to let it set up. This is my trick is that I always try to do lightly and then let it set. I feel like I have a little bit more Cosas on this side than I do of the NYX which is really not fair. We need to get it as fair as possible so I'm just going to put in just a little bit more NYX. I know that seems like a lot but when I get ready to blend it out it will be quite a lot less. I'm going to just do my foundation while we're talking about this a little bit. I'm using the Dior Foundation. This one is the Skin Glow the new one and I'm actually putting some of the matte in with it. I love both of these on their own but the other day I was thinking you know I bet they would be really good together so I just decided to kind of mix them together and see what I get. Plus my one color is pretty dark and my one color is pretty light so hopefully we'll get a good color. It seems a little bit light still but we'll go for it. I'm just going to put on my foundation. The whole premise to actually using the color pink is that it cancels out the blue or it brightens whenever you get a pink concealer. Lots of times that pink will say brightener on it. I know that L'Oreal Age Rewind does that so definitely something to think about. I feel like the concealer is quite set up enough yet.  This is Laura Geller's Baked Bronzer in Fair, one of my very favorites. This isn't quite a contour but I'm still going to use it in my hairline to warm everything up and kind of recede that high, high, high forehead. Also use it to contour my cheeks and then go into my jowl line. Alright so we're going to see if this will work. I'm using a very small dampened beauty sponge. This is from Eco Tools. It's their new biodegradable one and I love this little teeny one. It works so good. So let's start on this side. I always start on the outer part and I try to look up a little bit and I just bounce and blur and do as much blending as I possibly can and hopefully we will see some magic here. I don't know about this technique. I have never tried it before. You guys are the first to see me do it so there you have it. Okay so it's really interesting coverage. I feel like I knew that the NYX wasn't quite as full coverage as the COSIS but I do feel like it did a good job. However I still see this hollow right here on my eyes and the pink would be the color corrector in lieu of any sort of orange. So let me just kind of blend this side out with the other side of the sponge and see what happens with a little bit fuller coverage on that. Okay I have to say that I'm not super sold on it yet but it could just be that I put most of the pink in here and I didn't put enough of the concealer so I'm just going to put a teeny tiny bit of each concealer on each side and I'm just sticking into those troughs where it you know it gets really dark for me. I'm not putting it out anywhere else because I don't want extra because this is all covered up pretty nicely. It looks actually really pretty good. So let's move on just a little bit. Let's do a little bit of highlighter. As most of you already know I do take my highlighter underneath my blush first. So I'm doing the Maybelline Master Chrome today and I'm just going to put that. This is such a beaming highlighter but it's going to be so pretty underneath blush. I really love it. Don't forget to put a tiny bit of that highlighter right there on that cupid's bow.  Okay so it is it has set up a little bit. Let's just go across very lightly see what we get. Okay I can see the pink and I kind of like that. I kind of think that it looks good. The pink is actually making this area look a little bit brighter than normal I feel like anyway. So what I'm going to do is I'm going to just do a light dusting of powder. This is what I had on this powder puff before I started. So I'm just going to very lightly touch underneath there to set it. Otherwise it's going to crease on me. Since we already used that blush let's just go ahead and use it again. I'm going to just put it across my hand a little bit there. Well a lot I guess there and I'm just going to dip into it with my stipple brush which works so good for any liquid or cream products because you can start out very lightly which doesn't look so light but you can blend this like crazy and it works so well with this brush. And this blush is really like I talked about it's very pigmented but I think that using the blush in this area and using it underneath this area kind of ties the look together. Now painted by Spencer talks about going up way up almost up underneath the eyes with some of the blush. I think that's a great idea because not only are you kind of going where that color has already been but you're also tying your look together a little bit more.  Got a little bit too much up there on the top so we'll just pounce that out with the Beauty Blender just to kind of diffuse it a little bit and I really like that. I really think that it's really pretty. I am going to do my liner, my bottom lashes and my lips and I'll be right back. There is the final look on the under eyes. I think this is a great idea especially if you have olive or warm undertones. Sometimes that peach because you already have that warm undertone isn't going to work so the pink might be really good for you. I'm going to keep trying this for several times but I think it looks really good under there and I think that it's something that you might enjoy too especially if you've struggled with color correctors in the past. Now I would say put that a little bit of pink on there, dab it out a little bit. That's probably what I would do Nat instead of what I did where I just put it all together. Dab that pink out a little bit. Let it set for just a couple of seconds and then put your concealer on top of it. I think that that probably would be ideal because that's kind of what I ended up doing in the end where I put a little bit more on the troughs that I have right there. So my friends I hope that you did enjoy seeing this tutorial and whether or not it's game-changing I guess we will all find out. Thank you so much for tuning in today and please give this video a thumbs up. As we say goodbye today I will throw up another video right here that you may not have seen before. It's one that I've done in the past that's all about concealers and you might not have gotten to see it and it helps you to avoid some mistakes with concealers as well that might be getting your concealer to creasing a little bit. Everybody please have a great day and I will catch you in my very next video. Love you much. Goodbye my friends.


Hi there! This is Pastor Vlad and before we go into this week's content I would like to invite you to become a part of what God is doing at Hungry Generation today. This year we've seen a great blessing of the Lord and anointing of God, God's healings and salvation and deliverances. And in 2019 I know that God wants to take us further. And many of you who are watching this video right now and watching this message, you've been receiving from Hungry Gen, you've been growing, your world has been changing and we give to God all the glory. But we would like you in 2019 to become a part of what God is doing. And you can do that by sowing your best gift into this ministry or maybe doing something monthly like a partnership, a reoccurring gift. This will help us to go further into 2019 and bring more of what we're bringing to you to many, many more people. Me and my wife, we do that every single year. Our church does that every single year. Well once a year we give a special offering to God and then we also become partners of this ministry by our monthly contributions. And I give you that opportunity today to become our partner and to become somebody who contributes to what God is doing today. Below is a link where you can make that happen. So why don't you ask God, what would God have you give this year to this ministry to help us go further in God. And now let's go into this message. I started this thought yesterday. I just want to take the history and the sphere and share with you the practical steps of how we can live or gain the freedom of Jesus Christ. This was not my initial word that I had, but when I came yesterday I saw that this weekend was even more dedicated to the light, to get closer to God, to freedom, maybe to gain freedom in some different spheres of our lives. I just want to move in this topic as well as you appointed it on this Saturday and on this Sunday. When I bought, my wife and I bought one property, a duplex, when we got married a year after that we bought a duplex with such an old one to settle there and to give the other side a rent to help us with the finances. The owner who sold this house, he didn't take care of it. There were bulls there, on his knees. I remember I took my duplex and was afraid to go there, because he didn't know what was there and what was going on there and what was living there. He didn't take care of this house, they grew up there, there was no lawn, there was no green grass, it was all bad, all bad. And I knew that to blame this owner who sold it to me, it would be useless, the situation will not improve from this. But it took a couple of years to start digging out all these bulls, to sow new seeds, to sow this grass and to bring this property into order. And when my wife and I sold it a couple of years ago, we already handed over a new house, a remodeled house. There was already a lawn, there were already sprinkles, there was already green grass, it just looked beautiful. And you know, the story of the show begins from the fact that there was one ancestor, who, God told him to defeat the Amaleketians. And instead of listening to God, he didn't listen to him until the end. And he didn't destroy all the Amaleketians. And then, many years later, Amalek's descendants, they appeared in Babylon. And what is interesting is that Mordechai, the future of this knee, Kisa, and Kisa was the father of the king Saul, meets with a problem in Babylon, who left him his ancestors. Each of us here in this place, we receive something from our parents, we receive our surname, color of our hair, color of our skin, some characteristics. Many of us, we receive inheritance, maybe a house, someone will get a car, someone will get some financial blessings, someone will get some difficulties. But very often we also receive some things in our character. Here are parents, for example, they are very tough, and you see, children are the same. There are parents who do not like to bow to God. As soon as there is a blessing, they immediately close everything inside. And children also do not bow to God. They do not like churches, they always criticize the pastor. And children do not want to go to church. Very often the problems with which we are fighting in our life are problems that were transmitted to us by the previous generation. This does not mean that we should blame the previous generation or blame. This simply means that in the sphere of Mordechai, we must take responsibility for the battles that we face now and win them. And the next generation should not pass on the gifts, but pass on the holidays that were made in the sphere. This is a holiday, what was it called? Purim. Pass on the holiday to the next generation, but not the problem. Every generation will have their own problems, every generation will have their own experiences, every generation will have their own battles. But thank God, for the generation that is fighting, setting off some values, and not just telling us to do it, but transmitting it, and then it is easier for children to live a happy life, to live together in marriage, to serve God. You know, the Apostle Paul says to Timothy, he says, faith that was with your mother, but it began with your grandmother, he says, and I see she lives, she will be with you.  We know that salvation, no one is saved because your father is saved. But it is easier, many of us here today believe, because this faith was transmitted to us. It was transmitted to us by parents. Very often there is a battle with which we meet in our lives, which they started from us, they just go to hell. The Apostle Peter says so. In the first verse, verse 18, he says, knowing that in the eternal silver and gold, you were collected from the harvest of your life, we have inherited from our fathers. There are things that we inherited. For example, for me, divorce is incomprehensible at all. I can't understand how it can be, how people can live together and then divorce. Only for one reason. Because my father and my mother, my grandfather and my grandmother, they transmitted it to us. I have never seen where mother or father quarreled and threatened each other that we would leave each other. My father learned the secret of success in marriage early. You should listen to your mother, and everything will be fine. I saw it when you were young. I will immediately remember myself. You will listen to your wife, everything will always be fine. You will be tied up. You will sleep well. To take care of yourself physically, not to suffer many diseases. Not all diseases, but there are diseases that come from the fact that people do not take care of themselves physically. It was transmitted to me. My father always sat behind me. My mother always sat behind her weight and her health, how they ate and how my mother still wears some clothes, goes there to take these steps and everything, and she is doing everything to be healthy. Today all my brothers and sisters, they also take care of their health. Why? It was somehow investigated. We have people in our team who, on the contrary, their parents, they did not win this battle. They did not follow. They looked at the fact that, well, the main thing was that everything was right in the heart, and they did not even follow their weight and health, and children suffer from it today. Children are much more difficult, it is more difficult to win today. The statistics themselves say that if your father was an alcoholic, you have a chance to take alcohol or suffer with alcohol ten times more than the one that your father did not have an alcoholic. Even if you go to the doctor, and the first thing he will ask you on these surveys, it is not just that what is your medical history, he will immediately ask if it is your parents or your ancestors. You can say to yourself, well, doctor, I did not come here to tell about my parents. I did not come to fly my grandmother and grandfather, but the doctor knows. This is already proven by science that there are things that pass through our blood, pass through our DNA, and that pass through, as the apostle Peter said here, and what was fought by Mordechai and the sphere in Babylon. It was what was transmitted. Unfortunately, Saul did not pass the victory, he passed the problem. There are things in our life that we suffer today that were just transmitted. And I want to thank each of you that if Mordechai and the sphere could win what was transmitted, it means that each of us, through the Holy Spirit, through the blood of Jesus, through the death, through prayer, through the path, through the salvation of God's face, we can also win it and pass it on to the next generation. A holiday, not a problem. For me, this is a very dry question, because in our church, half, even more than half of our church are new-born people who come out of broken families. They do not even know what it is that mother and father live together for more than two years. One couple who goes to Mordechai, they are already married, eight years old, have three children. They do not know any of the relatives who are still married. All their family looks at them like a miracle. They say, there is no such thing. How are you still together? But when they surrounded themselves with a new environment and now this influence of the kingdom of God comes to their lives and we tell them that what you transmitted should not be a part of your life, your children may get a holiday, but not this problem. But maybe for us, this is not a sphere of divorce, maybe for someone, this sphere is physical, this sphere may be somewhere in finance, this sphere may be somewhere in mind, where there are always some complains at home, there are always some fights at home, there are always some disagreements at home, there is no love, there is no mutual understanding. I want to thank every person today that you are called by God, you are smeared by God so that what was transmitted, so that it would stop and so that you would pass the blessing and the victory of God to the next generation. Pass home, pass the money, pass the bitcoins, pass the stock market share, this is all good, but the best thing is to pass the faith, pass the real values, pass the holiness, pass the thirst for God, pass the prayer life, pass the respect to your wife or respect to your husband, to pass this is much more expensive, to pass the holiday, to this day the Jews are celebrating that there is a sphere of divorce.  What do I know? I am not an expert in these spheres, I have a little experience in this, but now the Lord even warns me that my goal in life is not just myself, we should leave something for the next generation and not wait for 60, 70 or 80 of us, but right now, all the victories that you are experiencing, brothers and sisters, they will not be left only for you, they will be passed to your children. If you are suffering today some kind of depression or fear, where the devil wants you to be in this, why, because he knows it will be easier for him to defeat the next generation if you do not get the victory now.   We think that Solomon came up with it, or got it from God, he wanted to build a temple, but he could not, he gave it to the next generation, but I want you to note what David did, which is very unique, he was a wise king, he said to his son Solomon, he said that when I die, I will not only give you blessing, but I have some enemies, whom I chose to live, and he begins to call them by their names, he says, this enemy, he has been with me for a long time, this enemy, he did this to me, he said to Solomon, please, you must understand, when I die, my enemies will not die with me, they will go to you, they will torture you, they will cause you any harm, you must deal with them, and Solomon, when he became a king, he already knew that, that, that, that, this is the enemy of my father, the enemies of David did not die with David, they went to Solomon, and the first thing that Solomon did, if we read it very carefully, he did not build a temple, he dealt with the enemies of his father, and when it all decided, and then it says so, in the third kingdom, the second chapter, the 46th verse, so, the kingdom was hopefully entrusted to the power of Solomon, when? he never won the war with the Philistines, this is already his father gave him this victory, the Philistines, there was no problem with the Philistines, there was no problem with other kingdoms, but there were things that David gave Solomon, and he prepared him and said, listen, there is this enemy, this enemy, and when you win him, then start building a temple, and then the kingdom of Solomon was built, maybe there are enemies who were in front of us, this laziness, evil, insolence, lust, jealousy, we have to win it today, and give the next generation the fruits of the Holy Spirit. It does not mean that people will automatically serve God, because we serve God, but it is much easier, 10 times better, when we pass, dear brothers and sisters, not only your fate, they decide the fate of other people, do not give up, do not share your life with the devil and the sin, go to the end, get this victory, because for this victory there is not only your life, your children, your grand children stand and receive blessing from what you give them. As for the sphere, I want to note now that such practical principles how to get God's freedom in our life. The first, we read from the beginning the sphere of the fifth chapter, the first and the fourth verse. On the third day, the sphere was dressed in royal clothes and it became in the courtyard of the royal house, in front of the royal house, the king was sitting on the royal throne in the royal house, he saw the queen of the sphere standing in the courtyard, she found love in his eyes and the king of the sphere and the golden scepter which was in his hands and the sphere came up and touched the end of the scepter and said to her, what do you want the queen of the sphere and what is your request even before the half-king this will be for you and said the sphere I want to note first that when Amman built this case and made this law to destroy the Jewish people it was a political movement it was a political action and the sphere that lived in the royal house it could be calm some contacts to start calling some governor some senator some people who make a law and start politically to solve the political problem but the sphere was a reasonable girl and she knew that what Amman built against Israel it was a political problem it was a law but the roots of this law they are deeper in the spiritual world and political methods they do not solve this problem and therefore the first thing that the sphere did it called the whole Israeli people to pray as if there is a sphere this is a problem with the law this problem should be solved on a state level this problem should be solved with some connections with some calls but Amman was not just an atheist these people were people who worshiped idols many of them they were held by beasts and through them the devil was acting to destroy the Israeli people and therefore all human methods they were not enough to stop these plans that were built by Amman we should understand as Christians that very often our physical difficulties or physical battles are rooted in the spiritual world I mean that it can be inside it can be in the mind because we are inside our soul they are in the spiritual world these are invisible means and we need to fight with the spiritual world not physical but spiritual when Maesey was on the mountain and Jesus was fighting with the Liketians it was a spiritual battle why I believe that it was a spiritual battle because what Maesey was doing on the mountain it affected what was happening below the mountain with Jesus it was not just a battle where who wins more but in the spiritual world something changed there when Maesey washed his hands to the sky and Jesus was getting victory brothers and sisters we Christians and we live in this world and very often especially we Christians because we don't want to be in the spiritual world and even there are demons or beasts we start very often like in Valhalla you know in Valhalla when she went to the dream and the dream began to go like this and he started to beat her and he beat her up and said if I didn't have a sword I would kill you and God opened the mouth of the dream and she told him what's wrong with you what are you beating me with what did I do wrong the problem is not with me in Valhalla but here is a sword and you are a prophet with love you don't see although you should see unfortunately the unbelieving world understands the spiritual world more than Christians unfortunately many Aslits see these angels faster than Valhalla who should see it I'm not saying that there is a devil behind every bush we live in the spiritual world he was created by the spiritual world and this world is a baby and the world that we don't see is what created this world and very often we solve physical problems with our hands not with our knees problems in the world we need to go to the consultation but we need to learn how to solve things in the spiritual world through prayer and through post for children it's good to send them to the Rept Center it's good to punish it's good to insist on the right path but other brothers and sisters in this world this world is spiritual and spiritual battles are not fought by spiritual means and the sphere and all people in this world she could immediately start to make some contacts and calls and she knows that it's a threat to death but it's not solved by political methods it's solved by prayer and post church growth it's not just our methods that we have good social media we have a beautiful face beautiful buildings we are like a church we are not a social club we don't bring people to think positively we save people from hell we save people from death we come here and there is a real devil who wants to take all people to hell and beautiful beautiful prophets beautiful music and beautiful photos and beautiful videos she can't do anything against the spiritual world and spiritual power it's the power of the Holy Spirit and she can take a person who doesn't know English well and use this person to bring many people to God and the other one says that it's better to have a church than to have a church on a thermostat and the church dies in America and the church is educated why? because the church is not just a social club and we live in the spiritual world where the power of the Holy Spirit it can solve many problems that can't be solved neither doctors, nor consultants nor our ideas lesbians, satanists people who are connected to you it's just a speech it's just the blood of Jesus there is a girl at school she used to be a satanist her parents dedicated her to her when I was 5 years old cutting off some things in her body and it was a blood clot you know, such a girl just beautiful, positive prophet they are not enough and when she came she got a freedom and so many personalities 13 or 15 different doctors she was sitting there she couldn't sleep she couldn't live such people need something more than simple words from the spiritual world and when we bend our knees we know that we can have some contacts we have some opposition we have some connections but we know that what we stand against we have the strongest power the power of connections the power of our name the power of our influence it's the power of the Holy Spirit Apostle Peter and Apostle Paul they made fun of their wisdom and their power even in Pharisees they saw that you are uneducated but there is a difference because in the spiritual world although they were not very educated but they were great people because all people with their knees they just fight against the problem they will solve things in the spiritual world the sky opens then it's easier for people to get salvation in that city because in the church in Korea there are many churches the 20,000, 20 denominations have the biggest churches in Korea Baptists, Presbyterians and other people have the biggest churches in Korea because Baptists, Presbyterians, 50 Charismans there is prayer I don't know when things are solved in the spiritual world through prayer and post not through our physical battles but through our battles in the spiritual world through prayer and post things start to be solved in the physical world I'm not saying that everything is easy I'm not saying that everything is not a problem but something changes later Amen I thank each of you don't rush to solve unresolved problems with your own hands it's good to go to a consultant it's good to look for advice it's good to look for physical physical answers there is nothing bad here you need to go to a doctor you need to get therapy and all this will happen in your places but if the roots of these problems are in the spiritual world all these abilities and that's why we pray then we don't rely on our power we rely on the power of God I don't understand the spiritual world but I know the one who created it his name is Holy Spirit and he can take an inexperienced person like me he can take an inexperienced person like you and lead all this labyrinth of the spiritual world and bring victory to our life and victory through our life to the next generation thank God the first thing that the sphere does is solving a physical problem with physical methods it's solving it with spiritual methods with prayer and fasting although she was a tsar but she doesn't go to the governor she goes to the post and prayer and after the post and prayer I want you to note the second what I read in the fifth chapter in the first verse that she was dressed as a tsar after the third day after the post she took off the clothes and put on the clothes in the royal or royal clothes the problem didn't solve when the post ended I believe that the sphere felt in the spirit or in the heart that something is changing that something is different although Aman was still alive Aman still did and still was close to the tsar but in the spiritual world I believe that something has changed and the sphere is doing she doesn't just solve things in the spiritual world because she prays and she goes to the post she doesn't just fight with the tsar she proves it, she panics or she does all sorts of big coalitions she prays and goes to the post and after that she is dressed in royal clothes when she went to the post and prayed the Bible says that Mordechai broke his clothes and he shouted God but when it ended and the sphere didn't come in the laxity in the presence of the tsar she was dressed in royal clothes the second stage of our freedom it's not just a prayer and a post the second is the update of our mind through the recognition of the truth there are many people who get freedom who pray and go to the post they get prayer so that God can help win this or that that when things change in the spiritual world they expect that something will change in their lives but in the sphere nothing has changed in their lives until they are dressed in royal clothes so we have to after the spiritual world we have to change in our mind we have to remove the laxity and we have to dress in the truth of the word remove our feelings and dress in the truth of the word remove the facts of the life we see and dress in the truth of the word the Bible says in the eighth chapter of John yesterday I read about it Jesus said that know the truth and the truth will make you free He didn't say that when you will find the presence of the truth no one makes you free just like the presence of the soap doesn't make you clean you can have a whole truck of soap in your garage and die like a rock why? because the presence of the soap in your house doesn't make you clean this soap has to take and lay on itself apply this soap Jesus says that the knowledge of the truth doesn't lie in your heart it doesn't do anything for your mind Advil doesn't help with the headache if this pill lies in a jar this pill should leave a jar and go to the mouth and go there and how does it go down and up it affects I don't know but it affects I don't understand how it works also I don't understand how it works how it changes my thoughts how it changes my feelings how it changes my behavior how I change inside although my situation doesn't change I don't know this but it works I don't know how there is green grass and white milk but milk is often drunk dear brothers and sisters most of the word it has to change before we see change from outside what does it mean practically to dress in royal clothes we have to say with God's word not with our problems I know that many of you have good cars it's noticeable on the parking but I think many salvage titles let's not raise our hands about 50% salvage title if Russian and salvage title it's like synonym salvage title means that the car wasn't bought in dealer, but in action we cleaned it and the one who doesn't understand he will say beautiful car until we start selling it and in this car there is such a document this document is called right to property title and there in this title there are a couple of words that immediately cut the price of this car these words, they scare Americans these words they immediately have a beautiful car but you just say this word the atmosphere changes immediately the price immediately falls and immediately look at this car and look at you only for one reason these words are written salvage or rebuild who is familiar with these words even those who speak low yes, I know that's why I still can't sell it banks don't even give loans many banks don't give loans because of salvage title the car is good 30 thousand miles nobody will know that it was in some accident because this car passed through the accident and then this accident was on the document which was hidden in the mess right to property is our thinking and very often we go through our life accidents, we go through the life of battles and these battles, these accidents, these problems they remain in our consciousness and they are inside something is decreasing in us and therefore the Lord wants us not to come to His presence and say, Lord, when everything is decided in your life, then I will put my mind in your truth and the sphere is not dressed in royal clothes when Oman died she is dressed in royal clothes when she felt something changed in the spiritual world and she immediately took off her clothes and came as a queen we have to change to change inside before our life changes from outside very often people say Lord, change your husband change your children change the situation change my boss change the pastor change the church, change the diacon Lord, change yourself change everyone but not me when things change in the spiritual world after that the second thing should change something in our heart and in our mind we have to think differently we have to feel differently and we have to speak differently even if the situation is still so that God would bring them to God so that the Lord would liberate them about these dependencies when you feel that God comes you have to change your feelings your look and your words about your children you can't call them drug addicts call them messianers you can't deal with them you are like this you are like this these words should change they should be removed go to the truth and see them through Golgowski's cross and say that the Lord will use you you will get salvation God is for you we will pray for you you will bring great harm to the devil the Lord will use you in the 12th chapter of the 2nd verse it says that our life is transformed by the update of the mind it means that our life is waiting when we dress in royal clothes our life is waiting when something changes and very often we are waiting for our life to change and then our life changes I pray there is always no joy there is a hard life there is a disease in this sphere there are financial problems there is a husband he doesn't help me children don't serve God I always feel there is always no joy I am always not sure but maybe brothers and sisters that prayer and fasting solve some problems in the spiritual world but what is left now is that we should allow not to define life what is in our hearts is it easy? of course not it takes a lot of humility and many of us it takes more humility than to fix our problems Bible says not to fix our financial problems not to fix our problems but to fix God's hands and takes humility to fix our hearts and minds before God tells us not that we feel it takes humility and there is a religious pride pride is that I feel and God tells me I don't want it God will change and then it will change me we should be believers we should fix what God says about us humility is when we say that we are sinners being righteous you are not a sinner who tries to be righteous you are the righteous who fights sin royal clothes you should put on yourself means you are the righteous Bible says the righteous falls seven times because it is not in the clothes the sinner falls once and can't get up and the righteous fights sin the righteous fights sin when the sinner fights sin sin will always swallow him but when the righteous fights sin he will win the sin so you should change your attitude inside Bible says that I am a new creation in Jesus Christ Bible says that I am not selfish people can look at me you are doing drugs people can talk about me but what should be from the word of God I should dress royal clothes what God tells me not about my feelings not what I see in the mirror not what I see in my bank account not what I feel not what I see but what God tells me I am a new creation I am the righteous it means that I don't have sin or I don't fight but I am the righteous who fights sin I am not a sinner who wants to be the righteous I am not a weak person who wants to be strong I am strong in Jesus who fights and weak I am not sick who wants to be healthy I am healthy who fights from illness I am the righteous who fights with obstacles and problems I am dressed in royal clothes dressed in royal clothes it just changes your world you are not fighting from the point of problems but from the point of victory getting victory does not mean that problems are gone God gave Jesus a victory and said I give you a account and now go fight what did he do with Jesus with royal clothes that Jesus does not fight to get victory because he has already accepted it my victory does not come from what I have won from what I have believed I am the righteous if you are fighting sin die before God but the second you have to remove the bad you have to change what God tells you you say how I fight and in this very sin you have to become the righteous here yesterday Boris said such a strong word for the spirit of fear and for the spirit of sin when we realize that we are the sons of God there are some things we will be able to hit them when we come to the royal presence in royal clothes not in laxmote when Isvir came in royal clothes I think she was not afraid of Amman I think she looked at him I think she felt sorry for him she already knew she already smells bad already smells kerosene already he was not lucky you know, she was not afraid of him that she even invited him to the pier twice why? something inside has already changed the same problem but the other was Isvir the first thing in the spiritual world and the second things change in our mind through the recognition of the truth therefore, less series and more worship of God less facebook and better face in this book load and so far from this book everything will not go here so far in your beard will not change what is written there you know, these automatic thoughts will not work but it should change I had a bad experience so we should get to know the truth Buddha said, I was looking for the truth Muhammad said, I am a prophet of the truth Jesus said, I am the truth know Jesus know the word of God she should make a change here do not wait for changes in life until there are no changes in the mind no changes in the spirit when you just walk like water this hurricane was there but Jesus walked over it because something has changed changed here and then the third we come to prayer Isvir comes presence of the king she prayed, she fell and I believe that something has changed in the spiritual world because some spiritual methods that change the spiritual world the second, she is dressed in royal clothes she comes as a king she she is not dressed in royal clothes for me it means that when I pray and feel after that I have to accept the word of God in my heart my feelings should change my vision should change I have to look at this problem from another point of view and I said, it does not mean that we ignore the facts please understand there is a teaching about faith which is just not true that faith is what you have to ignore that you have a problem but it hurts, how to ignore it it is impossible to ignore it is not faith it is self-invasion faith is when you are already fighting with the problem from another point of view you are fighting with the syriac not from the point of defeat but from the point of victory you are fighting with the problem from the point of Golgovsky Christ where everything was already paid and God already promised it you are already looking at sin not as a sinner but as a righteous person and you can fight with the same experiences but a different person is already fighting with these experiences the righteous person is already fighting not the sinner I know that it is so common in our language in the Slavic world oh, I am a poor sinner brothers and sisters, if you are a sinner you need to be a righteous person Jesus did not die as a sinner so that we lived as sinners or some of us we say, but I can't I don't feel like a righteous person let me ask you a question did Jesus feel guilty on Golgovsky Christ? of course not but he died as a sinner why? to make you live as a righteous person when you don't feel look at Golgovsky Christ he also didn't feel but he didn't die as a righteous person he died as a sinner I see Jesus saying you are my son I loved you, I accepted you you look at yourself and you feel the opposite we don't stop looking at ourselves we look at Jesus saying I believe in what you say I don't feel it now I don't see it now but I believe in your word I believe in your truth I am a righteous person I am your winner now in heavenly places I live in you I died as a sinner and now I go and fight I don't sit and put my hands like a couch and wait until everything falls into the sky and now I fight now I discipline my flesh now I give up now I close some accounts now I turn off this computer now I go and fight from the position of victory which I received from Christ drawing into his truth and everyone is with me Esfir is wearing royal clothes and she is standing oh, I have 12 seconds oh my God and what is interesting I want you to note that Esfir is standing before the king she doesn't create panic imagine his wife closed the credit card and there is a problem that everyone is killed all the Jews and this husband sitting on this throne sitting on this royal throne and he allowed it and she is standing in front of him if it were my wife and my mother we would have arranged hysteria how is that what did you think you don't know the king will kill us I don't understand come here we would have arranged such hysteria we would have raised everything on our feet Esfir enters the royal presence and look what we have asked him she says, dear king Esfir, what do you want Esfir, what do you need to fulfill the kingdom this is a good opportunity when the king asks what you need I want to live you know what Esfir says I want to be with you I don't understand your life your life will end in a couple of months and you have a chance to change it all and you don't ask him that he solved the problem and you ask to be with him for lunch she says, king I would like you to come with Oman I want to emphasize I believe personally it is the biggest secret to get answers from God it is not to look for them but to look for his face God will not deceive God will not deceive and God cannot manipulate I believe that Esfir she knew that I will talk about my problems I am his wife he is my husband I should appreciate him not my need I don't know how brave it is how strong it is to ignore the screaming problem and ask only one I want to be with you and I invite Oman why I am not afraid he will be afraid of me I am not afraid of him he comes to the pier and here is the opportunity and the king is waiting he knows that this wife did not prepare a pier for the pier he has something he wants to buy some slippers he wants to go to Dubai he knows for 30 days and I did not look something is there tell me I have already brought a cheque I have already brought a phone I will wire the bitcoins we will give you everything and she will ask the king can you come again to the pier he is frying it he is cooking for now he is asking please tell me what to do many of us this is a solution to the problem not to fall in love with him for him worthy so that you pray and pray for him I do not say that you do not need to pray and pray for the problem but if I pray and pray only for the problem and when the problem is solved and I no longer have thirst also pray also pray that then we do not love God and we use God for our freedom I know that for many people God wants to free me the first request is not to speak be free to speak love your God with all your heart and all your soul God knows that you are hurt God knows that this trick as if he allowed he is sitting here and you have to fight this pain to God God why if you love me so much why is there still a man here why is this problem here I serve you I belong to you we are together we have a master and I am suffering and we often where we lose our victory in life that we create hysteria we create panic and we increase our problems our difficulties so high that we manipulate God to solve them instead of loving him and then he solves them together with us dear brothers and sisters God wants to use us but we should not stop using him Israel used God to get out of Egypt and when he brought them out of Egypt they worshiped the cows why? because they loved God and Moses stayed there for 40 days he forgot that he had Israel which he had to lead he saw all the glory of God and the power of God which man can see and Moses had only one question I want to see your face Moses used the exit from Egypt to know God more some died in the desert worshiping the cows and the other looked for God's face and looked for the kingdom of God and his righteousness and all this else it belongs to us what I mean I never put my problem higher than your thirst for God even if it is a problem some sin with which you are struggling some habit that you look for God to answer this problem and when this problem will be solved and listen to me this problem will be solved by God never leave God never lead God never manipulate him I remember when I was addicted to pornography when I was a teenager I started to pray and pray and it was time after two years God freed me from it I survived God's grace such a sadness in the heart of Jesus as if now my appointment with Him is over as if the doctor cured me and now I will live my life and somewhere inside I don't know if it is right that I felt like I felt that God stretched my pain longer to hold me I felt this from him as if he said come when you are free you know when you are sick when there is some pain inside pain is such a strong motivation and we need to overcome this temptation to be motivated not by pain in life but with God's grace He loves your presence He loved your presence so much that he has a scar to prove it He is looking for you not manipulate him don't come like Esfir Esfir came to her and said I will die I want to meet you they met and he said tell me what I need I want to meet you maybe you have a sin to God why did God allow why didn't He answer why didn't Amman remove step back because the God who as it is now this Amman at the same time will become his slave and Amman will be on his knees and he will be the end but most importantly it is not death of Amman it is closeness with God because when Amman died Esfir did not leave the king she continued to be close to him until all plans of this Amman they were not destroyed God's presence she has to become the biggest and so that it happens we have to be satisfied that we do not do God our goal even if it is a goal it is what God promised even if it is a goal if it is a goal I want to gain more freedom I want holiness I want peace in the house I want God to bless me that God used me look at the Holy Spirit it is still a means it cannot be our goal God said in the frame I am your reward neither people neither Isaac nor finance neither my blessing I am your great reward He wants to be our treasure He thinks that He is worthy to be a goal for who for what we are looking for for our goals although this goal is not bad and I want to be in peace I want to be in peace so that we do not come only to His presence and say, give, give, give it was a stupid son but when his heart changed he came and said make me a servant I have no money I have nothing but I do not ask, give something changed inside God is taking us away from God give, give, give I believe in this, give, give I believe that God wants to give I believe that God is kind but God also loves us so much and He wants to interact with us He does not want to be used always He does not want to see that we are only a problem and when there is no problem and we immediately build our life make God be present in your goal and not your freedom or your help we are married you are 8 years old we do not have children I do not know why I pray for this, I prayed I gave it to God in my hands 4 years ago I did not go to the doctor and I will not go I know that it will be already there is a name chosen for this child there is even clothes even in the last 2 years for my whole life for my child I said to God Lord I am not looking for you for that I do not feel ashamed I feel ashamed why? Where? what is with you? or what is with my eyes? I pray for other people what is with you? my breakthrough in this sphere it is not for that what I serve God God has tested me and cleanses me I made such a lunch that was made by 3 Jews remember those 3 guys who said to the king God whom we serve He is capable of liberating us but if not for liberating us the king knows we will not bow to your gods and we will not bow to your stucans I serve He can bring a big breakthrough in my life but if He does not bring it my goal it was not a breakthrough my goal was not a miracle my goal was not a settlement my goal was not even my freedom my goal was God and when you do His goal they will die of lies to add God will bring this blessing but this blessing will not become an idol of your life because God is your treasure let us stand


Hello, and welcome to OpenShift Coffee Break. Good morning, Eres. Hi. We have our friend, Eres Kizong, from Run AI. It's time for a coffee for, it's early morning for me in London. Eres, where are you calling from? Eres, you're on mute. Eres, can you hear us? We did mic test before it was working. I'm not sure what. Eres, why don't you try disconnecting and reconnecting and we will see if that works, okay? So today, Eres will be talking about Run AI technology. That's, let me see. Let me see if he's back. Hello. Yeah, can you hear me now? Oh, there you are. There you are. The guy. Where are you calling from, Eres? Now I'm in Israel. I'm calling from Israel. But now all the internet and my coffee is ready for the morning. And you can see there's a nice load on it as well, so. He's not, it's not like espresso like mine, but. Yeah, yeah. So it's happy to see you. I'm very happy to see you. Good to see you. Eres and I used to work in the same team in Red Hat. Okay. So, Eres, tell us a little bit about yourself and what you're going to tell us about. We're going to talk about today. Yeah, perfect. So thank you very much for this opportunity to have this conversation. My name is Eres Kirsten. I'm a principal solution engineer at Run AI. And I came to talk about some of the challenges in machine learning and AI and workloads, a little bit about what's happening, what's happening around this resource management challenges that we have. And how Run AI tries to address and make this world a better place, especially when you use GPUs in a large scale. And as I understand, we're talking about resource authorizations, specifically about GPUs, in the context of Kubernetes and OpenShift. So one of the things that has been happening in the last few years is that high performance computing has landed on Kubernetes and it's slowly becoming the default standard for HPC in the AI ML community. Therefore, that's the need of adding these capabilities of managing those resources and optimizing that utilization for OpenShift. Yeah, exactly. I mean, it's exactly what you said. We see more and more workloads becoming cloud native. We see Nvidia with their NGC containers. But we noticed that today Kubernetes scheduling solution are not always optimized for GPUs. They cannot reschedule. They cannot have priorities. They don't have fractions. There's a lot of things that are missing that we thought to bring to the table and add it as features to everybody who's running Kubernetes. And I think the conjection with OpenShift as an enterprise platform will create a big and very good round solution. And I was wondering, not everybody that joins OpenShift Coffee Break has the AI ML background. So I was hoping that maybe with the aid of subslides, you sort of set the seed and help us understand what the problems are and the sort of solution that you guys are bringing to the table. Do you have something for us? Yeah, I do have some stuff made especially for you. I can share my screen one second and I hope this is going to work out. Let's see if we're lucky that we do a mic. Yes, let's see if this one works. Okay, so ready to share? Yeah, can you see my screen? Yeah, you can be maximized the presentation. Yeah, thank you everybody for joining. Again, I wanted to be really open conversation. You can add things to the chat. We do love open discussions and you'll feel free. And again, as the host, you can interrupt me. I'll be monitoring the chat. You'll be monitoring. Yeah, perfect. So yeah, in this conversation, we'll talk about how RunAI actually optimize AI workloads on RedEye OpenShift. And we'll talk a little bit about what's going on in the world of high performance computing, machine learning, IML. And we'll walk it as more of a journey conversation. Although we have a lot of slides, they usually prefer as a technical person to make it more of a journey and to jump from one slide to another if it makes sense. So this is a little bit about myself. I'm a principal solution engineer at RunAI. I used to work in RedEye before, amazing people, and I'm still in contact with many of them. In my background, there's more from Kubernetes, HPC, Cloud, so a little bit of both, so I can bring some of the knowledge to RunAI. So let's start first to see what are some of the challenges or some of the infrastructure challenges that we see that we see in the field is that there's a lot of data challenges that we see. But in terms of infrastructure and compute-related challenges, it's something that how do you manage a lot of resources and how do you make the infrastructure part less complex? And this is something how do you utilize all the hardware that you have? And this is something that we see, of course Kubernetes helps a lot, but we see it as a very big challenge when you talk about AI infrastructure, it's complex. It's a little bit different than running regular workload because they have different types of characteristics, the retention and the type and the people or AI predictioners who use the system have different topics. So making infrastructure easy is the first challenge that we see actually in the field. And how do you build the right foundation for your AI? So we have the cloud-native transformation for how you make a cloud-native, which is a challenge. And then you have the AI ML data scientist, which when I say machine learning data scientists, they use a lot of data. They prepare data, they have their own way to work with the data. And combining those two to hardware accelerators, because now people need to use GPUs. And GPUs or graphical processor unit, when people use data scientists, they can accelerate the workload using hardware accelerators, mainly from NVIDIA, so they can shorten the time. But when you build the infrastructure, and this is actually what we call the triangle, you have the cloud-native, we have the pods, the containers, you have the data scientist, and now you need to bring it. How do you make use of your hardware accelerators in the best way? So this is the foundation that we see many times in the field, how people and companies work building their own AI initiative. And I want to talk a little bit about the hardware accelerators, because at the moment, we see that many hardware accelerators are used from high-end enterprise customers, and sometimes it's hard to get hands on all those resources. They're becoming more expensive. There is shortage. Many companies, it could be Facebook, Tesla, Dubai, a huge amount of NVIDIA GPUs. And the reason is because they want to make and accelerate their workloads, to accelerate their machine learning type. So this is one of the challenges. How can I get my hands on it? After I got those GPUs, how can I make sure they're not idle in the system? Because about expensive GPUs, how can I make sure they're not just resting? And I want to make sure they're fully utilized. And this is some of the things we bring to the table, because we don't like idle GPUs. We want everything to work all the time. And once people have the GPU, this is a questionnaire that we've done is, okay, you have GPUs. How can you now share them? How do you share those GPUs? So if you decided to share those GPUs and not give each GPU to one researcher or whatever, we can see that many is a manual request. People create Excel files, say, hey, when you're done with your GPUs, can you please call me and give me the GPUs? Maybe they can do a tracking system. And the reason we see it is because it's not a dynamic location. Once, especially in Kubernetes, once Kubernetes gives you a GPU, you're hogging your handcuffs with that GPUs till or you finish your job, or you stop your job and give it to somebody else, because this is how the system is more designed. And we can see at the field that it's a very, very difficult matter. How do you share GPUs, especially when you have many of them, and manual request is something super popular with data scientists. And I'm going to talk now a little bit of a recap around the data science and the new tools. At the moment, there are 100 tools that you need to choose from to start working with machine learning data science. And if we're looking from the left to right, this is usually the process that we see in companies. The first thing is data gathering. You need to take all the data. It can be pictures. It can be from the web. It depends on your business logic and goals. And then you need to clean and validate the data. So you have the data, you clean everything you need from it, which is the stage, and then you label it and you're preparing the data for the training model. Now the business, they said, okay, I want to see, for example, I'm building an autonomous car. I want to make sure that when the car is driving, I want to train it to understand traffic light. So now you're giving a thousand pictures of different traffic light. Sure, that you can actually understand what traffic light is and you try to even mix it. So you're doing experiment tracking. And this part is very, very heavy because you just hold up GPU power. Long models creation can be sometimes days, hours. And this is the training part. And once you're ready, you want to take what's called inference. After the model is trained, now you want to say, okay, I trained the model, I want to see if now the car can really understand what the traffic light is. So you're giving, you're throwing it pictures of traffic light and it gives you, yeah, it is a traffic light or not, but this inference, it's a small portion, it's a small file of the training. We can see it by the way in mobile devices. When you take photos, you see those square boxes because actually the phone understand what a face is. But this circle is an ongoing process. You check the inference, maybe you'll find error, you need to retrain it and it's a cycle and always goes training and inference as a loop. But the thing is, is you need to choose a lot of tools and it's a different way of working if you compare it to DevOps, for example, because in MLOps, it's a different paradigm. You need to take the models and you should take the data. So it's a little bit different from DevOps in the sense that it's more focused to machine learning and AI in a workload. So many people who come from the DevOps field now say, hey, now we need to learn a little bit more because it's a different set of tools, a different set the system is running and the jobs are different. It's not like a web service or NGINX or something like that. It's a little bit different. So it's a new best practice that you have in terms of MLOps mindset. Moving forward, this is, I think this is something that Red Hat even actually knows much better than me in terms of the cognitive transformation.  I do have a question. He sounds like a question from the Pinger Gallery in the sense, but as it were, when you were speaking about the different phases, RGPUs mostly used in the machine learning phase or is there a usage also, let's say the execution of the learned algorithm in the field or that doesn't really require that much computing GPU power therefore, and because they are expensive, it's probably not a good idea to have them run in the field. So the GPU is mostly at the machine learning phase. Am I wrong with that? Is that a fast statement? No, you're right. It's a very good question. Is that after you train the model, model where the model is running afterwards, if we talk about edge devices, for example, we see that today, even in phones, even in cameras, you have small GPUs. You have created small GPUs, boxes as well. So you don't have to use it, but if you use it, you can get a little bit more performance at inference. So you have a decision to make, but yes, when you run training models, you need more GPUs than inference. However, we want to give the opportunity that if you run an edge, you want to use a GPU or a fraction of a GPU, you want to give the opportunity to use it, but yes, you can actually decide what to use. However, yeah, I do see a lot of small compact edge devices that use small portions of GPU just to accelerate the results. Right? Go ahead. Yeah, perfect. Yeah, so this slide actually from Red Hat, it's something that Red Hat did, is what's running on Kubernetes today, what workloads. So we see that it's actually happening right now. I mean, people are using machine learning and Kubernetes are using machine learning and OpenShift. So people are actually using it as we speak. So it is popular. It's getting a lot of momentum in the past. It wasn't that possible. And now, you know, Red Hat has a lot of tooling around it, the Open Data Hub and Open Data Sign. So it is something that becoming more popular and more companies actually use it. And but there is one problem, is that many people are using it, but today that the fact that Kubernetes is not designed to run all those workloads because of the way it was, but it wasn't designed to run AIML workload. For example, there's not no mechanism to do advanced priorities or policies. What about fairness algorithm? Maybe each team needs a different priority of amount of GPUs they want to use, or topology aware. So a lot of the management around AIML is missing in the current Kubernetes scheduler. And this is something that it wasn't designed in the beginning to use those types of workloads, but we see it in the field that it is causing some of the problems that we need to manage all those resources. And again, this is more of a Kubernetes thing that was not built for those new set of workloads. And I can show you some of the example or some of the challenges that we see in the field is when you have a small amount of GPUs, again, this is a Kubernetes cluster at the right, and we have two researchers at the left, and we promise each one two GPUs, for example. So it's not much of a challenge when you have only four GPUs, but when you need to scale and you have much more GPUs, you want to say, hey, you know what, maybe I need to take a light blue team, and I want to give them different resources. Maybe I want them to have 10 GPUs and the other team want to give less. So I don't have any tools today that act in set priorities and select governance around different teams that I have. And I cannot enjoy fairness scheduling algorithm. I cannot do joints, priorities, and maybe I want to run more workloads on the same GPU. And this is example of fractional GPU. Maybe I want to use not the whole RAM, but some of it, because I have a Jupyter notebook or something interactive. And today it's quite challenging to do it with not much tooling. So this is the first challenge that we noticed in the film. This is one of the reasons why RunAI was created in the first place, to handle all those resources, requests, making priorities and scheduling, and fairness algorithms to make it work. And the second challenge is monitoring how can we know how the system is really used? Because maybe you're running a job but you're not utilizing all the GPU. Maybe you're using 30% of it. So what does it mean? Maybe we can put you on a high-end GPU or on a low-end GPU or how to make sure that all the system are utilized. So giving a bird-eye view of how all the resources are used in the system is super important to understand how the system is used, who is using what and governance around it. And later you can build even build systems. You can do a lot of API calls. Could we support actually everything? So those are some of the challenges that we see today when you run AMML workloads using Kubernetes. And this is why we created RunAI. So RunAI was created to attract all the little GPUs in the back of a pool, the GPU pool, and by adding our smart or called super scheduler, which reside, in this case, if it's an OpenShift, it's another scheduler in the system. So you have actually another schedule of RunAI. And using this RunAI scheduler, you can now submit jobs using RunAI scheduler and get all the smart scheduling in the system. So this is the main thing, the main two layers that we create in RunAI is how we take all the GPU pools create one big pool out of it. And when we can give those GP resources based on the run it your way or run it your way, you can select whatever tier you want. You can use RunAI tools to submit jobs or you can use different, you can use Kubeflow, you can use Airflow, Jupyter, you can use OpenShift, Jupyter notebooks if you want. For us, it's transparent. It's very important for us to be open, truly open to other tools that are needed. We don't want you to select which tool to use, but you can use whatever you want from RunAI to what's happening in the field. And given this platform where we can actually manage the resource, give you a cloud-like experience, it both makes the system quite unique in the absence of how we can manage all those resources in a smart way to the data scientists and data practitioners. Yeah. We're interested here. So when you were talking about Jupyter notebooks, so whether Jupyter notebook runs on Kubernetes, it would be running in a pod and so the Jupyter notebook server runs in a container. That means as long as that Jupyter notebook server runs, it has an allocated GPU that is dedicated to that. So if the data scientist is like editing the old book and not running anything, still the GPU is not available to anyone else. Is that or does it work a different way? Yeah. Super good questions. So I will talk a little bit about roadmap as well. Yeah.  And yes, it's called an interactive session. Interactive session means that we will not preempt it. We will not kill it because it's interactive. However, if nobody's using this pod for a certain amount of time, we can have a way to suspend the job or even delete the pod. If nobody's using it for, let's say, one day, two day, you can decide. What you said is true. When you use Jupyter notebook, you use a full one GPU and you actually not using your resources in a smart way. However, one of the tools we added are actually two tools. The first one is maybe you can use a fraction of a GPU. Maybe you can use a third of the RAM of the GPU. So you can run more workloads of the same GPU to be more productive. So imagine you can run five or six Jupyter notebooks on one GPU. So you're using your hardware in a much smarter way and let the data scientists work and interactive. In the roadmap, which will be released very soon, the problem you addressed is a well-known one. Somebody's using the Jupyter notebook, but he's not using the GPU inside. He's writing some code and this GPU is not doing anything. So it's going to be released quite soon. We have a term called swapping. So what will happen is he will work on a CPU, but the minute he needs a GPU, we will copy the memory from the CPU to the GPU and then he can allocate a smart way of GPUs between Jupyter notebooks when they need it. And then he can use that GPU and to move on the fly from one user to another. Again, this is roadmap. It's not there yet. But at the moment, the way we solve it is we use fraction GPUs in Jupyter notebooks. Yeah. Okay. I think I'm guessing the picture. Slowly guessing there. Thank you. No, no problem. No problem. Yeah, but Jupyter notebook, it's a, yeah, I can actually do it later. No problem. And I'll talk a little bit about our platform. There is actually a question there. Yeah, yeah, of course. And I guess the question is about the intrinsic capabilities of some NVIDIA cards. They say, wouldn't having the NVIDIA A100 card able to slice multiple profiles up to seven Jupyter notebooks? And I suppose that's the capability of the NVIDIA thing, but is Kubernetes going to be able to do that? So that's the question for you, of course. Sure. No problem. When we talk about fractals, you can let me just, you know, I'm going to go to the fractals. Okay. Our motivation when it is, the first motivation is to use any GPU. So imagine you can have, you can slice or fraction a GPU. It doesn't have to be A100. You can even use different types of GPUs. These are motivation to be open, use any GPU and fraction it. When we talk about MIG, you can have MIG profiles. And we can do dynamic allocation on the fly for dynamic MIG profiles. You don't need to read with a server or something else. And yes, you can run, and we can, you can even say, hey, you know what, I don't want to use MIG profile at all. I want to use one AI fraction. We can slice the A100 using this method of fractional. So even, you don't even have to use, maybe you can use, let's say 10 up to 10 containers in one GPU. So we give you actually the flexibility, or you can use a MIG or you can use R-Solution. You can really, it's going to depend, we understand MIG, we understand MIG profile. So when you run jobs, you can select what MIG profile to run. We can do it dynamically, dynamically on the fly. So you have actually the decision, but the first motivation was, what happens if you want to run fractional GPUs and you don't have A100? We want to give you the opportunity actually to do it. A bit of what, actually a question, because I can't see exactly the chat or... I'll ask Andy and if that answers this question. Okay, we can have it in the meantime. Yeah, we can read more here. Okay, so now I want to talk about more around a deep dive about how our smart schedule actually works. And the first term that we have is called guaranteed quotas. Guaranteed quotas is a way to provide quotas for teams. So let's say each team has, we're providing, I have a system with eight GPUs. I can provide four GPUs for each team, for example, but it can have an option saying allow over quota, which means if there's available GPUs in the system and the other team is not using it, I want it on the fly to give it to the team who needs a GPU. I don't want to keep idle GPUs in the system. So we take care of the queuing and we take care of the GPU pool. And once the team is back in business, so let's say I'm using eight GPUs now for teammate, and now team B is back in the office and now they want to run some jobs, that's where the fairest algorithm will kick in and say, hey, listen, I want to take four GPUs back to team B that are back in the office. So I'm going to preempt the jobs, put it in the queue till we have more resource in this dynamic allocation, what makes the system utilization much, much higher. And again, everything's on the fly. Zero touch, the researchers don't need to do anything to just run the jobs. And I can show you a quick example. I hope you like the animation, but I'll show you a quick example before a live demo, how it looks like. So in this example, this is a quick example of we have researcher A and researcher B, they're using a system and this is the run I scheduled. And we can see that two jobs are running for user A and two jobs for user B, and we promised each user guaranteed quota of two GPUs. Okay, but however team A decided they want more jobs, but right now they're in the queue. So team B is on vacation or they came to listen to OpenShift Cafe with an AI, so they're not doing any jobs. So we have two additional idle GPUs now in the system. And remember, we have two jobs for team A. Although we programmed two GPUs, we now gave them four. The reason is because we had three GPUs in the system and they can go over quota. And now that they're going over quota and being more productive, they can run work workloads. And this is only available with run AI. We want to make sure they run as much workloads as possible. And now team B is back in the office is, hey, I want to run a job and you promise me, I'll always get two GPUs. So what will happen is we'll take one of the jobs, we'll put it in, we'll make it work at a GPU and we'll preempt one of the jobs and put it in the scheduling in the queue system of run AI. In this scenario where everything is dynamic and the researchers just get their guarantee quota and they get over quota when needed, it's what makes the performance, because our motivation is that partitions will be as well as the ability to run more workload, more productive and finish on time. So this was a quick example of how our schedule work in terms of preemption. I haven't even allowed that one because you have to get the possible, but I want, this is important for me just to emphasize how our schedule work and how we provision GPUs between users. And again, this is dynamic. This is the additional layer of the run AI schedule we added to the system. And now I want to talk about the architecture. I tried to make it more open shift friendly. By building the layers, I looked inside open shift and I feel free to ask any questions. So this is from the deck of Red Hat OpenShift that we have the operating system. We have Coral West, we can run it literally anywhere. We have Kubernetes, which is administration. What we do in run AI, we have an operator that you can actually, it's a different operator. I'm going to release another one soon. And we added additional scheduler that is side by side with OpenShift Scheduler, which is Kubernetes scheduler. So this is the first layer that we actually need. And we added another pod that communicates with the GPU. This is some of the secret is how you can provide fracking on GPUs. It's another layer that sits in the middle of communication with the, communicating with the GPUs. And in terms of run AI, we need the no discovery operator because we want to make sure that we know how to communicate with the right node and we need the NVIDIA GPU operator installed before installing run AI because the way we work, we communicate with a DCGM exporter, which is a layer in the NVIDIA operator to get all the metrics and data from the GPUs. We can show it in our GUI, we can show the utilization based on the current NVIDIA GPU operator. And once you have those two prerequisite installed, you'll get to install run AI and install our operator. So this is the first layer in terms of the integration of our scheduler system when you install it in OpenShift. And the second layer that we installed is the Atlas platform, which is the whole GUI interface to give you a cloud-like experience when you work with run AI. So you can use our GUI to submit jobs or CLI. Or again, we keep it open in terms of integration. And because we're an operator in the system, so if you scale, if you add more worker nodes, we will see them at the system because when you add or scale a worker node, the GPU operator kicks in, it will install the driver, whatever needed. We see it as another worker node and you'll see it in the GUI interface. And the last bit is you can select integration, how you want to run whatever jobs is needed. You can run Jupyter Notebook. You can use Interactive like VS Code, Qflow, inference. A lot of those naming were open, whatever you want to use. And again, if you're using an operator that's using Jupyter Notebooks and a lot of Qflow, not a problem. You can integrate fully with us, so with RunAI too, you can use the scheduling. And what we see in the field is many people using all those tools, which are great. And then they add RunAI to help them with the resource management layer. So that's how we work, that's how we glue things together in terms of the integration. By the way, are there any questions because I cannot see it in the chat? I'll read a couple of questions. I have one myself first. All those workloads that you mentioned there, do you need to make any changes to the images, let's say of Jupyter? Or can I use any anyone? Can I use any that come with my own tool set? Let's say I use Qflow. Do I have to inject something into those containers to run it and manage it? This is perfect. This is a great question. You can run whatever Jupyter notebook you want. When you run the job, you just have to select how many GPUs or fractional GPUs you want to use. And then when you create the pod, it will download the image and then we will give it, we will show Jupyter how many RAM or how many GPUs it will see inside the pod. So when you run the NVIDIA SMI, you can actually see the output. And you ask a very good question around Qflow, for example, all you need to tell Qflow in terms of integration is use run AI scheduler. There was a place in Qflow, and we have a lot of documentation around it, you can add to, hey, use run AI scheduler when you run the jobs. So you can still keep enjoying Qflow as you use today, but when you run or submit, it will kick in the run AI scheduling system. So the integration will be fully, will be easy, so the decision is going to keep working. And at the end, maybe I can send in the chat links to how to glue together all those, yeah, I can show that. Absolutely, I'm going to copy them. There was a question from Dwayne, I admit. I'm not sure I understand it myself. I'll read it for you, Dwayne says, I thought for HPC, the parallelization of data was a major task before GPU application. Is there a container nest practice for this? Personally, I don't understand the question. I'm not sure if I need to ask Dwayne to phrase it, or maybe, can you read it yourself, or shall I? Yeah, you can, I can stop the screen sharing, and I can try to read the... So that's the, you can see it here on screen. Okay, I want to see, if I understand the question correctly, so there's the data preparation before running the training load, and the second thing, if you want to run multiple, you call MPI, which means multiple process, you can run each node, each part in a different node to have a multiprocess work. So this is something as well, and then there's the training. But yeah, I mean, the data scientist should prepare the model and all the algorithm inside. We provide the resources for it. But if needed, we can take it offline. Dwayne, if I'm gonna have no problem, if we need another part of the presentation, just understand more later. Yeah, yeah, yeah, sure. And, Erez, did you... I don't see your screen sharing anymore. Did you want to... Yeah, I stopped it just for the... Did you have something like a live demo to show us? You want to complete the presentation first? Yeah, okay, can you see my screen? I can, I can now. But you need to go back into the screen. Can you see it? Thank you. Okay, so yes, I will show you a demo of our system.  And, you know, doing an OpenShift, this is something we have to show. Because RunAI integrate with IDP of OpenShift, so when you're still running AI, you can actually log in using OpenShift credentials. So this is just an example of how it looks like when you log in to, I hope I have the right password. So once you log in, RunAI credentials, you can actually go into the GUI interface of RunAI and you can actually start working with the system, Red Hat Cafe, it's a cluster I created where you can see the GPUs and the resources and everything. But I want to show you a different demo because I want to use a more robust system that I created for the demo purposes. Tell me if, okay, is the screen big enough or you want to increase the front end? Probably a bigger front end, I think. Bigger front end, okay. Better now? Yes. Okay. So what we see here, this is a live, let me just increase, can make it be a bigger. This is an example of a running RunAI system. I can see that I have two nodes, I have a total of four GPUs and I can see that four of my GPUs are currently allocated and I can see how I have Team B and I have Team A. If you look at the project, the project in RunAI, we call it, it's like an open shift project. It's a project with steroids because we add a little bit of more annotations inside the namespace to give a little bit of more capabilities so RunAI can actually read from it. And what we see here, we have Team A and Team B and we assign each team two GPUs, okay? However, if you look at the project itself, we have GPU over quota, which means if there's idle or free GPUs in the system, I want to use them, although, but the guaranteed quota is two for each team. If you go back to the overview, we can actually see that one of the team is using more than two GPUs. Team B is using three GPUs and the reason behind it, because we have GPU in the system and Team A is using only one. So what I'm going to do now, I want to spin a job for Team A and I want to see how the system automatically kicks in the fitness algorithm and balance the system. So each team will have two GPUs. And the way of doing it, again, we have a CLI interface and we have a UI. You can select how you want to submit jobs and you just see that I'm kicking it in the right place. I want to kick it to Team A. So I'm going to submit a new job now for Team A and this interface is for the researchers where I can select how many GPUs I want and what's the name? Let's call it quota back. I can select, this is just an example of POD that we have that loads the GPU. I can have resource allocation, storage if I'm using OCF anything I can put in storage class on FS. I have a lot of flexibility and again, everything is scriptable and available in YAML and you can even pull in get repository if you need for your POD. You have all needed tool sets you can easily train your model. And I'm going to submit this job now and what will happen, the system will actually pull the image and try to run it in one of the nodes. And you can see here that we have Team B with the three GPUs, Team A1. First part will be kicking it inside the pending state and say, okay, what decision should I make? I have a POD that is that's run in one of the teams and I want to make sure that it's fully balanced. Wait a few seconds for it to kick in. So you can see now Team B already, one of the job is preempted to a pending state and the state, you can add webhooks to make sure that the system will keep track of where it stops. So the next time you run the containers will continue from the same place. So this is done automatically and now we can see that system automatically preempted the job. In a few more seconds, Team A will get the additional GPU to make the system balance. And here is example of how the fairness algorithm actually worked. And you see one of the job is pending and now they're balancing again. The researcher didn't need to do anything. He just ran his job and we took the heavy lift and shipped for him. So now Team A is more productive. We can allocate more GPUs are allocatable. We want to make sure no idle GPU is in the system. And this was a quick example of having I'm just going to leave some jobs. I want to show you the Jupyter part. But this is a quick example of how the system runs in a very fast way of preempting and dynamic location, which is only available today in OneAI. And before I show you Jupyter, we can do the same thing. Imagine you have a system with much more GPUs, for example. And now you have different project organization or maybe each organization needs a different type of GPU. And you can see that only organization C can be, for example, over quota. But the rest, I don't want them to be over quota. I want them to use AGPU. So if I just load the system, this is a script to just run jobs. It's going to show you the syntax how it looks like. But in this example, it's more to show you how real system works when you schedule a lot of pods to the system. And it will kick in in a few minutes. And here you can see how you can govern and make sure that each group is not using more than your resources. You have a lot of plays for it to work. Fill it slow. I want to show you the Jupyter part, how the Jupyter notebook looks from a GUI interface. So this was what we did. We kicked in a training job. But this time I want to kick in an interactive job, which is a Jupyter notebook. This is an example of a Jupyter notebook. Again, it's not something special. It's just a Jupyter notebook. And I can actually select, I can say, either a fraction of it or a full GPU. I can actually really decide. And when I kicked this job, we understand it's an interactive session. We're doing some of the jobs because we don't need them anymore. And we'll wait a few seconds for it, for the notebook. And we can see it as it's going to download the pod. It's searching for the right place. I can see the logs. And we understand, you can see here, it's an interactive session. So we treat it as an interactive. Let's wait a few more seconds until the pod is actually that you're running. It's now pulling the image. We do it from a private registry. From your registry, we support generally everything. So when the job is tagged as interactive, what happens? You can say that the job is allocated a fraction of a GPU. If you have a team, and let's say you have 10 researchers in that team, and you've given them, I don't know, five GPUs, would that fraction allocation happen dynamically? Or each of them will have to choose when they start their Jupyter notebook. What fraction of GPU they were allocated? Yeah, when you run the Jupyter notebook, you need to decide what fraction of the GPU memory you want to use. It's not, it's not automatic on the fly. You need to decide, the researcher needs to decide how to do it. However, we're trying to add more capabilities of how you can govern the system with some admission controller and stuff like that. But at the moment, the researcher is responsible for the fraction, sorry, for the fractional GPUs. My other question is, I suppose you can extract those sorts of metrics from here. And I imagine, and I'm thinking of a project that was done where you have a really large number of several teams, many researchers belonging to different parts of the organization. Is there a way to extract metrics that would allow for chargeback? Yeah, yeah, it's a very good question. I think I need water now, I'm out of coffee. Yeah, that's perfect. Yeah, we do support using, actually in the next release, we were going to add a building dashboard where you can actually, which will be a much better looking field, but at the moment, you can actually decide and select data in terms of projects, jobs, you can have API calls, and you can actually carve down whatever information you need from a researcher or from a project and you can make more decision and billing metrics out of the system. It's something that is supported, API calls, you can get all the data, again, it can be used behind the scenes, Grafana, Prometheus, where you can actually get all the data. Yeah, you can get all the data just, but we're adding now more tooling, we're adding to the GUI interface, so you don't need to do some API calls yourself. But if you want more information that is not part of our GUI, you can actually curve and actually make, and some of our customers actually use it as the metric between departments. So they can actually see what departments need more GPUs and actually carve it down. This is a demo system, I can show you a real live example if needed, but here it's just this data. Yeah, so if I'm going back to the system with all of my GPUs, we can see that each organization now has a different amount of GPUs being used, and I can, organization C can actually grow to more GPUs, I can add more job to it. If I enable the over quota, I can actually add even more workloads. So I'm giving the capabilities to the organization to decide what teams need, what type of amount of GPUs. In additional, you can select which GPUs go to each team. Imagine you have very expensive GPUs and you have T4 which are less expensive. You can decide that maybe the students in the university will use the less expensive GPUs, but the PhDs, they're going to use the P100. So you can actually govern what GPUs of the nodes you can actually use in the system. And this can be done in terms of the node of 50. You can select, you can create node affinity and select which GPUs you want to give to each project, just to give you a little bit of more control over the system. And again, we have everything is scriptable. Everything is CLI. Everything is, we try to make it as easy as possible for the researchers and for the administrators actually to give and provide jobs to users. So we have another question. And I'm actually not sure if this is the scope of run AI. I can try. Because the question is around management. So I'm reading the question to you. How intentional is the data provenance recording whereby a data set can be shared and validated for reproducibility or trading? I guess this has to do with how the data sets of manners are used by the workloads. But I don't know if that's the scope of run AI at all. Yeah, if I understand the question, there are a lot of tools like question biases and Jupiter workspaces and a lot of third party software or bonus software actually give you that capability of tracking the data and tagging it and all the data sets. There are some solutions that are not part of run AI. However, because we are fully aware of what's happening in the field, we listen to the customers and see if it's something we should add in the product. And the integration is super important. But if you have data set that you want to use, you can definitely look it up and understand better. What does it mean in terms of the cycle? Because some have CI, CD pipeline where they have training and then they have inference and then go back to training. So we do support those pipelines because they're adding as a scheduler or as an inference because today we have more capabilities. But yeah, usually today third party software actually is in charge of the data as of today. So essentially it's being able to track what runs are using what data sets so that you can reproduce the same run, making sure that you know exactly what data set has been used for a specific run. Yeah, I mean, basically we have when you run job and run AI, you can, again, if it makes sense, of course. Jobs. Because we have all the information you can do run AI, you can actually describe some of the jobs and you can add some information what it's actually using. You can actually make a YAML file out of it and you can get some data. Again, if it helps, it's the train one. So consistency. And you can get some info. Again, if it makes sense, there's a lot of information, we can see what the command line or what data set you use or what NFS path or what storage path or you can actually script something if you need more information how the pod is using but at the moment it's not part of run AI. Again, if it makes sense, we can help you build something around it but at the moment it's not a part of run AI. Okay, there are more questions on the chat. We do have another four minutes to go. Yeah, so I'll give you a bonus, something bonus. Yeah, okay. Because we have to say something about open source, right? So we have run AI top or we have run AI top which is actually a tool is to give you information how your current GPUs are being utilized. So we have a GitHub page, we can actually download this tool. You can run it everywhere inside of pods, machine machines, whatever, and what this tool is doing, doing actually two things. It gives you, in terms of the CLI, you can see how the GPUs are utilized in terms of the node from the node level and you can see which GPU and you can get information if you want to see it on the fly. And the second thing, which I think is a very cool feature, you can export it to a file and actually look how over time the total utilization of the cluster. So you can see maybe it was 50% utilized or maybe it was utilized more by once server zero but wasn't utilized for server five. And this tool can give you a little bit of information how your current system is being utilized, it's open source. We have more tooling in the future, so be tuned. And this is something that you can definitely be part of. We'll send it later in the chat and just get a clear understanding how this system is used today. And it seems that we have a lot of tools, open source tools, actually. If you could catch and paste the GitHub, the URL for the GitHub repository and then put it in the chat when you have a moment. There you go. I'll try to do it. Yeah, try to do it quite fast, live as we speak. Yes, no worries. And you know what? I tried and are you sure it works? Let's see. Yeah, maybe there's a typo there. Because it says page of time. All the testings. This is the right one. Yeah, no, no, it's good, it's good. This is, yeah, live. Try this one. Okay, there was a missing. Yeah, yeah, the good clone command. It might work. Yes, I'm going to put you in the chat for everybody to check it out. And also, if that's okay with you, I'm going to put your email address in case somebody has was actually with you. And I guess that's all we have time for today. Any final comments, Eres? Other than thank you so much for joining us today. No, thank you. I mean, I feel at home, you know, I feel at home here. And thank you for helping organizing this thing. And I can give you, you can put it in the page, the RunAI webpage. If more information, feel free. It's needed. We can open a Slack channel if you have questions. Feel free to, you know, involve us. We have a big production team who would love to listen to some of your work cases. And many thanks to those who joined us today with those also questions that I admit, sometimes I didn't understand. And I'm not an AI expert by any transmission. Also wanted to say that coffee break goes on holidays as well for a couple of weeks. And we're back on the 24th of August. See you all in the 24th then. Thank you guys. Thank you. Bye.