Okay, welcome back Any questions before we start if you have any questions any if you'd like to share your thoughts We're sharing. I hope well we're all understanding and learning Any questions any thoughts you'd like to share as well feel free any challenges that You know people have shared with you and you feel that You need to grow more in that area. You can always feel paid to ask questions. You all are learning together Okay, so there's no questions we'll get into our notes. We just present the notes Okay, so we stopped here Give no room to the enemy This shut every door that you feel You know that the enemy is trying to gain entrance close that door Just pray God's word pray for The empowerment of the Holy Spirit to enable us to live that holy life right next point is receive healing for offenses Hurts and wounds Now in life each one of us whether we are leaders whether we are cell group leaders pastors Associate pastors senior pastor Pioneering a ministry whatever it is We will have opportunities where we will be hurt by others offended by those who are near us and those wounds are very hard because You know many times if you you know you confide in somebody and Feel they're like your brother or your sister and they turn their back against you or the You know say wrong things Ridicule or hurt you. It's very difficult to receive healing for those offenses Luke 17 one says then said he unto the disciples it is impossible, but that offenses will come But woe unto him through who they come To be effective ministers of God We cannot carry offenses hurts and wounds not in a beam We cannot carry that baggage everywhere we go Right because this will hinder the work that God wants to release through us right and Are there people who hurt me in in in these many years of ministry? Yes, they are Right, they have been some who have said things that have not been true and things that have you know may have I wish I could clarify But one of the things that I personally learned is to forgive and to let go Because many times when we hold on to these things I Realize that I'm holding on to the past so much that it's not allowing me to move forward These offenses these hurts are so much It could be something very simple many years back, you know, I was leading worship and this Very well-meaning Individual this man came up to me and said, you know, you need to change the way you lead worship songs are The songs that you choose and so I think you should really spend time in God's presence and Choose the right song. I was deeply I took it Very offense. I was very offended by what he said. Why because I should spend many hours You know, I was in God's presence like, you know asking God to minister to me minister to people and Many times people have said, oh Paul you were you know, I was blessed by the worship. It feels nice But now all of a sudden this one came up to me and said, you know, you need to change the way you lead worship I said then I remember, you know, justifying that and saying, you know, what you're talking about because so many people said the worship is good. So many people like my Worship, maybe you should change the way you look at things and my mind was I Was so upset But the reason I thought that way was because that offense I'm talking out of that offense Out of being hurt and I held on to that and from many many I'll say weeks many times that I led worship those thoughts would come up. Am I Am I not an effective worship leader? Should I be leading worship when people think like this? I know, you know You know, maybe my songs the song selection was not right and this is happening during the worship. What happens? I have not received healing for these offenses And I am trying to minister to people And you know bring them You know into the presence of God But there's a hindrance Because I have something in my heart so If there is a hindrance Then God will you know It stops the release of what God wants to do through us Right. So whatever that hindrance it could be something very simple, right? It may not be a big offense something very simple Right Oh Or it could be a big thing as well a big hurt a big wound that is just not being not yet healed As to be effective ministers. We cannot carry that offense. We need to let go of it Acts 24 16 says and herein do I exercise myself to have Always a conscience void of offense towards God and towards men Boyant the word boyant means empty a conscience that has An empty or a void of fence towards God and towards men by towards God sometimes you can have an offended God Why God I'm so much better than this other person, but you're using him More than using me You're giving him or her all the opportunities more than me. What happens our offenses with God this time Right and I love what that parable how Jesus Spoke in that beautiful parable. He said He called three workers. I forget the chapter was but he called these three book Workers and he said go and work And he chose three of them at different hours of the day And he gave all three of them the same amount of money And the other two said hey, we've been working more hours, but Jesus the land owner says what you agreed to this amount Do I not have the right to give what I wish to the people who work in my field? I've agreed to this It's powerful. Now sometimes we say we get offended with God So how can you use him? You know, I ensure I spend more time in God's presence. I've got a You know more of the anointing. He can't win prophesy, but I can Or he cannot You know speak in tongues, but no this other person can we get our offense against God? Paul is saying Empty of offense With you and with men Hebrews 12 14 and 15 says follow peace with all men and holiness without which no man shall she the Lord Looking diligently lest any man fail Of the grace of God lest Any root of bitterness Springing up trouble you and thereby many be defined Strong versus Matthew 5 23 and 24 Therefore if you bring any gift to the altar and there remembers That their brother had ought against the Leave there as I give before the altar and go thy way first be reconciled to the brother Then come and offer the gift with that God will be pleased with that offering mark 11 25 26 And when we stand praying forgive if you have ought any against That your father which also in heaven may forgive your trespasses But if you do not forgive forgive neither will your father which is in heaven forgive your trespasses So what is the main? a sense of all these verses Avoid offense avoid living with it By forgive one another Matthew 5 we just read 23 if you have something against your brother and you're coming to the altar Leave that offering there Go be reconciled Whether it is his fault or your fault go be reconciled And then come and offer your offering which will be pleasing at God's eyes We must learn to forgive and forget release people This could be within the family This could be within the church family. This could be between brothers and sisters and christ Um between mentors and disciples whatever We need to Learn to forgive it is a choice that we make To hate someone is to give them too much importance We will renounce all feeling of hate and ill will and resentment Problems 1814 says the spirit of a man will sustain Is infirmity but a wounded spirit who can bear Who can bear wounded spirit? It's very hard. It's very hard to you know sometimes Uh a physical hurt is something that can be we can bear Right, but a hurt in the wound in the spirit or the soul There's no medicine apart from God Right you get a physical hurt. You have all kinds of medicines go to the doctor. They have different options. You Get it, right? I have a fracture or you dislocate your hands or legs It's temporary Right you can go to the doctor. Yes, of course, you may not be People to function as normally as before but you'll be back on your feet But a wounded soul nobody can see it Right it can be inside and at the wound is not dealt to it. It becomes like cancer It can just spread and spread and spread and destroy person's life Right, so if there are things in our lives, just let go of it Let go of it. If you haven't forgiven there are things we haven't people we haven't forgiven for years Years Maybe the Holy Spirit will remind us you say god. I want to be released forgiveness Why because that's what you called me to do and As a leader as a minister of God I want the work of the Holy Spirit to be released more through me and if this is a hindrance I'm willing to let go of it and humble myself and Bring forgiveness to the people who have hurted me right Know and maintain your priorities right know your priorities your ministry your cell group leading your you know your all the planning and preparation With the volunteer teams and everything is very very important But maintain your priorities and life first one your personal work with God maintain it without this personal work with God You know, we can we can work we can do everything but We'll be relying on our own strength Right, we need to rely on the strength of God to family If you're a youth family, right your parents are important. Yes if you have brothers and sisters, they are important to spend time with them family is important because You know as leaders and we've talked about this before as well as leaders many times we get so engaged in the things of ministry that we forget About our wife and about our family Hey, I'm doing God's We're doing God's work, but God has entrusted Your wife or your husband and the children into your hands. It is our responsibility Then we look at a job and our work That God has given us Right, then it's ministry now when you look at these four look at the list Right first is personal work with God second is family Then is the work or whatever we're doing and if you're in full-time ministry, then that's your work Then is ministry so ministry comes forth Interesting, right? Sometimes the ministry is first Ministry is an outflow for personal work with God. What ministry can we do if there is no personal work with God? There's not gonna be not gonna be any fruit Right and fruit is important. Otherwise we keep doing things if you see no fruit Uh, then there's there's no work with no right. So maintain Know and maintain your priorities and as you do this at a young age or maybe even as you're leading cell group or ministering in church Make sure that these priorities are met right Don't ignore your family just because you you know, you want to raise or you want to Be known in the ministry or you're doing God's work doing know the family. Yes, you're doing God's work But God is interested in the family to you So we must look after We must spend time.  Go on. Right. There will be times, right? That seasons that we have And just just go along with it. No maintain your priorities It's some things to consider while you Follow these while we follow these four priorities activity does not equal accomplishment We can keep doing something and that something can be keep we can keep doing it for days and weeks and months And eventually it may go on for a year or so, but it does not mean you've accomplished something Work smart not harder Right. It's just the same. Yes. We work smart and harder, right? Organize or agronize evaluate or stagnate Schedule your priorities Right. Again, priorities is coming here Reacting is not leading Say no to little things Right. And how to say no gracefully If a person gives an idea, you know what in the church we must come we must do this Right. Now you know it's something that you don't want to do in the church or you don't want to do in the cell group Say for example somebody in your cell group says, you know what In the cell group, we must pray in tongues for one hour fully entire one hour sort of Is it sinful? No but You as a leader know that okay, this is something that The cell group is for fellowship. We need to talk to each other here from each other build each other up So how do I say no? You're not saying no to the person But you're saying no to the idea Right and say no gracefully Right. There's a way of saying no Many times, you know And also it can come up when you're having discussions in a cell group. Somebody's going on talking and going off topic Can gracefully say no right Hey, you know If you don't mind can we Go back to our discussion because this is going Of our main topic or there will be times when you know We have to say no And we put our foot down right as leaders. So there's nothing wrong with it like Because sometimes we say no and then we look at the reaction of the person And maybe they don't come back to the cell group and say, okay, you come we'll try it No, if you know, it's not helping those that It's all right Come up with alternatives Uh respond in terms that is in best interest for the person asking Right. So as leaders each one of us, right, it's okay to say no Yeah, because you're not saying no to the person. You're saying no to your idea Right. It's okay to say hey, I'm unavailable as of now You know, we don't take on the position of God. It's it's not like if I say I'm unavailable. Oh god will get so upset with us It's all right If you're if you're on a holiday or you're on leave in your family in at a vacation maybe Just in a different city somebody from church calls you if it's something important you handle it you try to handle it But it's not important.  We can discuss more of this Once I'm back Nothing wrong. They may take it in a different way. Let me say he's not You know, he's not talking. I mean, he's not Discussing this important issue that we have right now. It may be important But right now what is important to for your family is not for you to be on the phone The whole time you're in on vacation because family is also important Right. It's just priority Right. Next point being an example First Timothy chapter 4 and verse 12 right now, this is the most difficult part Let no man despise thy youth But be thou an example of the believers In word and conversation Majority and spirit in faith and a priority The greatest example that we can preach is the life that we live Right all of us can you know Stand by that, right? This is the greatest example the life we live We can speak or preach a thousand messages But our life speaks stronger than that A genuine leader is consistent with what he says and what he does Right as a sign of a genuine leader if he says he's doing it He should be able to do it If he's not been not doing it for some reason he must inform his team Whoever he's working with The reasons as to why he's not able to do it and probably get more time and Ask for you know more time or whatever the reasons could be Right, but a genuine leader is consistent with what he says and what he does Your character is a foundation on which your ministry stands Right Always remember we reproduce our own kind Right We're brave, bold, godly, passionate, strong leaders will raise up the same kind of leaders Leaders who are insecure Always late Always procrastinating Leaders who are always grumbling about things Or they don't want to take risky calls Or leaders who are lazy Raise up leaders that use the same kind of leaders So be an example Be an example If if you are a cell group leader start now, you don't have to wait till you become a pastor You don't have to wait till you're leading 100 200 500 people God gives you the opportunity wonderful but start off With the small so that when you get them big you know your priorities are set Right, so the greatest message you can ever preach is the life you live Right any questions any thoughts? Seems very quiet today any questions Everyone are able to track along. Uh, please let me know if I'm going too fast in any way Should we continue Thumbs up Okay All right, so let's continue Okay, thank you so much. Thank you Okay, let's continue Let's look at the heart Of a leader Right the heart of a leader every one of us Are called to some kind of leadership and We can just be a volunteer team leader in the church these are certain priorities and certain responsibilities That we must have right Being a leader in god's kingdom is not an easy thing Leadership brings great responsibilities. It's very true, right? I'm sure all of us have Tasted leadership and tasted that it comes with responsibilities Right. Let's read this verse first point Leadership involves a new life of servanthood Avoid both the inferiority complex and the superiority complex Because both are Not good for a leader. Now. I remember when I joined In the ministry Very clearly I remember right because I always thought that was Not qualified enough or You know, I always thought of a very shy boy So will I be able to do this? And you know people who are so much better than me And that inferiority complex which I had from from the time I was a little boy It was not something new to me right and everything I had an inferiority complex But even when I joined ministry, I knew I was a believer I knew that god is working in my life, but that Because I have that inferiority complex from the devil's hall I began to work with these, you know, wonderful men and women of god and They were so learned and I always thought oh, why am I good enough?  I was a leader You know leading the cell groups, but I didn't take on the responsibility. I always thought, okay They're all much better than me. They can lead it better themselves So that inferiority complex now I Thank god for granting us granting me the grace to help me to shed off That inferiority complex Let's read john 13 was 3 to 17. Yes. Can any one of us read it? It's here on your I can just zoom it a little bit. I hope you can see it. Yeah Can any one of us read it? John 13 3 to 17 Jesus knowing that the father had given All things into his friend and that he was from god and went to god he he raised it from supper and later set his garment and took the towel and girded himself after that he He poured water into a basin and began to wash the disciples feet And to wipe them with a towel wherewith he was girded Then come it he to simon peter and simon said into him lord Do is thou wash my feet? Um Jesus answered and said to him um What do I what do sorry what I do? No, it's not now, but thou shall know it hereafter Peter said up to him Thou shall never wash my feet Jesus answered and answered him if if I wash That now Thou has it no part with me Simon peter Simon peter said unto him lord not my feet only but also my hands and my head Jesus said to him Uh He bet his watch and even not Save to wash his feet, but but his clean Every width and he are clean but not all for he knew who would betray him Therefore said he Ye are not all clean. So after he washed Their feet and had taken his garment and And was set down again, and he said unto them No, ye what I have done to you He uh, you call me master and lord and he uh, say well for I for so I am For if I then your lord and master have not washed your feet He also ought to wash another's feet for I have given you an example That you you should do as I have done to you very very I say unto you the servant is no greater than his lord Neither is he that is sent to greater than he sent him If you know these things happy are you if you do them? Right. Thank you. Right. So this whole example here Lord Jesus Is talking about how he washed the disciples feet and he's telling them He set the example and he says now you people call me master and lord Now being master and lord I have washed your feet Right. Now you also go and wash other's feet again. What is the principle? Right. The principle is be humble be willing to serve have a servant attitude No matter how big how strong how great you become as a leader Be servant like Like three temptations here to avoid to be self sufficient. Oh you know, especially when we are You know very good in our work. It's very easy to be self sufficient Okay, so for example, you know We've been leading worship for many years and I've used this example before like many years. We've led worship many many places So one hour of worship no problem. I just choose four five songs We can do it. It's not a big deal. Right. We've done it before we can do anytime Bar the temptation is to be self sufficient Self-reliant. Okay. I've done it before I can do it now. Yes. We thank God for the grace for the gift and the talents but I need to depend on the Holy Spirit. I need to depend on God I cannot be Self-sufficient. I cannot say, okay, this is something that I can do on my own No In every area be self-sufficient. It would be even preparing a sermon Preparing for your life group preparing for seven groups raising up leaders. So now you see it's it's something that it's It's my forte It's the word that they use it's something that God has gifted me and I can just do it but remember Don't be self-sufficient to lie on the on the Holy Spirit to Temptation to avoid is to be spectacular Right.  Of course, none of us are like that Uh, but it is sadly happening around us and I'm sure all of us can Uh, you know, give us examples, but there is so much that's happening. This whole celebrity mentality So it's it's not from God God says be self-sufficient. Now doesn't mean that we don't enjoy the things of this world You want to buy we need to buy a house buy the house buy a car. You need the car Right, but we are not Being spectacular big celebrities Buy things Thank God for the blessings that he has blessed us with Right and third one a temptation to avoid is to be powerful to be in charge That is why we focused earlier on on teams Right the moment you build teams You know that it's a teamwork So for example, there's a church with hundred people And you are this leader of the church Right now it's not like it's the law in charge of the church. Yes, you may be making the decisions But without those volunteer teams without this sound and setup team the well media team the You know the ushering team the creators team you can't do all of it You're gonna be doing as a you know, the The pastor may not even know anything about Speakers and how they set up the mixer everything No, there's just the feeling of Being powerful Or being in charge of everything should go we must come to a place of saying You know, hey take on the steam do this do that right and Encourage others even as they do that right servant Christ like Servant leaders Are motivated To by love to serve others They possess a security that allows them to minister to others and Now security is the brief requisite to grade under day now. That's why we You know always emphasize right who we are in Christ is who we really are And we were talking about this in the mentoring hour last week where You know, what about all of these titles that people have you know reverend reverend doctor pastor apostle prophet and all of these titles? Are they important? One of the students asked us and and the point is if we are putting our if you know our Everything that we are on that title If we are walking based on that title Then one day when the title goes we will fall But if we are walking out of our Knowing who we are in Christ are standing as sons and daughters of Christ Whether there is a title or there's no title It won't matter You will be willing to do anything Everything that God has for you. It could be a smallest task to be willing to do They'll say, oh, I'm the pastor. So I will not set the chairs It's the odd church. You have to do it Or I'm the cell group leader and leading all these cell group pastor leading all these circles So I will not call up this calling up is something very media. They should call me up No right only Insecure insecurity will cause us to you know to fail so Be be willing right initiate servant ministry to others receive servant ministry from others teach people about servanthood People do what you see and there are plenty of examples, right people do what we see second aspect to look at is Keep a constant examination on your heart attitudes And we talked about heart attitudes, right? Leadership begins with an attitude. Why I'm doing what I'm doing My attitude here are a few biblical principles about attitude Second My attitude Is as I begin a task will affect its outcome more than anything else. So imagine I'm I'm given a task If my attitude to us this is oh, man, this is not going to be Something that's going to be easy Oh, I wish I didn't have to do this now or all the times right So for example the pastor comes and says, okay, you be in charge of You know, we're going to have a youth concert Christmas or Easter is coming up We're going to have a Easter program Easter concert. So you'll be in charge Two ways to look at it first one. Oh, thank you God for the opportunity. Thank you pastor for the opportunity. I do my best Can you help me? What should I do? I should form teams do this do this Or the second option is Oh, Easter Easter is too close now. It's too late. Uh, how will we form the teams? How will we do this? Can we do it next year? Let me skip it this year or Who will come for the Christmas for the Easter program already this whole Anti-conversion to start and we can look at things So negatively it will affect the outcome of our work So our attitude matters My attitude towards others determine the attitude Determines their attitudes towards me If I think this person is able to do something and you spur them on and you encourage them to do it They will be able to do it. They will give their hundred percent Right, whether they achieve it or not, they will give their hundred percent But if on the flip side if we have an attitude or this guy can't do it Uh, this person cannot do this. It's it's too much of a task for him then That attitude will rub off and maybe if they notice it and they see it That's what's going to happen. My attitude is the major Difference between success and failure My attitude can turn problems into blessings My attitude can give me an uncommon perspective of on life Right, my attitude is either my best friend or my worst enemy My attitude and not my achievement will give me success or happiness So important, right My attitude and not my achievement right achievements are good. It's wonderful right, uh, but the right attitude to you know to Reach those achievements. That is what gives will give us more success more happiness right My attitude will change when I choose to change it Right, I may think of a certain thing or a certain ministry or certain people in us in in a particular way but My attitude will change only when I change to choose it I look at people and think okay, this person cannot do something or he's not capable enough and I keep thinking that I You know, I I may lose out on a good leader in the future I need to make the choice and say okay, go on You know, I need to change my attitude about this person. You may have you know Failed the first time but doesn't mean you'll fail again. Maybe he can do better. Maybe he can You know, just build this ministry much better than what I've been doing Right It's a change that we must make my attitude needs continual adjustment And remember our attitude is contagious How we feel how we express our thoughts our words our emotions is contagious Let's picture this, you know, you get into office on Monday morning and you're So, you know, you're not talking to anyone.   We need to change our attitude right so Have biblical principles Even as you Well, you know continuing ministry check on your heart Always ask the question Lord. Is this right in your presence? Is this right what I'm doing? You know that last Sunday we were talking about this WW JD What would Jesus do? What I'm doing. Is it pleasing in your eyes? Is it right? Does your word say is it right? Am I obedient to God's word? Am I obedient to the Holy Spirit? Am I doing what is pleasing in your eyes? Is this something that please you? Stop ask questions check your heart motive check your attitude And then carry on Right and even as you do this be a dreamer right God does not give dreams and visions to entertain us a god-given Dream begins a god-given task that we must accomplish And a god-given vision Birds a god-given purpose that we must pursue right so of course put a vision in your heart Go ahead. You know just dream dream big dream Bigger things for God if you have a church, which is maybe 100 people dream big things say God In two years. I want to see you know 200 people in the church Think wrong with that. I want to see the ministry teams growing I want to see volunteer teams growing and see hundreds of people sitting here and being ministered to But there's a part that we must do Right, there's a task that we must accomplish. There's a purpose here to pursue it because it's a vision and god-given dreams and visions are usually several times greater than ourselves Because God does not look at our capabilities to accomplish them But what he's doing is he's just looking for yielded vessels Like he's just looking for people who are available. Are you available if you're available? I'll use you but God. I don't have the skills. I didn't ask you about that. I will anoint you Look at Peter. I'm sure Peter would have thought that right? Fishing in the sea of Galilee Like God says the Lord Jesus chooses him that time only he says why are you choosing me? I'm just a fisherman Then he's with Jesus all of a sudden he feels he's the greatest Then he denies Jesus. He goes back to fishing And then Jesus says to him hey Peter Take care of my church You know, look at that Yeah, Peter says I sure Lord And then he agrees to it Peter was not you know, we know from if you see later on in the book of Acts you see isn't this man unschooled Where did he get all this learning from he's a fisherman? He's not talking about the waves and the seas and the different kinds of fish He's talking about the book of Joel and all these Prophecies all these prophets of the Old Testament and he's Comparing that and he's talking so much of wisdom and authority. Which school did he go to? No, I'm schooled fisherman God used them so powerfully And if you look at history, there are so many of them like Not not really great in terms of Anything any capabilities, but they were willing To do something look at the missionaries who went into Africa went into India came into India They may not have been great minds with great intellects, but Just ill at vessels and God used them that will you carry Started all these universities came to India You know the in those years they didn't have Transportation they didn't have trains and you know where they could just travel to one one place to another call each other up Hey, how is the work there? Nothing was there Towns and villages of India came here You know left the luxuries of the other You know of the well developed countries came into India with a vision Started these Bible college centers and this did such an amazing work Amazing work patience 320 powerful powerful was another him Who's able to do exceedingly and abundantly more Above all that we can ask think or imagine according to his power that's working with us. So be a dreamer Think big have a big vision work towards it right Be totally committed Be fully in Right be positive. God has never failed an optimistic attitude energizes and transformed us.  But what did david do run away? No part He encouraged himself Right, he encouraged himself and the lord his god and david did something after he didn't just encourage himself and You know sit there in that place Say waiting for god. No He maintained a positive attitude. He said, okay, we're going to get back. What have we have lost? So he says verse seven and eight and david said to uh Abithya the priest abhimanyak son i pray they bring me hither the effort and uh abitha brought The effort to david and david inquired of the lord saying shall i purse you after the troop Shall i overtake them and he answered him Pursue for thou shall surely overtake them and without fail recover all so to end the story david inquires of the lord god says go You know, he's He was distressed but now his attitude has completely changed Enquires of the lord god says go you will you will recover all of them. So he goes End of the story is he brings all of them back and not when one person had lost their lives He recovered everyone From the from the easter light camp everyone came back Right You and I overcome evil with good. We overcome fear with fate We overcome adversity with tenacity We overcome negativism by being positive Right. Look at what david did. It's a wonderful example for us Always maintain a strong positive spirit Declare what god is saying God has always said he will cause us to triumph There will be seasons of ups and downs failures. We will fall down We will be weak. We will be weary, but he will lift us up He will cause us to trial Now if we allow Things that were designed to harm us You know God Can perfect that right? Satan will remain defeated before us. He's able to take things what the enemy meant for evil He's able to take it and turn it for good Right. So this one this is a wonderful encouragement for us as leaders Even as we are leading teams leading churches leading ministries Our hard attitude counts Our personal life our character all of this really speaks Right So let's stop here We just close in prayer Any thoughts any questions? All right, we can close All right All right, let's let's close in prayer father. We want to thank you For this time. We thank you for your word. We thank you for teaching us lord and even as we continue to lead ministries that we've been assigned and Leading our family leading in the workplace, whatever we are ago help us to Walk with servant like attitudes help us to We walk with the anointing of the holy spirit of god Thank you for all that you've done for each one of us. Bless each and every student Pray god that you will empower strength in an evening and be with each one of them Oh god Even as they learn that the words Word of god be a seed in good ground The bare fruit in each of their lives a god we thank you and praise you in jesus name. We pray amen Amen, thank you everyone. Have a wonderful week ahead. I'll see you next week. God bless


Tom here from Lauren systems and Unify has released their six point zero point two zero unify controller software But stop don't upgrade it And this is one of the reasons that I'd like to do testing and then do some videos about whether or not that testing went Well, and I do appreciate all the people said hey, Tom I kind of wait for you to tell me whether or not I should upgrade my controller because well I don't have the network engineering time to spend Re-engineering network because I have something else to do I completely get it and I'm telling you don't upgrade That's one of the reasons of doing this video. This is very fresh information I guess you could say some of these forum posts are only hours old But I wanted to get the word out there for people who are Feeling as though they should jump on that six point O version of the unified controller So we're gonna say not to upgrade it if that's all you want from the video do that I'm gonna leave all the links on below if you want to continue on further great I'm gonna talk about some of the details and some of the mitigations that appear to be working We're still like I said Solving some of this and waiting for a unified response on it because that would be interesting Maybe they have a patch coming soon. I really really hope they do and if you can click the like button and first If you'd like to learn more about me or my company head over to Lawrence systems calm If you'd like to hire short project, there's a hires button right at the top If you'd like to help keep this channel sponsor-free and thank you to everyone who already has there is a join button here for YouTube and a patreon page your support is greatly appreciated if you're looking for deals or discounts on products and services We offer on this channel check out the affiliate links down below They're in the description of all of our videos including a link to our shirt store We have a wide variety of shirts that we sell and new designs come out well randomly So check back frequently and finally our forums forums dot Lawrence systems comm is where you can have a more in-depth discussion About this video and other tech topics you've seen on this channel now back to our content So over here in the unified forums is the unified network controller six point zero point two zero with look at all these New features improvements and bug fixes. They need another one known issues Vlands beyond UAP Where list down links may not have connectivity creating VPNs and UDME results and commit errors That's the least of your worries here. Let's dive into it. So me and Riley Chase I'll leave the conversation poll up here. We started talking this morning now Riley is the owner of hostify I've reviewed the product before the I would say Riley and the hostify team is the foremost expert on Unify hosting because they do it at scale and I really like their product that also means people that are using them to manage and Can take care of all the updates they do their own testing as well and because of the volume of clients they have they get a lot of insight and They're really recommended don't do it so me and Riley talked this morning And we're gonna talk about a few forum posts now obviously if you just go to the hostify Twitter account They've been tweeting don't do it. I want to help get the word out. Don't do it So we can get that out there So the first one is there's some type of mystery Vlands that get created now This is some type of bug that appears to be Creating an extra VLAN that causes a Issue with connectivity and breaking some of there So there's a few people that have found kind of mitigations for it Just by deleting this VLAN zero that it creates this extra that you didn't create VLAN and by doing that They're able to get things online most people just seem to be rolling back and that's kind of an easy solution The problem is once you roll up to six it pushes out some settings that causes the devices start meshing to each other so there's two forum posts in Reddit that I'll leave links to where people have some mitigations But this one kind of made the most sense to me over in the unify forms leave a link to this one here and This person figured out a way to do this without upgrading So essentially what they decided was going on well Like I said did the time looked at some network engineering and this is a forum post I have not validated this is true, but feels pretty true and people talking about mitigations here That they found that the device the unify access points start meshing to each other Even though they have a wired backhaul So the unify APs have the ability to mesh to each other and like an isolation mode So if you have one that has power, but for some reason loses its backhaul back to the switch It can actually still be working as a mesh unit. It's a great feature that unify has It's apparently broken in version six because if you start meshing something that also has a backhaul You create a loop and then spanning trees should be fixing that not cause a broadcast storm But that isn't happening and that's something that's apparently wrong in the new version So they have some mitigations and fixes for this now for people that are stuck in this broadcast storm This actually is what creates the challenge. So even though, you know how to mitigate the broadcast storm a broadcast storm Well, it's a storm and it breaks the ability for you to log into these devices So sometimes you have to start rebooting switches to kind of stop the storm from happening So you can start getting into the devices or start offlining your access points Obviously, this is extremely disruptive if you have a bunch of users and No one, you know, you it should be something almost transparent where nobody knows what you're doing that way They complain like what do we pay you for?  We have problems type situation You know, sometimes IT is a thankless job occasionally But this is all those times where you will definitely get the user's attention if you push this update in your environment I'm hoping you see this video before then you go wow I know how to mitigate it and you can I don't know and I may play around and do a follow-up video There's some pre mitigation like before the upgrade. Can I just mass disable it? The problem is I need to be able to do this on all the sites we host So that seems kind of tedious and this is we're talking for people like it hostify because Riley does things like looks at the database and flips switches and database at scale as opposed to trying to go through each menu And do it which is obviously really tedious and so there's going to be some You know tooling around that we're going to have to do to look at this for people like myself that want to play with it For people that just go I want this to work. What should I do wait? There's nothing wrong with the old five version controller. It works great. We're not experiencing any trouble with it it's quite stable and Yeah, so I'm not upgrading anything that is mission critical. We do have other lab systems So I may do some more testing with to see you know how the bug progresses and of course once again We'll wait for an update from unify that hopefully just flips those switches or unflips them I guess which way is it going is kind of the curiosity So I'll keep diving into this and I'll leave links to these forum posts and don't worry. There's more people are still As this upgrade comes out and it suggests that you upgrade or people who do crazy things like tell systems to auto update I auto update for security on like my Linux systems because I feel very comfortable with that We do have auto update on Windows systems because I feel comfortable Well, it's comfortable. She could be with Windows, you know, that's a damned if you do damned if you don't you're Insecure if you don't and well Windows updates. That's yeah, it's not like this is updates are unheard of problems in the IT industry They're actually the biggest reason updates don't get done in certain name of any company and you'll find them to be Yeah, the reason that we're always busy in IT why there's an endless amount of work for us So don't upgrade Hopefully you don't have auto upgrade turned on and if you do rolling back seems to be a solution that works But sometimes you may have to do some reprovisioning best of luck I'll leave links to these forum posts and go from there. Thanks And thank you for making it to the end of the video if you like this video Please give it a thumbs up if you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like YouTube to notify you when new videos come out if you'd like to hire us head over to Lawrence systems calm fill out Our contact page and let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums dot Lawrence systems calm Where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you and once again, thanks for watching and see you next time


The question is from Beardfit91. I know you guys previously said that jumping in the sauna right after a workout isn't the best thing, but if time is limited, isn't it better to do it before or after a workout than not at all? Oh, I didn't even know you picked this question. Okay, so there's a little bit of controversy around sauna use post-workout. Yeah, I wouldn't jump in the sauna just so you know. Yeah, thanks, Dad, Joe. So sit down, relax. So here's the deal. There are studies that show, and these are pretty good studies when it comes to endurance athletes, that they took a bunch of cyclists, there were a couple of these, and they had the cyclists train really hard, and they took half of them, and they had them also use a sauna post-workout. And what they found was the cyclists that used the sauna post-workout had far greater improvements in endurance adaptations and performance.  It could be, right? Because it's the same type of study that we would see with professional athletes that ice afterwards, right? There's lots of benefits to, and you see this in the NBA all the time. LeBron James goes out in the fourth quarter, they're basically already won. He's already got ice packs wrapped around his knees to bring down inflammation. He has to turn around and do like the next day. Right, exactly. And so then it makes sense because I care more performance. But if I'm an athlete who is actually trying to build muscle, there's a lot of benefits to the inflammation and allowing the body to go its natural course and repair itself, right? It could be, but I think that that is splitting hairs. I think that, oh, we don't wanna reduce the inflammation from the sauna post-workout because that'll reduce the muscle building signal. I think that's splitting hairs. And there are some studies that suggest that you may actually get stronger and build a little bit more muscle by using a sauna anyway. So the whole thing is about, should I use it post-workout? Now here's a deal. For me personally, I notice benefit to using a sauna post-workout. I really do. If I lift and then I go into the sauna, I seem to improve faster in my lifts and feel better. Now it could be because I tend to push my body towards it's kind of its limit. Do you also think that it has something to do with that you're using our infrared sauna versus a regular traditional sauna? Regular traditional sauna just takes much longer to heat up the body, to get that core temperature up. Because infrareds don't need to be as hot because they raise the core temperature up. There are supposed to be benefits for infrared versus traditional sauna. Again, splitting hairs, they're both awesome. They both, because their traditional sauna is hotter. It doesn't have the infrared that heats you up internally, but it's hotter overall. You're still getting the mitochondria benefits that you get in the infrared? Oh yeah, you're getting all the same benefits. Oh really, I didn't know that. Absolutely. I thought that was the main benefit of the infrared opposed to a regular sauna. Immune boosting benefit is one of my favorites. So you know how the body, when you start to get an infection, what happens?  It's also stimulates some of the white blood cells and stuff that are coming out to fight these infections. So a fever is part of the infection fighting process. Well, a sauna simulates that. So if you feel like you're getting sick, like, oh, I think I might be getting a cold or I might be getting, now I don't recommend going into sauna when you're full on already sick. But if you feel like you're getting sick, sauna use theoretically should help. And studies show that. Studies show that people who use saunas regularly have less infections and less illness as well. It's actually, there was a huge study, a Finnish study that I think there were like 5,000 or 10,000 people in the study. They found that regular sauna use reduced all-cause mortality significantly. I noticed just when we started doing this, what almost four years ago, when we started talking about hot cold therapy, this was back when we were talking, this was way back in early episodes, like 200 something when we were talking about the benefits of like cryotherapy and sauna. And I started to implement that for the first time in my life. So all of my training career, yes, I've used a cold plunge, yes, I've done cryo, yes, I've done sauna, but I've never like done it and treated it like a different system of the body and tried to train it. And for the first time in my life, a few years back when we talked about this, I actually, okay, I'm going to cold plunge and I'm going to train in the sauna at least three times a week every week.  I like, I've always, I've always had a very weak immune system. In fact, I'm very sensitive to getting sick. And somebody I used to say, anytime someone around me is sick, I'm guaranteed one of the people that's going to get it. Where since I started doing that, I've been more resilient to colds than I have my entire life. Yeah, I look at it too. Like I felt, I started to use it post-workout and have felt like better acclimated when I do start to heat up. So I've been playing basketball every now and then still trying to keep that up. And I do a fair bit of cardio, not that much cardio, like mainly just movement. And I'm trying to make sure like I'm, you know, everything is in check with my joints, but like using the sauna after I'll work out, I get those, that same sort of a feeling afterwards where it's just like my whole body gets even that much more exaggerated in terms of like heating it all the way through. And that has kind of had some carry over into energy when I am playing. You don't overheat as easily. Right. Yeah, so now I have also messed around with doing the sauna before a workout. Have you guys tried that yet? No. So sauna before the workout is interesting. I wouldn't think I'd like it. Oh no, no, no. If you want to just get a good, if you want a muscle, if you want to connect to muscle, get a pump, have really good mobility.   Then go into the gym, go do some full range. First of all, you'll notice the range of motion is instantly better just because you did the sauna heated you up. But the pumps you get are just insane. Now I don't know if that's going to lead to, I guess indirectly it could lead to better muscle gain, right? If you're because you're getting a better pump because the blood flow is already improved or whatever. But at the end of the day, you know what? The time that you do your sauna doesn't matter nearly as much as the fact that you're doing it anyway to begin with. So if you're like, hey, I can't do it post workout or I have to do a post workout or I have to, the only time I would say you probably don't want to do too much sauna is right before bed. I think that will probably disrupt your sleep if you get your core temperature up too high and then go straight to bed. Other than that, I don't, it doesn't really matter. Don't worry too much about the time. If you can pick the time, then I would say do it post workout if you're gonna have a really hard workout and you think it may benefit you to have a little faster recovery. Otherwise, who cares? Do it any time of the day. The best I've ever felt is cryo or cold plunge before a workout, a workout then sauna. Oh, so you go boom, boom. That's been my comment. Dude, go hop in a cryo or do a cold plunge. Freezing shower, I've got it. And then, yeah, or a freezing shower and then get into your workout and man, I feel, the adrenaline rush that you get from that gets me into my workout more. I feel so alert and I feel connected for those reasons. It's like hyper focused. And then afterwards sitting in the sauna for me, that's the perfect format. But again, I think it's a great point and we talk about this on the show all the time that these are those things that sometimes annoy me about the fitness community is, we'll argue and debate over all these studies of, oh, this is better. Before the workout, after the workout, 22 minutes. No, 25 minutes. Yeah. Everybody relax. Right, exactly. It's the creatine argument. It's any of these. How is your body specifically responding?  It's like, if you're doing it, you're doing it. That's what matters.


So, today's first talk is rewritingpars.com. I am Abhishek Kona, I'm a software engineer at Parsh, which is a subsidiary of Facebook. I was an ex-flipkart engineer. My Twitter handle is shaky. I've worked on RocksDB, the database engine for a bit, so that's about me. What is this talk about? This talk is roughly organized into three sections. What are the scaling problems we had at Parsh? How we solved them and did we learn anything from it? Are there any principles we kind of found useful? It's roughly organized, it's basically kind of a one-on-one for a back-end team sort of. Let me begin by introducing whatisparsh.com. How many of you have heard of it or used it? It's not bad for a DevOps conference. So what is Parsh.com? Parsh.com is a platform to build mobile apps. The idea is, if you're an app developer, you do not write any back-end code. All your abstractions are present on the client-side code in Android and Java, iOS, in REST, and in JavaScript and PHP. So you just do not manage servers, there's no back-end as such, you start with building an app. So that's Parsh.com, the marketing term for it is back-end as a service, Parsh. So Parsh was acquired by Facebook in 2013 and that's how I ended up working there. So what was Parsh like in 2013? Parsh was one big Ruby on Rails app, like every other startup coming out of Y Combinator. Just one massive monolith. It had 60,000 active apps on it. We had 10 engineers working on the product. The request rate was around 3,000 requests per second. Our deploy times were around 30 to 35 minutes. So that was Parsh around 2013. We've been growing a lot in the last two years. Right now, there are around 500,000 apps built on Parsh. We have 100% year-on-year traffic growth. So our current traffic is around 15,000 requests per second. Our stack is primarily built-in-go. We have replaced most of the Ruby. And that's why this talk is generally about and how that helped. So what were the main issues with Parsh in 2013? The biggest issue was our uptime was around 90%. For any infrastructure service, this is pretty bad. I'll get into reasons why it was later on. Another big problem we had was a single app could take down Parsh. What happens in the app world is an app gets featured on iStore, on Google Play Store, or on Tech Ranch. And a lot of people start installing it. One typical example of that is Yo. It just went crazy in like a day and everyone starts installing it and it takes up all the capacity on your side. If an app is hosted on Parsh, they become really big. They go viral and they take up all your capacity. This is all right for the app because that app is mostly working, but it hurts other apps on the platform. And as an app developer, you really don't want to see this. So we had this issue of tenancy. We could not basically silo apps very well. Our code base was unmanageable. We don't know what was happening. At times we were guessing. This was mainly because the code base was in Ruby and it was dynamically typed and it was hashes all the way down. So we had to guess, yeah, this is a hash here. This is an array here. So yeah, it was basically a sheet here. So what we did is we started by listing down our problems into a list called the beast list. This was a list of 10 to 12 items we put down, mentioning like these are the reasons why our uptime is not 99%. Yeah. And then we went about like, how can we slay each of these beasts? What is a software or a tool we can build? So that this problem goes away. And we went from there. I think a beast list is a great idea because it also helps people to be onboarded. If someone new is joining your team, you can point them at the beast list and be like, yeah, these are the problems we're having. X is working on this, Y is working on that. So you can pick whatever is left. So I think any infrastructure team generally needs to have a beast list of all the problems they have. We kind of use the beast list and it was pretty good. So some of the concrete issues on our beast list where Unicorn or Ruby HTTP server was a resource hog. Has anyone dealt with Unicorn here? Yeah. What do you guys think of it? It's good. Okay. For our case, it was not very good because it was a forking process based server. So it had a upper limit of the number of requests. We could serve a box. As an app scales out, one of the trade-offs we want to do is give up a little bit of latency for higher throughput, which we could not actually do with Unicorn because it had a upper limit of the number of requests you could do on a box. And it was not very efficient compared to modern standards. So we could not just keep adding on nodes because it became very expensive for us. So one of our biggest problems actually was Unicorn. Another big problem we had was our deploy times were very large. 30 to 35 minutes to roll back a bug causes a huge dent in your uptime.  That's pretty bad. So our deploy times were pretty large. Unicorn was a resource of that. Various other problems we had listed down on the list, but these were the two biggest ones. So yeah, we decided to rewrite our stack and go. I know it's not very revolutionary, but we did. There is also a lot of literature against rewrites. I think a lot of people would like to point this block saying that never do this thing is rewrite your stack, but sometimes rewriting is not that bad, especially when you do not understand your current code base. So I'm gonna go with this other block which says when understanding means rewriting. So if you have another block to counter you. Why rewrite? We could not understand the Ruby code base. The estimated performance wins of go over Ruby were fairly large, five to seven times in our dry runs. That's pretty big. And other big reason was our new code base would be statically typed compared to Ruby's maps of maps. So a statically typed programming language is also good for onboarding new team members because when they come in, they can see what an API does. It's easier to read and you can basically figure out stuff, you can extend APIs in a much nicer way in a statically typed languages. Especially in large companies, it's kind of like tribal knowledge that if your software grows, you need to move into a statically typed code base to maintain it. We kind of moved into that phase and we wanted a statically typed programming language. So that's why we decided to rewrite. Why did we pick go? As I said, it's a statically typed programming language. It has great concurrency features or it has working concurrency features. It outperformed Ruby in all our tests, not only in execution times, but in build times, which was very critical for us to shorten our deploy time. Honestly, we didn't want to write Java or C++ because no Java and C++ is hard to do the memory management. Our second biggest choice actually was .NET. It is a very stable language actually. Back then we couldn't run it very easily on Linux. Maybe today we can. Maybe we'd have picked .NET today. But yeah, our second best choice was .NET. C-sharp. Yeah, status of the rewrite. So we were three to four engineers working on it. It took us like one and a half years to complete it. This is the traffic graph, which shows how the rewrite went. We did some experiments in August 2014. You see some traffic spikes. And then we started in earnest in September. And yeah, as in when we launched more endpoints, we bumped more traffic to go and we basically completed it in May of 2015. So that was way more than our estimate, by the way. So how did the rewrite help us? It got rid of Unicorn, which is our biggest problem because SETP stack is great. It is asynchronous. And you can again do the trade-off of sacrificing latency for throughput because the model does not have a upper bound of number of requests you can serve per machine. Our deploy speeds went up because Go's build times are pretty sweet. We could add capacity very quickly. So this helped us to scale out when we had traffic spikes. So this helped us to use the elasticity of Amazon Cloud fairly well. And for now we have a readable code base, which is kind of statically typed and you can look around what's happening. That's for now though. Yeah, so I'm gonna start going into things we kind of figured out on the way. This is where I'll spend a little bit more time. Yeah, so one of the first things we learned is there's no silver bullet. Like rewriting, just rewriting does not help actually. You have to also build tools around your problems. You have to build other software. Just replacing a single binary with a binary in another language, mostly it does not solve a lot of problems. You would have to think about problems in a different way.  How would I have done it differently? That's what a rewrite helps you in. It's more of a thinking exercise. Yeah, and most of the problems would not have an immediate solution. The best thing you can do in those situations is like basically kick the can down and be like, okay, I cannot solve this. What is the minimum I can do so that the client is not disrupted? And then just move on. So yeah, the rewrite actually was not magical, but it helped us think and we kind of built a lot of software around it and that kind of elevated our problems. The first thing we learned is monoliths are all right. Who does not agree with me or do everyone agree with me? Monoliths are all right. So what is a monolith? A monolith is kind of an opposite of a microservice. Microservices are all a rage nowadays. Everyone is building small services. They have a lot of tools to do this. There's GoKit, for example, to allow you to write services.  You can build and test the single binary on a single machine. You can compile it locally. It works great. Development times are much faster than a microservice landscape. Yeah, so we build pass.com mostly as a monolith.  It serves a bunch of different APIs. There's not really that many services we rely on. Yeah, it's partially also inspired by how Facebook does things internally. The Facebook's binary is one big PHP code base compiled into C using hip hop and just deploy it everywhere. They're like different terms for it. It's called cookie cutter deployment, mono repos. So we just one code base, one binary kind of philosophy. Another thing we found out is microservices work great if you have different teams managing each of the services. If you have one single team maintaining the stack, it's just easier to build a monolith. Another thing we found out is proxies. Proxies are a very useful resource. You do not usually assume this, but opening a lot of connections to the database consumes precious memory on the database. And this memory could be better used by the database to cache indexes, cache data, make your queries faster basically. So every time you open a connection to the database, there is a cost to it. That's memory cost on the database. To give a concrete example, our database MongoDB would consume one MB of RAM on every new connection. So at 20,000 connections, this would be like 20 GB of memory, which is like a lot in a 144 GB box. This is a lot of memory which could be used for index caching, yeah. So proxies effectively help manage your connections across app servers. They reduce the number of connections open to the back end and make it more efficient basically. So invest in a proxy fairly early on for your database is what I'd say. Another side effect of having a proxy is you can manage or you can monitor the performance of your database from the proxy. What happened once we deployed this proxy in our code base was we had a single view of how each of our database was performing. What are the most expensive operations? What are the cheap operations? You get all this information because you could watch it from the point of view of the proxy. Yeah, so we wrote our own proxy for Mongo in Go. It's available at that URL. So also modern databases make it very easy to write proxy. Database like Mongo, Rethink, et cetera, have very nicely typed protocols. You can write serializers, desalizers, easier than MySQL for example and write a proxy in two to three weeks. It's kind of easy and Go makes it also performant and the network stack is pretty good. Yeah, proxies are not that hard. Invest in one fairly early. Metrics, I know this comes out a lot but when in doubt measure everything. For example, we measure every incoming request and outgoing request latencies on our stack. We kind of do it automatically using a re-writer. But yeah, measuring everything is useful. We started with using Ganglia. It froze after around 100 graphs. The UI just stops loading. Yeah, so we started using Facebook's internal tools called Scuba and ODS. I'm sorry that they're not really open source. They're like great tools. There's a nice blog post explaining what Scuba can do. This is startup which is trying to replicate it. I hope they succeed. Yeah, so metrics are important. Find a metric service. Hope that you don't have to write one yourself. That's like really tricky and painful. Yeah, another thing which we found very useful in our rewrite was shadowing live traffic. Real bugs kind of show up under live traffic. How many of our integration tests you have? How many of our unit tests you have? A bug shows up in live traffic. So we had a mechanism to track our playback or live traffic on a test cluster. So we did that and we found a lot of bugs and we went about solving it. This was an important tool for our rewrite. We would bring up a new go endpoint, play the same traffic on the Ruby stack, then play it on the go stack and see what were the differences and go fixing bugs around it. So yeah, we have a mechanism to shadow live traffic. It just takes us a config change and we get 1% of the traffic on a new node. This mechanism is kind of built into our stack now. We find it really useful. Yeah, so the setup is fairly simple. We have a go HTTP proxy which sends requests to both test and product clusters, measures the delta, puts it in a log stream. We see what's happening and yeah. It works great for read APIs. To do this for write APIs is kind of tricky but we also have a solution for that. It is not great but we just take a snapshot of a database. We siloed two versions of the code base and then we run the load against it and then we compare the database deltas and see what's happening. So that works but that's fairly complicated and like a lot of scripting and it's kind of breaky so yeah. But we do have a mechanism for that. One of the interesting bugs we found using this was the go time library truncates time to a single zero if it has three zeros. So if you have a time ending in zero, zero, zero it truncates it in zero and the Ruby stack would return zero, zero, zero. The Android SDK would crash on not finding three zeros. This was, we never knew what was happening. We just saw 1% error rate every day and we started shadowing traffic and we're like okay, this is the difference. They're like no two zeros here. So yeah, those kind of bugs are hard to find out. I think shadowing traffic is the only way to go about it. Throttles, one of the things we also did in the second version of the software or in the go version was we built throttles throughout. So we have the capability to throttle any client or any backend instantaneously. So if you have an app which is going crazy the first thing we do is okay, we're gonna throttle them before we figure out a nicer permanent solution for them.  Please try again in X seconds instead of 500s or there was a server crash. So throttling works in both ways. And then yeah, so throttling is important. We have simple memcache based counters for our throttling service. We tried ZooKeeper here. It was pretty complicated. So memcache is simpler and it works for us. We have at this point evolved this into auto throttling service. So there's a auto throttling demon which monitors request rates. If there's a spike, it throttles and then notifies the ops person saying that hey, this app has kind of gone crazy. Can you look at it instead of an ops person coming in and doing it manually? So yeah, throttles are great. Always build throttles for all your network operations. Again, if you could rewrite your client libraries to do this, it'd be great. If you have, for example, a thrift library, you can build it into it. That works great. In our case, we have it on a Mongo code base automatically but in other places we have kind of manually added it. So yeah, auto throttling is great. Invest in it. Gatekeepers and deciders. Gatekeepers or deciders are feature flags of production hooks to control rollout of new code to a fraction of users or traffic. This is a good way to gain confidence when you're rolling out controversial changes. Roll it out, deploy the code, make sure that it's only rolled out to 0% of the traffic and then keep bumping it up, observe your graphs, see what's happening and then gain confidence and roll it out. Yeah, so deciders are great. We have our own in-house system again. It's written and go up. It's based off Redis. It's called Decider. The Facebook one is called Gatekeeper. It's based off ZooKeeper, things like that. Another thing we found was Decider code or Gatekeeper code tries to pile up very quickly. When you start launching features in this, you tend to have like four or five things going out together. You don't know what's happening and then the code base becomes this huge mess of if and else statements. You're like, oh yeah. And you cannot basically follow what's happening while you're reading it because you don't know what the decider value is at that point of time, et cetera, et cetera. So the most important thing to do is actually to clean up your decider code once you have rolled it out to 100%. I've seen a lot of people not do it. I think it's a fairly important thing to do. We have a linter in place which kind of checks after one week saying that, hey, this Gatekeeper has expired or this Gatekeeper must have rolled it out so probably you should get rid of this code. Yeah, so keeping your code base healthy in the land of Gatekeepers is kind of important. People do tend to miss that out. Deploys, yeah. So as I said earlier, our biggest problem was deploys. We had 35 minute deploys. We kind of wanted to bring it down. Also, we had fixed release schedule before. It was Monday and then we'd have to talk to the ops engineer or the release engineer to get a deploy later on through the week. That was kind of bad because we would always be blocked on one person when there was a critical fix to go out. You'd have to wake someone up at night and be like, hey, I need to deploy this, this buggy. So yeah, so we moved away from that where we wanted to get rid of the human in the deploy process. So our philosophy is every engineer should be allowed to deploy at any point of time. So it's the engineer's responsibility to figure out if this deploy is safe and if it did roll out very well. We have a lot of graphs to give you a very good high-level overview of this, but it is the engineer who deploys. We also try to deploy as many small changes as often as possible. On my flight here, I deployed six times because I had like six dibs going out and I had to figure it out what's happening. So yeah, deploy as often as possible. The marketing name for this is continuous delivery. Yeah, I just call it deploy as often. Yeah, so we have a tool again, it's called deploy CTL. It's written in Python, it compiles Go code, makes static binaries, ships it to all the machines. Yeah, it works very well. This is like probably one of our only critical services which depends on zookeeper. Yeah, another important feature I found in our tool was deploy locking. Deploy locking is the ability for an engineer to lock deploys for a particular binary. If you are doing some risky changes, you do not want someone else to deploy the code base you're kinda in flux with. So you should be able to lock deploys. We have a nifty feature where we can put in a message saying that I'm locking this for X, it goes out, nobody can deploy that binary till I unlock it. So deploy locking is good. The ability to canary your software is also pretty good. You should be able to roll out only one binary and see how that performs with the rest of the binaries. Is there any difference? So yeah, those features are built into our tool and it works well for us. So I think those are important concepts. Another thing is cockpit. A cockpit is an admin service which runs on every binary. In our case, it's an admin STDP service. We use this for debugging. It exposes health checks. It exposes git versions, build times, the uptime the server has been running for GC stats. We also run it at a thread pool with a higher priority than the server thread pool or the main service thread pool so that this is always available even if there is a lot of load on the regular service. So we can debug and see what's happening. Because we have mostly written it in Go, we can connect Pprof over it. We didn't do much here. It's just available in Go. We just like build some glue around it. Yeah, so that works very neat. Another thing we use cockpit for is to activate verbose logging. What we can do in our stack is go to one server, log into via cockpit, enable logging for that app. And then you can see all the incoming requests and response bodies dumped on log file. This is again important when you want to debug a particular app. You see, okay, what's happening with this app? Why is it crashing so much? Or you just enable debug logging for that app in a regular case for that client and then you get all the stack and you can go from there. So this is, also you can change logging verbosity levels using cockpit. In our case, we found it fairly useful. When you want to figure out what's happening, you set the logging level to debug without actually restarting the binary. So yeah, cockpit is kind of useful. You can put in a lot of stuff around it. Yeah, I think every binary should in generally have a cockpit. Context, a context is an object which you basically tag along throughout your stack. This is an object you can add additional information to and it is available in any part of the stack, basically. So in our case, we use a context and we tag along the request ID and the app ID together. So you can find out who did this request and take a decision based on that way down in the stack, not at the API level, maybe at the Mongo level and see what's happening. The App Engine people use the context concept to actually introduce billing. So they pass along a context and they decide, okay, I'm gonna bill X because this request has already done these things. So yeah, context are an important tool we found. One of the nice things we did with the context was actually have a request ID tied along on the context object, pass it down to our Mongo driver, add it in a comment in the Mongo query so that it ends up on the slow query log on Mongo with the request ID tagged. You use this request ID to figure out which app is the culprit. And then you can basically get in touch with the app, you can do some more debugging with it, but this is really neat. If we pass it on a context, we could see in our slow query log what is happening and which is the culprit app. So I'd say, yeah, context are super useful. Context objects are a great idea if you're writing a new library. Please allow for them. Golang has a great context package. It's called Golang.org, yeah, it's pretty neat. I think owning your database. So when your app gets successful, the biggest problem usually is the database or you're probably doing something wrong. Databases are hard and databases have different trade-offs. So it helps if you can understand the internals of your database from the beginning. Databases, modern databases have a lot of concepts like query planners, DB caches, indexing trade-offs. You can also, there are a lot of major locks in them and each of them have a different tuning parameter. So it is important to understand how these tuning parameters kind of affect your app and your workload and tune them. Every database comes with really bad config out of the box. So it's very important to find a good config for your database. One nice way to do it is actually spending some time on the internals of the database, reading the code, what's happening. Again, modern databases are not MySQL. They have much more readable code. They're not, so you can figure out what's happening. We started that, basically, we started hacking on Mongo. We started adding metrics around the major locks, what is happening, and then we found out that it is actually not that difficult to add a new storage engine to Mongo. So we started this project called parse plus rocks. So this is a new storage engine for Mongo. Three people built out in six months. RocksDB based storage engine for Mongo. We just announced it, I think, day before yesterday. This works great. We could only get there because we already had a good grasp of how our database code, database code base was arranged, what are the layouts, what are the major locks, et cetera. So investing in your poking around the internals of your database is very useful. Keep doing it, do not shy away from it. I'm gonna talk a little bit about our code base. We use dependency injection a lot, but we use it only at boot time. There's no runtime dependency injection. So while we are booting up a service, we just run the dependency injection graph once, and that's it. You cannot run it again, or else our binary kind of panics. So dependency injection is great, but if you only use it at boot time and don't do magical things at runtime, it also helps you boot up test instances very quickly. Organizing your graph for the tests is fairly painful, and if you have dependency injection, it is easier. We write a lot of smaller libraries instead of building one complex library sort of thing. So we have, I think, 50 libraries at github.com slash Facebook Go. A lot of them are used outside of Facebook too, so it's pretty sweet. We try not to fork any software. We always try to submit patches upstream.  Some of us are committed on the Mongo database. So it's very hard to maintain multiple forks of the code base. We had like pretty bad experiences with this in Facebook land where MySQL is forked, then it was sent upstream, and then it's kind of at a messy state right now. We do not want to get into that state with the software we use. So we always submit our patches upstream and we make sure that they're accepted or work around with it. Our tests, yeah. Tests are one of the important things which made our rewrite very easy. We could always write good tests and figure out, okay, every time we found a bug, we wrote an integration test for it. And then that is how we kind of gained confidence. Our test suite is basically our spec. If we have a behavior argument, we go look in at the test. I'm like, this test was written because of this, so that might be the right behavior. One of the things we found was, yeah, integration tests are greater than unit tests. Unit tests are nice to write, easy to write at times, but integration tests are the real deal. You kind of run like sample from your live traffic. You put an integration test and it works better. We are very proud of our test run times. They're under like two minutes for a code base of around 200,000 lines. We use Go, it has parallel test suite built in and we also arranged our code base in such a way that you can run it parallely. There's no every, there's no like global lock sort of thing. So yeah, parallel test runs are a great idea. I've not seen a lot of languages do it. I think that's, you should always try to get your tests to run under like a few minutes so that you can try it out locally. Other, another neat thing we do is we actually boot multiple Mongo and Memcache instances in memory while we are writing, running our tests. So every test gets an individual Mongo or a Memcache instance booted in memory and it's basically a fresh start so there's no old data on it. You can start, yeah, so this just quickens our test times. We have two packages which allow you to do that in Go. It's a Facebook Go MGO test and Facebook Go MC test which is Memcache test. So at one point of time we had 30 Mongo DBs booted up by a single binary to run our tests. So in the two minutes we boot 30 Mongo instances, run all our tests, clean up and die. So that works pretty well. And I think this is again with modern databases being very lightweight, good configurable options. You can run them in like tens of MBs of RAM. So I think that's a, that's something you can use. I've not seen like a lot of frameworks and tools do it. So I think I'm just putting it out there. Yeah, so I think I'm at the end of my talk. This is like a lot of gyan I guess, but yeah. The closing thoughts is, rewriting software is not the worst idea if you invent some parts of the stack, if you look at what you can do better. Blind copying of code sometimes is not just the thing. You have to be smarter about what you're rewriting. You have to re-architect things. Go is a great programming language. I'm a big fan. I use it at the next startup I'm doing or the next company. It's got a great standard library. You do not depend on that many external libraries, which is good. Yeah, so Go is great. Thanks Google for it. And yeah, have you guys used Pass.com? If you're not, you should. It's a great platform for building your app. Like a bunch of apps already using it, 500,000. So, usepass.com. Yeah, that's it. I think I'm gonna leave it up for some questions. Hey, great talk. Thanks. So, I had a couple of questions actually. One is, did you consider OCaml as an alternative when you were choosing languages? It's also statically typed pretty fast. The second question is, you talked about microservices. So, would it have been perhaps easier for you to do the rewrite if you had smaller services that you could replace one at a time? Okay, we didn't try OCaml. I don't know why, because nobody mentioned it, I guess. That was the, like, yeah. So, I guess nobody had written any OCaml ever in my team. So, we didn't, we did talk about Erlang. We tried Erlang, and it was like, again, dynamically typed. So, we gave up on it. Yeah, no idea why we didn't try OCaml. About the second question. I think if it was microservices, we probably wouldn't have to rewrite all of it. We would have to rewrite only the ones which were problematic. And that would have been cooler. But I don't think any successful startup, any successful company, is not looking to build microservices at the beginning.  So, Parse was in the running for one year, and they just wanted to get an app. And I think they were very successful at it. They got 60,000 users. And that's what they were looking for, and they built it as a monolith. So, yeah, microservices, if done from the beginning, are good. But, yeah, one of the biggest problems I have with microservices personally is testing them. When you have to build them and test them on your local box, you have to boot up like three or four services. You probably need some sort of a service discovery thing, locally running, if you don't want to keep entering port numbers and stuff like that. So, I think monoliths get rid of that problem. I've seen some newer tools with Docker which take away this pain mostly, but I think monoliths are still simpler. You can just have one binary poke at it, see what's happening. Hey, it's a good talk, so I just have a question. Hi, thanks for sharing your knowledge. Hey, here then. So, just could you share how you were able to replay the live traffic for your testing purpose? What is the solution? So, we had a HTTP proxy. Our pass.com is, at the end of it, a REST API for the client SDKs. So, we had HTTP proxy, which would, it had two modes, it had log mode and it had live mode. The log mode would basically record the incoming bytes and the outgoing bytes for every request and put it in some sort of a log file. And then, we had another tool called flashback which would read this log file and play it on another server, basically. So, that was one mode. And then, the flashback would also basically measure deltas and then produce nice colored output. You would watch on a screen and you're like, yeah, there are these many errors, this is the error, you go about it. We also had to write a library to do nice JSON diffs. It's called, and remember, it's called spewediff, spewedump, something, it's there on the Facebook Go. So, yeah, we did this. This was the basic idea of replaying live traffic. The other mode we had was live mode where we would have two Go routines spawned up, send the request to both the Ruby and the Go stack, getback results. If they were fine, if they were the same, just return it, don't do anything. If there was a delta, return the Ruby version because that is still the correct version. And then, log the difference again into a log file on the server. It was not just a log file, internally, we have this concept of a scribe stream, which is basically a log stream. I guess, Kinesis is the Amazon equivalent of it. So, we'd put in something like Kinesis, it would go down to an aggregator and we'd have an early job which would be like, hey, these are your deltas in the last one hour. Your deltas are super high in the last one hour, so there's something wrong probably. Things like that is what we built around it. But the crux of it was actually finding out the deltas. Thank you. Hello. Just have a question that you have different stacks and different applications running, right? And different services. So, what was your main base that, this is the programming language that I have to take? And in fact, this is my first question. And then the second one is that, how do you come up with an errors, how do you handle it? When there are like 60k apps running and then you talk to different protocols and then how do you get it, the events of this particular things and you do a call back of that and do you note somewhere where these errors go and then you automatically rectify with some of the techniques where you use it? Yeah, actually I did not speak about that. We have an interesting part of the stack there too. We did not build it for errors. I'm gonna talk a little bit about the errors. So, for errors we have this system called log view. I guess the open source equivalent is discusses century if you've used it. So, what log view does is we track every non 400, non 200 response on the server to this stream or this machine, this service. And this would aggregate all the similar ones into one collapsible UI and we'd see okay, we have a spike of this, we have a spike of this and that's how we would track errors. Another important thing we did in this system was compare errors with past history which we found was very interesting. If the error rate was much higher than last week, same time, there's something wrong because at times there would be errors and you cannot go looking at every single error. So, we'd go look only if there was a delta in a major delta between errors this time and errors last week this time kind of. So, that was very important. I think our system allowed us to do it, yeah. So, that's how we did errors. So, man, how does you manage the threading and when you're writing that in Go, like through Grow, GoRoutines, how you handled it? We try not to spawn any GoRoutine in a request. Kinda, that's the idea. We try to keep it dumb. I mean, the HTTP stack underneath does GoRoutines. We just try to rely on it. We're like, this is a request. We're gonna think this is a blocking. That's the, I think that's generally the beauty of Go. Do like, write code as it is blocking. So, I'm gonna do this, I'm gonna do this, I'm gonna do this. I'm gonna, and then respond. And we just generally wrote code like that. We had libraries which would do things asynchronously like metrics, like reporting logs. Our billing was asynchronous. All those were generally library components and in our request code, you would not see that they were concurrent. You would just feel that they were like regular calls and then there'd be some buffering in the background and that happened in that individual library. So, that's how we kinda like organized our code. We have a neat little library called muster which we wrote for generic background processing jobs kinda thing where you would like put in a bunch of things, batch them together and execute at once please. So, if you had to write to a log service like scribe, you don't wanna write every time, you don't wanna write one request every time. So, we would put it in a muster. A muster is a group of 50 objects and then we'd send over the muster to scribe. That's how we generally organized it. But our request code would look kinda like dumb. It'd be like, do this, do this, do this, do this, and do this, yeah. Is muster is open source. Hi, I'm here. Yeah, sorry, I joined late. So, pardon me if you have already answered my question in your presentation. Just wanted to know what engineering process do you follow? I mean, you're doing tests, you're doing, all these things you do, keep doing experiments and on how do you prioritize? How's the roadmap laid out? So, my boss calls it cowboy engineering. That's what we generally do, I guess. So, there's no answer to it. We have a goal for six months. We try to hit our goal for six months. But apart from that, it's kinda like, depends on what your day-to-day problems are. So, parse personally has operated in two modes. The earlier first six months was the house is on fire. How do we like fix it? And that would be very chaotic. You'd be like, okay, I'm doing this thing for the next one week. I'm figuring out what's happening. I need to put in this bandaid so that we don't break when this big app is gonna go on Japanese TV shows. So, things like that is what was the first stage. And the second stage was, we need to kind of build out products now. And how do we go about it? That's the stage we are in currently. So, how we prioritize that is kind of, I don't know, we don't have a process of what we are doing. But one of the things I think we do differently is we have a timeout on discussions and meetings. We're like, okay, we're gonna discuss this for four days. And at the end of it, we're gonna do something. It doesn't matter what it is, but it might be wrong. But there's gonna be a timeout at the end of four days. We're gonna figure out, okay, we're gonna write this new API because I felt like it. And then that could be the reason. And then we just go do it. And we try to minimize the time to do that. Like, okay, we're gonna do this for two weeks, launch it out and see how that goes. And if we can get some information around it, do that. So, an example is we launched a slow query tool recently. It took, I think, three weeks from concept to roll out. We had a lot of planning around it. The UI needed to look shiny and neat and stuff. But we launched it out. We got 2,000 page views a day. And it doesn't really work, so we just gave it up. Kinda. So that's what we follow. Another thing we do at Parse differently, I guess, from Facebook, the rest of the company is, because we are a developer platform, we focus on not rolling back stuff as such. Like, we do not deprecate anything. If a product is not doing very well, we just leave it at that stage. We don't try to kill it. Which I think Facebook does a lot, is like, kill a lot of products. So, because we are a developer platform, we give a little more respect to that. We don't want developers to, developers are using that to be paced. So, yeah, those are the only two considerations we have. I don't know if this answers your questions. Kinda generic. Yeah, we have, we've faced a few issues. One of the issues, I think it was in Go 1, 2, was the stack size was fixed. So, when we were writing out our push service, we would open, like, a bunch of connections to the Apple push service. It's called APNS. And it was consuming, I guess, four megabytes per connection. And this was okay for most of the applications, but for our application, it was kinda not doing very well, because it was a very dumb application just opening a lot of connections. So, we kinda found that irritating. We actually compiled, re-compiled Golang with a lower two megabyte, it was a constant in the code base. So, we changed that constant to two megabytes. We re-compiled Golang, we called it our own version, we deployed our own binary. Kinda worked out well, but I think in 1.5, 1.5, 1.4, they introduced a splitable stack sizes that kinda made our problem go away. But this is one of the problems we found out initially. We found out a few bugs in the HTTP library. We've started discussions, we've got some of them fixed, some of them are not fixed. Go is opinionated at times kinda stupidly. A nil in a JSON gets an empty string representation when you're printing it out in JSON. It caused a bug for us and we didn't realize what was happening out for a long time. Then we're like, oh, in Go, you gotta allocate it to a size zero array, or else it's gonna do weird things. So they're like tiny, tiny things like that, but overall, we're generally happy. So I think you said that the quota-based system in a multi-tenant kind of a setup was managed by the memcache counters. Was it on a per-app basis? Because if it is on a per-app basis, then some app is underutilized, but it has a high memcache counter. The other one cannot really use up the resources. And the second one was, you talked about team the production traffic to your test cluster as well. But then how do you, that's a big traffic. Did you have a test cluster which was equivalent to the production cluster in terms of size? No, we ran only a percentage of the traffic all the time. We didn't run the whole traffic. That was kinda like, or we could run one app's traffic, that is what we would do. We could set up a cluster which could take one app. We had the concept of siloed app clusters. So we would be like, okay, bring one more, there's an app called Shorts. Bring one more Shorts cluster up. And then Shorts versus Shorts, what we would do, things like that. We could do that. The first question is about multi-tenancy, co-tenancy, it depends on your business model. It depends on what you're offering to developers. Amazon offers this as a service. We generally do not offer elasticity as a thing. We're like, we're not elastic, we are plastic. We are like, we'll be like, we have an upper request limit of this much. You got to decide what you're gonna do. Every app has an upper request limit, which is like 30 requests per second. That's the minimum we do. And then you can go up from there like thousands of requests per second. As soon as you hit that, I'm sorry, you can't like get more than that until you pay for it or do something else. So yeah, we are not elastic in that sense. So we do not really have this problem where it'll be like, hey, we have some free resources, take it, we have some free resources, sucks for you. I guess this is the last question. We have a slow rollout process in the sense that it depends on the size of the cluster at that point, but there are n nodes which go out at one point at a time. N, n, and then that's how it goes. We have a batch of n going out each time, and then it just increases. There's no, it's linear, so the n is fixed. We don't do n to n and stuff like that. So that's fairly simple, it works for us. There's no math just because it was easy to code.