The title of our sermon this morning is Prepared and empowered to preach prepared and empowered to preach and we are in Part 2 of this little sermon series in John chapter 16 from verses 1 through 15 As we come to our text again We began this text last week and took a look at the first few verses here in John chapter 16 But as we come again to this text John chapter 16 verses 1 through 15 Again to remind us of our context the Lord Jesus Christ is walking toward Gethsemane With his disciples. It's the eve of his crucifixion. The Lord is about to be killed murdered He's about to be arrested in the garden tried mock trials He's gonna be scourged and he's gonna be hung on the cross and murdered by lawless hands according to the foreordained purpose of God and he's going to depart by means of the his Cross depart by means of the crucifixion to the father and going to depart his disciples now as the Lord Departs to the father as he leaves his disciples these men are gonna face the Undiluted hatred the undiluted hostility of this world as We do in the Lord's absence when we preach the gospel We face the hatred and hostility of a dark and dying world First here in this text from verses 1 to 15 Lord Jesus Christ wants to warn them of this He wants to warn them of what's coming and that warning serves Two purposes first purpose is that it prepares them For the work of the ministry that they've been called to to be forewarned is to be for armed So to speak He's preparing them for the work that he has called them to but second purpose that these warnings serve is to build their faith in him He says when those things happen to you remember the words that I spoke to you in Remembering those words and remembering the Lord's gracious warning of the persecution that's coming It has the effect of building their faith and trust in him Now secondly in this text he wants to reassure them first. He wants to warn them Secondly, he wants to reassure them and he's gonna reassure them with the promised coming of the Holy Spirit First the Holy Spirit's work in the world and we'll see that today as we work through this text and secondly The Holy Spirit's work in the church and we'll see that next week as we conclude John chapter 16 verses 1 through 15 The Lord's warnings here these warnings show gracious care And concern he loves them right he loves them having loved his own who are in the world He loves them all the way to the end and he warns them here out of loving concern It's a part of his his loving kindness toward them to warn them of what's coming He loves them his parting instruction here These last words before he departs from them Show his gracious care and his concern for them his parting instruction demonstrates that love now Several times beginning in John chapter 15 now as we go into John chapter 16 Several times he warns them of the world's hatred He warns them of the persecution that's coming the persecution they're gonna face And he talks to them in John 15 into John 16 like friends not like slaves But he's giving them inside information so to speak he wants to reassure them He wants to help them face what they're going to face. He's talking to them Like a man speaks to his friend not like slaves And we see the purpose of that loving concern given to us in verse 1 He says these things I have spoken to you that you should not be made to stumble so that you shouldn't fall away That's the concern of the Lord here and speaking to his disciples and warning them in instructing them He wants them to persevere in the faith He doesn't want the persecution that's coming the Hatred and hostility of the world He doesn't want trials and difficulty in adversity to cause them to fall away to cause them to stumble Now we see that same loving concern for the disciples as the Lord prays for them in John chapter 17 Flip the page and look at John chapter 17 and drop down with me to verse 6 The Lord has this concern for them that they are kept by the power of God that they Persevere in the faith right that they hang in there. They stay with it. They don't fall away. Look at verse 6 The Lord is praying here. He's praying and he says I have manifested your name to the men whom you have given me out of the world They were yours You gave them to me and they have kept your word Now they have known that all things which you have given me are from you For I have given them the words which you have given me and they have received them And have known surely that I came forth from you and they have believed that you sent me He says in verse 9 I pray for them. I Do not pray for the world But I pray for those whom you have given me for they are yours and all mine are yours and yours are mine And I am glorified in them verse 11. He's departing right now I am no longer in the world But these are in the world and I come to you holy father Keep through your name those whom you have given me that they may be one as we are His concern is that they persevere in the faith God the Father keep them. I'm departing keep them He says in verse 12 while I was with them in the world. I kept them in your name those whom you gave me I have kept and none of them is lost except the son of perdition that the scripture might be fulfilled But now I come to you and these things I speak in the world that they may have my joy fulfilled in themselves I have given them your word and the world has hated them because they are not of the world just as I am not of the world I do not pray that you should take them out of the world But that you should keep them from the evil one see the Lord's concern here Right and his prayers in his prayer for his disciples that God the Father should keep them They are not of the world verse 16 just as I am not of the world Sanctify them by your truth your word is truth set them apart to you God, right? As you sent me into the world. I also have sent them into the world and for their sakes I sanctify myself that they also may be sanctified by the truth and we see that loving concern for the disciples We also see that loving concern for you and I don't we The same loving concern the same concern of the Lord is that when we face Difficulty when we face adversity no matter what shape it comes in specifically here. It's persecution Persecution from a hateful hate filled hostile world and we face that kind of difficulty. We're to persevere in the faith We're to trust Christ and press on He is faithful. He's faithful We're to remember him and press on in the work, but whatever adversity whatever difficulty you face whatever it is Trust Christ press on He has that same loving concern for us look at John chapter 17 and look at verse 20 Here he prays for you and I For the same thing for which he prayed for them. He says in verse 20 I do not pray for these alone, but also for those who will believe in me through their word That's prayer for me That's prayer for you Lord Jesus Christ praying for you praying for me We would persevere praying for us that they verse 21 all may be one as you father are in me And I knew that they also may be one in us that the world may believe that you sent me and the glory Which you gave me I have given them that they may be one just as we are one I in them You and me they may be made perfect in one and that the world may know that you have sent me and have loved them As you have loved me what a glorious blessing right now the Lord Jesus Christ praying for you praying for me Saw this this last week just charged me up Robert Murray machine Said this if I could hear Christ praying for me in the next room. I Would not fear a million enemies Yet distance makes no difference. He is praying for me. It's back in John 16 the Lord Graciously warns them. He graciously warns them. He warns them with a purpose in verse 1 They should not be made to stumble so they wouldn't fall away They warns them again here in verse 2 More specific this time he says in verse 2 they're gonna put you out of the synagogues Yes, the time is coming that whoever kills you Will think that he offers God service These things are gonna do to you because they have not known the father nor me I explained that again In verse 4 these things verse 4 I've told you that when the time comes you may remember that I told you of them So when we face persecution when we face the hostility the hatred of this world for our faith in Christ We are to remember we're to remember keep in mind Those things which the Lord has told us and remembering his words, right? Remembering his words to us is a means by which he holds us He keeps us. He preserves us. It's a means by which we keep from falling away Now the things the things he told us here verse 1 verse 4 The things he's told us aren't restricted to his warnings Not restricted to his warnings He's given us both warnings to heed and he's given us promises to hope in right Promises to trust in he's given us doctrine. He's given us reproof correction Instruction in righteousness. He's given us encouragement. He's given us comfort and he's given us warnings and he's given us promises all of his word all of his word is given to sustain us To cause us to persevere Remembering his word is to keep us from falling away You want to Persevere in the faith. Do you want God? By his spirit according to his power. Do you want God to preserve you in the faith? Then take heed according to his word Remember the Lord's words to you Remembering his word keeps us from falling away the grace of his word like a fetter is to bind our wandering hearts to him Our sinful hearts are prone to wonder aren't they all the time all the time So our responsibility our responsibility is to remember what he told us and Trusting him we do what he's told us And that is one means by which he keeps us from stumbling one means by which he Preserves us in the faith the Lord's concern in John chapter 16 verses 1 through 15 is that we persevere in the faith These things I have spoken to you verse 1 that you should not be made to stumble now someone might say right You might foresee the obvious objection based on teaching of the modern church today Someone might say I'm a Christian right. I'm a Christian. I can never stumble I can never fall away I will never lose my salvation. We hear that Let's you and I for a moment Consider that objection consider that thought and remember the Lord's words here and Heed his warning. I want you to follow me now with the Lord's train of thought And this is important. This is something we have to constantly Remind ourselves of one because our own hearts are deceitful our own hearts are self-justifying Deceitful above all things right we have to remind ourselves of these things because our own hearts can deceive us We also have to constantly remind ourselves of these things because the world is teaching a lie The modern professing church Often teaches a lie when it comes to these truths Remember the Lord's words Consider the context listen what the Lord is saying here in John chapter 16 the 11 disciples following the Lord right now The Garden of Gethsemane here, but the 11 disciples are saved men. They're saved men He said to them the Lord did in John 15 3 you are already clean because of the word which I've spoken to you These 11 men minus Judas who's off on his traitors errand right now, right? The other 11 are saved men and yet the Lord's primary concern for them in John chapter 16 verse 1 is that these saved men Right would not stumble or fall away in the face of persecution He tells them that when persecution comes they must remember his words He even prays for them right prays for them and ask the father to keep them in John chapter 17 verse 11 Now a modern-day evangelical right a modern-day Typical professing churchgoer professing Christian walking along with them along this path to the Garden of Gethsemane might interrupt And might say to the Lord might even rebuke the Lord Jesus Christ Lord. Why in the world are you? Giving me these warnings. I can never lose my salvation right once saved always saved Why are you warning me like this? Why are you talking to me as if I can lose my salvation? By Satan's design here There is so much Damning error that comes stuffed within the skin of truth And it's so that's what makes it so deceitful so insidious is that it comes Wrapped in the skin of truth, but that's skin of truth stuffed with a lie and our deceitful self-justifying hearts Are only too willing to suck it now now. Here's the here's the damning error. All right. Here's the damning error so many Treat salvation like a vaccination You receive your shot When you are 6 or 12 26 36 right and after you receive your shot You don't have to do anything.  Just keep remembering the shot that you got There's no need to sell out for Jesus. No need to get radical No need to strive to enter by the narrow gate. The way is not difficult which leads to life No need to exercise yourself toward godliness. No need to be concerned with pursuing holiness No need to be diligent to make your calling an election sure no need to labor Striving according to his working in you. No need to work out your salvation with fear and trembling No need to strive for obedience to him in all things. Just remember your shot Just remember that vaccination you got when you were three Here's the biblical truth right here's here's with the Bible to you recognize all of those are from scripture, right? Here's the biblical truth There's a sense in which our salvation is like a vaccination There's a sense in which our salvation is like a vaccination You got a shot of new life in Christ when you were born again from being dead in trespasses and sins, right? You are declared innocent and you got the shot of Christ's imputed righteousness Praise God, right The penalty that you deserve from your sin was paid in full at that time by Christ on Calvary's tree Praise God the power of sin Defeated broken by Christ now in him you will never die What an awesome shot right to our dead wicked deplorable disgusting needy hearts Now you can't do anything to earn that you can't do anything to buy it You simply trust Christ for it you and trust yourself to him to receive it by faith right you trust him That's a repentant faith a repenting Believing a believing repentance. You trust Christ entirely for it. You and trust yourself to him. However However, here's the difference The Bible clearly teaches and the Bible clearly teaches here in John chapter 16 verses 1 through 4 When speaking of the warnings that the Lord Jesus Christ is giving here Consider how this fits together the Bible clearly teaches that the Lord must preserve us We must be kept by the power of God for that salvation ready to be revealed in the last time, right? Now we are kept by the power of God. We're not kept by our own power. We couldn't keep ourselves one second We're kept by the power of God and listen his ongoing preservation of his own is just as necessary to our salvation as Anything else that Christ has purchased on the cross for us. You hear that you understand that in other words If he doesn't pervert preserve us, we don't go to heaven when we die It is his preservation of us his keeping of us is as necessary to our full and final salvation As anything else that Christ has purchased for us on the cross Christ has purchased that for us by his death on The cross by his sacrifice the reason That we are kept the reason that we're kept is because God has promised to keep us God himself has promised to keep us and he is faithful to his promise. Amen. I Am persuaded. I am persuaded. I'm convinced. I'm persuaded that he is able to keep that which I've committed to him until that day Now the way that we are kept The way that we are kept is by the grace of God in the power of the spirit through appointed means God uses means It is necessary that God keep us that God preserve us, right? It's necessary that he do that. It's necessary That's the way it's gonna be for eternity, right? It's not that you and I one day got our vaccination so to speak and now we can do whatever we wanted we're Saved and so from now for all of eternity God can be sort of this ancillary afterthought because we're saved now So in essence, what do we need him for right?  He'll never forsake us. He'll never abandon us But the way that we are kept is by the grace of God and the power of the spirit through the appointed means That God has appointed those means which God has appointed for our perseverance. This is important all right These are the means By which the Lord applies The effects so to speak of the vaccination These are the means by which the Lord preserves us under the healing balm of the vaccination Right and the Lord has in his wisdom Ordained that it should be this way. So in other words to all those who would say I'm a Christian, right? I'm a Christian And so the all that I need to do all that I need to do now now that I'm a Christian now that I'm saved Now I got my vaccination. All I need to do is I just we need to remember that shot I got and whenever I'm concerned Whenever I you know, just don't want to strive don't want to obey Don't want to follow him. I just need to remember that Vaccination that I got I need to meditate on that vaccination and just remember that I'm I'm a Christian. No, no no God has appointed means by which he preserves us to the end that we might be saved And we are charged by God with the responsibility of Applying those means in our own life in the power of the spirit through faith in Christ, right? One means by which he keeps us one means by which he preserves us John chapter 16 verse 1 He keeps us by causing us to remember his word and we take heed according to his word And we trust him and we follow him Causing us he causes us by his spirit to heed his word so that we don't stumble that to preserve us in the faith He tells us Listen to preserve us in the faith. He tells us to exercise ourselves toward godliness by faith in the power of the spirit and You and I we persevere in the faith by doing just that The Lord commands us to exercise ourselves toward godliness That is an appointed means by which the people of God persevere in the faith What are you doing if you're not exercising yourself toward godliness? You're falling away You're falling away You've heard the analogy right before that shark in order for a shark to breathe It has to continuously some sharks have to continuously press walk past water over their gills in other words Keep moving keep moving if the shark stops the shark starts dying Listen you're a Christian and the Lord preserves us through appointed means so if you stop pursuing Those means what are you doing?  He tells us that in order to live we must put to death the deeds of the body by the spirit He tells us that if we do not do that He says that we'll die So what do we do we persevere by doing that in faith? We persevere by doing that in faith He tells us to work out our own salvation with fear and trembling. Listen. He tells us to strive He tells us to agonize. He tells us to be diligent. He tells us to labor He tells us to fight sin. He tells us to pray. He tells us to meditate on his word He tells us to evangelize. He tells us to love the brothers all of this by faith in his power He tells us to do these things and it's through these means that God preserves us in the faith, right? Now if you're not if you're not persevering in these means by faith in the power of his spirit Then you're not persevering in him. You're not persevering in him And he has said that it is he that perseveres to the end that is saved And do you see the difference, right? You see the first the error the error that is stuffed into the skin of a truth, right? It comes to us in the form of a half-truth, which is a deadly whole lie Do you see the difference, right? We do get a glorious one-time vaccination We get a glorious sacrifice on our behalf But our preservation our preservation involves going to rehab His persevering of us is a Rehabilitation process it's us going to rehab in essence And he has promised to never leave us he has promised to never forsake us But he has promised that if we are truly in him if we are truly in him then we will be in rehab We're gonna be going through that. Amen That rehab comes through means our responsibility our responsibility is to trust him to trust him every day Now notice the difference here Between the lie and the truth notice Satan's deception One is the error of antinomianism lawlessness One is law anti namas against the law. It's the error of antinomianism. It is easy Believism, it's a Repackaging if you will of the kesec error of let go and let God you just got to do what you want to do Just do what you want to do As Satan worshippers say that don't they that's not the The other Represents the Christian life Now even if you have difficulty Fully understanding or grasping these things these things are worthy to be meditated on this is what scripture teaches all right to think Through to process in your heart and mind to understand what the Bible teaches these things are very important to understand But if you don't fully grasp them right if you don't fully grasp them It's all very simple and very clear by God's grace When the Bible says to strive What do you do? strive you strive Pretty simple right pretty clear the Bible says strive you Strive you trust him and you strive you strive by trusting him you strive in the power of the spirit trusting him But you strive right you strive when the Bible says to exercise. What do you do? That's right. You get catch on quick right you exercise You exercise you trust him and you exercise yourself toward godliness when the Bible says obey What do you do you obey trust and obey for there's no other way right be happy in Jesus What happens when you don't obey? Means that you don't trust him means that you don't trust him and you begin dying What happens John chapter 15 verse 1 2 through 6 7 8 When you don't abide in him when you don't abide in him what starts happening You start becoming withered as a branch eventually you are cut off Cast out as a branch those are gathered up and they are burned in the fire When the Bible says abide what do you do you abide you abide you trust him you trust him You entrust yourself to him and you abide Now someone might retort to all that Where's the grace in that? Where's the grace aren't we to rest in his grace aren't we to rest in his grace? Yes, yes, we're to rest in his grace, but listen Rest in his grace from the guilt of condemnation Rest in his grace from the fear of death Rest in his grace from the just penalty of the law that is our due Rest in his grace in some part of right, but not in all parts from sin Victory over sin the Lord gives us That's not all rest right that other sin We battle it's a fight. It's a battle Grace in the New Testament is not rest from labor you see There's a sense in which that's often presented that way That somehow when we become converted when we're saved all of a sudden now it's rest from labor Why try why labor why do? We forget that it's the grace of God that teaches us to deny ungodliness It's by grace that Hebrew says we may serve God acceptably Listen to Paul the first Corinthians chapter 15 verse 10. Let's know what Paul says Paul says by the grace of God. I am what I am and His grace toward me was not in vain But I Paul says labored more abundantly than they all yet not I Paul says But the grace of God which was in me What did the grace of God in Paul caused him to do? Labor and it caused him to labor abundantly more abundantly than they all That was the grace of God at work in Paul If the grace of God is at work in you you're gonna be at work in him Right, that's the grace. That's the grace that we're talking about. That's biblical grace. That's Christ purchased God given grace And we get all that From an understanding of the Lord is communicating back in John chapter 16 verses 1 through 4 all right The disciples the disciples then and now disciples then you and I disciples now We must persevere through persecution and difficulty by remembering his words. That is an appointed means to our Preservation it's an appointed means to our perseverance Remembering his words. It is the means by which they and the means by which you and I are Preserved one means by which we're preserved from stumbling from falling away. Okay now all of this all of this is to be done by faith in Christ by faith in Christ When they face persecution when someone's kicked out of the synagogue verse 2 right or killed There to remember his words and trust in Christ trust in Christ in verses 5 through 6 last week We saw how they should trust his plan. He has a perfect plan They weren't considering his plan in verses 5 and 6 and then in verse 7 last week We saw that how they should trust in his provision He's going to provide for them the helper. What a glorious blessing now We aren't to persevere on our own strength If we should persevere on our own strength then somehow we get the glory for our perseverance That's not going to be the case We are to persevere in the strength of God's spirit God gives his spirit as provision for us So that our strength is in him our persevering is in him and God gets the glory Peter says this in first Peter chapter 4 verse 11 Peter says if anyone ministers Let him do it as with the ability which God supplies that in all Things God may be glorified through Christ Jesus to whom belong the glory and the dominion forever and ever amen Right if you're going to minister if you're going to labor if you're going to face persecution if you're going to evangelize If you're going to read your Bible if you're going to pray if you're going to love the brothers Whatever you do do that by faith in Christ in the strength which God supplies by his spirit and in all of that With the strength that God supplies by faith in Christ All glory goes to God all glory goes to Jesus Christ to whom belong the glory the dominion forever and ever amen Now this promised help God sends in John chapter 16 verse 7 and He says there explains there verse 7 nevertheless. I tell you the truth. This is an emphatic statement very important It is to your advantage that I go away For if I do not go away the helper will not come to you But if I depart I will send him to you. I'll send him to you as we come to now The Lord Jesus Christ saying in his departure They're sorrowful over the persecution. They're gonna face They're sorrowful over their loss at him leaving. They're sorrowful sorrow has filled their hearts But again in his grace and his mercy to them his grace and mercy to us He reassures us that it's to our advantage that the Lord Jesus Christ goes away And we're gonna explore that advantage in point three and four on your notes in your bulletin, right? Point three is this The spirit is at work in the world. It's to our advantage the Lord Jesus Christ departs Because in departing he will send the spirit and the spirit is at work in the world We're gonna look at the spirits work in the world in verses 8 to 11 look there with me verse 8 When he when the spirit of God spirit of truth has come He will convict the world of sin and of righteousness and of judgment of Sin because they do not believe in me of Righteousness because I go to my father and you see me no more and of judgment because the ruler of this world is judged and when the spirit of God comes all right He has a convicting ministry to the world a convicting ministry to the world that word in the Greek has a Semantic range to it a range of meaning and I think that much of that semantic range is Included here in the use of this word in verse 8, right? Usually translated that word means to convict to Reprove some of your translation say to convince or to expose That word is used in Matthew chapter 18 verse 15 when in that church discipline Process or resolving conflict within the church you're to go to your brother who sinned against you and you're to Tell him his fault between you and he alone Tell him his fault translates the same Greek word right it has that kind of understanding to it The convicting work of the spirit in the world is to in essence Tell them their fault. He has a convicting a convincing a reproving an Exposing work to do in the world seems to be that sense here in verse 8 In that sense he renders a verdict if you will he renders of verdict in other words The spirit of God and Lanko he publicly Exposes the guilt and shame of the world and he calls people in the world to repentance That's what the convicting ministry of the Holy Spirit in the world is okay in that ministry the spirit of God reproves and Convinces So that many are genuinely converted that's a positive side to that spirits work the spirits work in the world, right? He convicts he exposes he renders of verdict. He Exposes their guilt their shame. He tells them their fault so to speak so that many in the world Hearing the gospel hearing the preaching of the gospel are genuinely converted. The spirit does that work in others That ministry of the spirit of convicting the world of sin convicting the world of righteousness and of judgment hardens them and That hardening results ultimately in everlasting torment those people that reject the Lord Jesus Christ They reject the revelation of the Lord Jesus Christ by the spirit in the world Burn in hell for all eternity Now the spirit of God this ministry of the spirit is Facilitated in one sense through the preaching of the gospel through the preaching of the gospel and a good example of this once again Is Peter's sermon at Pentecost turn there with me to Acts chapter 2. Let's take a look at that sermon one more time Another just a good example of this very ministry of Conviction that the spirit does in the world the spirit is at work in the world convicting the world of sin righteousness and judgment Look at acts chapter 2 and Look at verse 32 acts chapter 2 verse 32 now Peter has begun preaching his sermon at Pentecost His sermon is full of conviction Convicting the Jews here of their sin against God the rejection of the Lord Jesus Christ look at verse 32 Peter says this Jesus God has raised up of which we are all witnesses therefore Being exalted to the right hand of God and having received from the father the promise of the Holy Spirit He poured out this which you now see in here This is the coming of the spirit of God right the spirit poured out at Pentecost so the spirit now At work in the world convicting of sin righteousness and judgment as Peter is preaching his sermon at Pentecost Right verse 34 for David did not send into the heavens But he says himself the Lord said to my Lord sit at my right hand till I make your enemies your footstool Therefore conviction right that all the house of Israel know assuredly that God has made this Jesus whom you crucified both Lord and Christ Now if Peter Had preached this sermon One day earlier Richard Phillips says this one day earlier and the only result would have been Peter's arrest No one would have tolerated what he said However here with the spirit working with the spirit at work through the preaching of God's word to convict Them of sin righteousness and judgment and to convince his hearers to expose Their guilt to tell them their fault the result here was completely different look at verse 37 When they heard this by the grace of God by the working of the spirit of God They were cut to the heart and said to Peter and the rest of the apostles men and brethren What shall we do and Peter said to them repent? Let every one of you be baptized in the name of Jesus Christ for the remission of sins And you shall receive the gift of the Holy Spirit Opposite result we see in Acts chapter 7 look at Acts chapter 7 Again, this is the spirit's work the spirits work To some when they hear the gospel preached the spirit of God breaks their heart over their sin Brings conviction to their heart it makes that conviction makes its way from their head to their heart The spirit of God causes them to be born again, right? They see the exceeding sinfulness of their sin and they come to repentance and faith in others the work of the spirit of God Hardens it hardens look at Acts chapter 7 drop down to verse 51 Acts chapter 7 Beginning in verse 51 here Stephen preaching very similar very similar sermon you stiff-necked and Uncircumcised in heart and ears. He is telling them their fault, right? the spirit of God Telling them their fault exposing the unfruitful works of darkness that exposing there the unfruitful work By the way, it's the same word elenco Exposing their wickedness, right? He says you stiff-necked and uncircumcised in heart and ears You always resist the Holy Spirit as your fathers did so do you which of the prophets did your father not fathers not Persecute and they killed those who foretold the coming of the just one of whom you now have become the betrayers and Murderers who have received the law by the direction of angels and have not kept it Now when they heard these things they were same phrase right cut to the heart they didn't respond by Turning from their sin their wickedness and repentance and faith they turned in hostility in hatred They gnashed at him with their teeth verse 55 But he being full of the Holy Spirit gazed into heaven and saw the glory of God and Jesus standing at the right hand of God By this convicting work the convicting work of the spirit in the world the convicting work of the spirit of God Paul says in 2nd Corinthians chapter 2 verse 15 Paul says that we are to God The fragrance of Christ among those who are being saved and among those who are perishing to the one We are the aroma of death Leading to death and to the other the aroma of life leading to life Those are the two effects of the preaching right the spirit of God at work convicting the world of sin Righteousness and judgment when the preaching of the gospel goes out It is going to be the aroma of life leading to life for those who are being saved It's gonna be the aroma of death leading to death to those who will reject the Lord Jesus Christ and fall into hell Remember what the Lord said back in John chapter 16 remember what the Lord said in Chapter 14 verse 12 right when Lord said in John chapter 14 verse 12 He said that when he goes when he departs His disciples those men at that time and us down through the ages His disciples will do greater works Greater works. He says there because he goes to the father right and we established there in John chapter 14 That he was referring to the coming of the Holy Spirit and what greater works do his disciples do They don't they don't raise people from the dead. That's a pretty great work, right? They're not restoring withered arms They're not restoring people born death restoring their hearing people born blind restoring their sight So what is the greater work that his disciples do they preach the gospel and the power of the spirit and conversion comes Right, that's the greater work So back in John chapter 16 verse 8 then Jesus is referring to the same Coming of the spirit and so he says in verse 7 It's to your advantage that I go away so that the helper can come that just means that in his Death in his burial then in his resurrection and his ascension and in his session at the right hand of the father That then the Holy Spirit can come the Lord Jesus Christ Purchased by his blood the promise of the new covenant in the coming of the Holy Spirit And the Spirit then does that convicting work that is necessary for any sinner to be saved The Spirit of God in that sense Takes what Jesus Christ has accomplished at the cross and he the Spirit applies to the sinner that truth that Sacrifice through repentant faith in Christ Just as Jesus Christ came into the world to save sinners the Holy Spirit takes that truth Convicts the world and applies the work of Christ to save sinners and unless the Spirit Comes and does that work? No one would ever be saved right so you and I you and I Are testimonies of the Spirit of God at work? When you were genuinely saved it's because the Spirit of God caused you to be born again Granted you new life in Christ Opened your blind eyes Opened your death stopped ears and you were given repentance and faith So how does this then encourage us in the preaching of the gospel?  And we see that attested to in church history all those men were killed John exiled eventually died in exile, right? They're going to do these things to you. How would this truth then? Encourage them in the preaching of the gospel The Spirit of God is at work The Spirit of God is at work And we've used this simple analogy before but if someone was standing right there in the lobby and they're standing there Right and they're hearing the preaching of God's word and they're weeping and they're weeping their heart is tender So and you knew you knew God had revealed it to you in a vision You knew that if you walked out in the lobby and you shared the gospel that you preach the gospel to them That person right there on the spot is going to be saved there'd be Log jam trying to get out the door to go share the gospel with that guy, right? You'd have to wrestle somebody to get to him to share the gospel, right? What keeps us from going out and sharing the gospel all the time knowing this gracious promise of the Lord Jesus Christ Him having departed now the Spirit of God has come the Spirit of God is at work in every one of those conversations When you're standing at the door and you're pleading with someone to turn to repentant faith in Christ So to turn from their sin and be converted when you're pleading with them God the Spirit of God is pleading through you The Spirit of God is at work in that a summit's going to harden But others are gonna be saved You're gonna be the aroma of death leading to death and listen settle in your heart That's the way it goes you're gonna stink like death to some people But others you're the aroma of life leading to life and in all of that God is that where there are no Chance encounters, right?  There's coming a yes All right, so what does sales guys do they just keep making calls? We're so that's sales We've got the greatest message ever Conceived of conceived by God himself in his grace and mercy Preach the gospel right and sinners get converted the convicting work of the Spirit Has three objectives three objectives convicting the world of sin Convicting the world of righteousness and convicting the world of judgment first in verse nine Convicting them the world of sin of sin verse nine because they do not believe in me Now the first work of the Spirit of God is to convict of sin McShane says again It is to give a person a sense of the dreadfulness of his sins and to make him feel How surely he is a lost sinner You see this conviction The work of the Spirit the work of the Spirit in convicting of sin does not merely result in an admission of sin All you got to do right is admit believe and confess. No, I beg your pardon. That's not the way that works It doesn't result in a mere admission of sin. I'm a sinner and you know besides we're all sinners. No, no The conviction of the Spirit of God the Spirit of God the conviction that the Spirit of God uses to convict one of sin Seeps into the heart and wrecks a man Whitfield talks about how often this comes regarding one primary sin where the Spirit of God convicts of one primary sin Uses that sin to wreck a person before saving them He talks about the woman at the well remember in John chapter 4 Lord Jesus Christ talking to the woman at the well says go get your husband go get your husband Instantaneous conviction right? She answered the woman at the well answered I have no husband. I have no husband Jesus said to her you have well said I have no husband for you have had five Husbands and the one whom you now have is not your husband in that you spoke truly that Powerful conviction over that woman's adultery led to further conviction over all her sin Says there that the woman then left her water pot went her way into the city and said to the man come see a man Who told me all things I ever did? Remember the account of the Lord Jesus Christ knocking Saul off his horse on the road to Damascus Lord convicts him Why Saul are you persecuting me? Says that the Paul was so convicted over sin that he didn't eat or drink for three days a Conviction by the spirit of God and the heart of a sinner Wrecking the man now. Let me ask you Has this happened to you? Has this happened to you? The Bible says that when he comes when the spirit of God comes he will Convict the world of sin the primary reason for this conviction over sin is given in verse 9 He says it's because they do not believe in me. They do not believe in me Unbelief is the root of all sin, right? It's the foundation of all sin. There's no sin that is worse Than unbelief many hear the gospel, right? Many are called what if you were chosen many are called many hear the gospel They acknowledge what Christ has done They see the price that was paid the blood that was shed and they respond with Indifference they're hardened by the deceitfulness of sin What should cause them? To put their head in their hands and weep They respond to you with cold indifference or with hostility They are cold hearted Let me ask you at some point Have you been made by the spirit's work?   You have turned aside. You've done what's right in your own eyes.  Where you have nothing Right all you have left is to throw yourself at the cross of the Lord Jesus Christ and plead with mercy God be merciful to me the sinner Have you been brought to the point where you have nothing left you have not a shred of your own? Righteousness not a shred anything that you think you've done good is as a filthy rag in God's sight Has the spirit ever moved you Moved you to the point where all you want in this world is to flee to Christ Have you ever experienced that? That's the spirit's work of convicting of sin Zechariah chapter 12 verse 10 God says this I will pour on the house of David and on the inhabitants of Jerusalem the spirit of grace and supplication then They will look on me whom they pierced Yes, they will mourn for him as one mourns for his only son and Grieve for him as one grieves for a firstborn Ever lost a loved one before?  If you've never mourned over your sin with a godly sorrow, you're not converted You're not converted the work of the spirit of God is to convict of sin We're standing at a doorway with a man And he was telling me I just I've Never felt that I was really that much of a sinner Really never thought of myself as a bad person listen On the authority of God's word if that's you you're not saved you're not a you're not a saved person The Holy Spirit Convicts the world of sin All of this comes through the preaching of the good news It's the preaching of the good news that must be preached in the context of the bad news They have to fully understand the bad news, you know, it's interesting That considering this convicting work that the Spirit of God is called the Comforter Now why is that? Why is that? The Spirit of God doing that convicting work Wrecking your heart over your sin the gracious work of God by his spirit in the in the heart of a sinner Why is the he then called the comforted the Comforter? He's called the Comforter Because blessed are those who mourn Blessed are those who mourn for they shall be Comforted that's right Come to me all you who labor in our heavy laden and I will give you rest rest from what? Rest from the burden of your sin Remember Christian right and Bunyan's pilgrim pilgrim's progress He's carrying this load on his back at first He doesn't even realize he has on his back He comes to understand the weight of that sin that big huge packet sin on his back Right and there are many different directions that he can go to remove the weight. He can go to Morality mr. Morality and get it removed he can go to go toward legalism get it removed Ultimately he goes to the cross of Christ to have his sin removed The Spirit of God convicts the world of sin Secondly verse 10 The Spirit of God convicts the world of righteousness Verse 10 of righteousness because I go to my father and you see me no more Most people that you talk to today would profess their own righteousness We know that from being out witnessing right they say i'm a good person.  Yes, I do Right they profess their own righteousness. We've already established that is not the case. There are none who do good. No not one There's only one. There's only one who has lived Who deserves to be in the presence of god only one Only one who perfectly fulfilled the law fulfilled all righteousness And unless you have his righteousness you will never see god Unless your righteousness exceeds The righteousness of the scribes and Pharisees you will never see god the root of all sin is unbelief But it is faith or belief in christ for his righteousness that saves If you're to be saved you must have his righteousness of the lord jesus christ Perfectly fulfill the law of god. He goes to the cross cross as a perfect sacrifice God imputes if you're in christ through repentant faith god imputes your sin To christ and christ bears your sin on the cross The very same time god imputes the perfect righteousness of the lord jesus christ to you You have his righteousness And you become the righteousness of god in him in him you become perfectly holy You don't have to turn there. Let me just read it to you philippines chapter 3 Explains this well beginning in verse 7 where paul says but what things were gained to me These i have counted i've counted them all paul says i've counted them As loss for christ yet indeed i also count all things lost for the excellence of the knowledge of christ jesus my lord For whom i have suffered the loss of all things and count them as Rubbish that i may guide christ and be found in him Not having my own righteousness which is from the law But that which is through faith in christ the righteousness which is from god by faith That i may know him and the power of his resurrection and the fellowship of his sufferings being conformed to his death If by any means i may attain to the resurrection from the dead The righteousness Of the lord jesus christ was proven acceptable to god by the fact that god raised him from the dead He raised him from the dead Because i go to my father and you see me no more verse 10 His righteousness proven by his resurrection verse 11 spirit of god also comes and convicts the world of judgment Of judgment verse 11 because the ruler of this world is judged It's interesting that word there is judged Is past tense it's past tense but the word is speaking here of the cross the ruler of this world Judged at the cross the lord jesus christ speaking of it in his omniscience as a completed Final act that is coming But he speaks of it as being so certain that it's in the past tense here The ruler of this world is judged If you think about it verse 9 verse 10 verse 11, this is the preaching of the gospel This is the preaching of the gospel you preach a sin Of righteousness and of judgment Act chapter 17 if you think about the preaching of paul the areopicus paul said These times of ignorance god has overlooked but now he commands all men everywhere to repent why? Because he has appointed a day on which he will judge the world in righteousness by the man whom he's appointed Jesus christ is that man when you preach the gospel you preach sin you preach righteousness and you preach judgment You preach christ what does the spirit do the spirit convicts the world of sin of righteousness and of judgment And he testifies of christ in all of this for those disciples walking To the garden of gethsemane with the lord jesus christ on that night And to all disciples through the ages that have come after them This should lead us to boldness in our preaching Boldness in our preaching faithfulness in our preaching Trust in christ This should lead us to fervent hope in him Right god has given us a tremendous blessing as the lord jesus christ departed those men As he ascended to the father as he sits now Raining and ruling at the right hand of god the father even now We have the promised holy spirit helping us Helping us in our preaching Helping convicting the world of sin righteousness and judgment Let's be faithful to him right we've been given every glorious blessing that can be imagined Is given it all to us in christ so we need to be faithful to him amen amen. Let's pray Father in heaven Lord thank you for this blessed assurance this reassurance that we have from john chapter 16 lord We praise you and thank you lord that you prepare us in your word not as slaves, but as friends in john chapter 15 and you lord Allow us to see the reality of of your plans and purposes fulfilled in christ You allow us to see the persecution certainly the world's hatred that Is ours because we're in him because the world first hated him it's going to hate us also and we praise you lord for your blessed reassurance The hope that you've given us the joy that you've given us you said that our your peace is ours your peace you give to us You've promised to go and prepare a place for us and you promise to be with us even to the end of the age and Glory upon glories you've given us the holy spirit who's at work in the world to convict the world of sin Righteousness and judgment so that when we lord by faith in christ and the power of your spirit go out to preach the gospel The spirit is at work in the world already The spirit is at work in the world through the gospel to convict And we know that our labor is never in vain in the lord We thank you lord for that beautiful blessing that beautiful truth Help us to rest in it. God give us a holy boldness thinking about these things Let us be fervent and faithful in the ministry that you've given us to do and keep us lord by your word Keep us from stumbling Preserve us in the faith. I pray that no one here would fall short God I pray that if any if there's anyone here not saved god break their heart over their sin Wreck them over their sin bring them to the end of themselves Help them to see their righteousness as a filthy rag Help them to see their sin more the way that you see it and save them lord for your glory Cause them to be born again grant them repentance and faith May they see the beauties and the excellencies of the lord jesus christ and be saved We love you lord. We thank you for our time together today. Thank you lord for the blessing of going to the park this afternoon to see Um converted centers baptized to hear testimonies and to welcome new members into our church We're very grateful for that lord. I pray that you bless our time together there All for your glory god in jesus name. Amen


I swore I wouldn't do this, and here we are. The sunglasses are obviously because the sunglasses match my shirt, and because today is not just any video. Today is a holy grail unboxing. So I have only two boxes today. This is actually a really good bubble wrap. Oh my gosh. This is the Briar Fest 2013 CC Shuffle in Grula. Yeah, get those shades on because she's hot. These models are really hard to get a hold of. She's a really nice color. This marking isn't actually part of the original model. This model has been etched, and unfortunately, this was my concern with her. The etching is not great, and a lot of the plastic has scuffed off with the etching, so it's a little uneven. But I mean as a shelf piece, which is the majority of what my collection is, it's cool, it's fine. She smells really strong. She smells like soap. Alright, so this is the one that I'm most excited for. Not the holiest of holy grails. I kind of ran out of grails. I purchased most of them that I wanted to. This one can be my grail. What? What? Seriously? This horse came in like a legit pony pouch. Dude, I am impressed. I always get nervous when I pull them out, because I'm like, did they send me the right one? I've never had it happen, but in every shelf. Should we put on the sunglasses? Because things about to get bright. It's really hard to see the model when wearing sunglasses. I would not recommend unboxing grails with a pair of sunglasses. This is the Briar Fest 2014 Pop the Quirk of Bay and Dakota. I was looking for this guy for a long time, and I finally found one that was affordable. I think I only paid $80 or something for him. He was not very much. I paid $75 for her. So that's really cheap for these two models, because they're really hard to find, and I think they're really gorgeous. So thank you so much for watching. This will be the last of the unboxing videos for a while.


All right. Good morning, everyone. So I was told to speak a little bit louder because apparently this room has a weird echo, and I can realize that because I hear myself speaking, and that's kind of disruptive. Good morning, everyone. I'm very excited to be presenting and moderating this panel here on hybrid cloud. My name's Sebastian. I found an open source project that's called Scaler, which is an open source replacement for things like horizon and heat, as well as a lot of other hybrid cloud management. So I've got a special experience and a lot of experience working with large and small companies deploying hybrid cloud applications. So with that, I'd like to introduce our panelists here, Randy, Mark, and Jena. I'll let you guys introduce yourselves, and then we can kick it off. I'm Randy Bias. I'm Randy Bias, the CEO and founder of Cloud Scaling. I'm on the OpenStack Foundation Board of Directors, and I guess I've been a cloud guy and blogger and pundit since December 2006 when I put my first application up on Amazon's Elastic Compute Cloud. And I'm Mark Williams. I'm the CTO of Redapt. Redapt is a systems integrator that enables large-scale data center infrastructure and turnkey cloud solutions, both private and hybrid. Before that, I was actually a customer of Redapt at Zynga where between 2008 and 2012, I was responsible for infrastructure operations, private data centers, our expansion into a lot of Amazon consumption, and then ultimately, we built a private cloud called ZCloud where we moved everything back out of Amazon. Hi, I'm Jena Hossain. I've worked at Google for about 10 years. I'm currently an engineering director leading the cloud networking efforts. And before that, I've been doing that for about two years. And before that, I spent the last, the previous eight years working on site reliability and production engineering. So I was part of the team that originally productionized Borg and shipped a bunch of the cluster management and networking components that we use today and that we are trying to now turn into cloud products for public cloud. Excellent. So hybrid clouds are pretty hot topic these days. And it's pretty much on the checklist of all IT departments. Why is that? Any idea why it's such a hot topic? Well, I mean, when we're talking to customers, they just want flexibility. I mean, what I've seen so far is that pretty much everybody is trying to look at both public and private. And they kind of want to hedge their bets and have a little of both. And they're not sure if they're going to wind up being 10% private, 90% public, or 10% public, 90% private. They're not sure if they're putting DevTest QA outside or inside. And it sort of seems to change vertical by vertical. So financial services guys are like, we're never putting production out on the public cloud. But if we can put DevTest and QA out there, that's great. And then people like Netflix are saying, we're all in on public cloud. Maybe we've got a little bit of onsite capacity, but that's it. So I think it's about flexibility and choice more than anything else. I think another thing that's influencing it is the fact that price drives a lot of change in the prioritization of what standards you might have around governance. I've seen a lot of relaxation over the past several years of those primary concerns about public cloud and the safety of data. You don't hear about just catastrophic exploits in public cloud because of hypervisors being shared by multiple tenants. So I think that the lack of horrible news there has influenced that. And I agree with Randy, too, that just having the agility, the fungibility of resources to handle the unexpected surprises, I mean, certainly that was advantageous for us in deploying unpredictable workloads. So just having the ability to do both is a natural conclusion to that. So I think a lot of it that I've seen that comes down to both trust and cost, and not just outright cost, but also outlay, like whether or not you are focusing on your capital expenditures and that's your primary motivation or you want to focus on OPEX. And you look at a company like Netflix, and they really want to put all of their money into serving customers, right?  No, that's not accurate. I talked to Adrian about this extensively. And what he said is that they didn't know what their growth was going to look like. They didn't want to try to build a data center and have an environment and then blow through that growth. That was number one. And number two is they wanted to enter new markets, like the European Union, without having to take down data center space or buy servers or hire boots on the ground. And that was the primary motivator for them. It wasn't so much the conversion of CAPEX to OPEX. I mean, you're making good points for some people that's true, but it tends to be startups not established $2 billion companies. OK. I mean, the last thing I want to do is get into a debate. Why not? A lot of the things you just mentioned, a lot of the things they didn't want to spend on is CAPEX. That is boots on the ground, pouring concrete. That is hard work. That is outlay. It's not just money. It's also the agility to do it. But I think it's all wrapped up together. And definitely there's different kind of motivations. And in my case, initially, Zynga was very happy consuming a lot of public cloud. And then being pre-IPO, looking at that OPEX line item and how that materialized into valuation of a company versus the ability to spend cash on hand to do private infrastructure was actually part of the catalyst to which led us to build private cloud. So again, it definitely depends on the economic kind of value of CAPEX and OPEX internal to the business. Yeah. So I mean, I think the thing that's important to understand there is that when you are thinking about, is it CAPEX or OPEX? Like you just suggested, Netflix might have been for going into, say, the European Union. Sometimes it's not really actually about that. It's about the ability to experiment and have very minimal consequences. If Netflix went into the European Union with the Amazon region there and nobody in the European Union signed up, they just turn it all off. No harm, no foul. It's a few thousand dollars, maybe 10s of thousands of dollars to do that experiment, whereas if you have to go in and you have to take down data center space and sign long term leases, it makes it a much bigger thing. So it is about cost and there are factors that are related to money, but I think business agility is the primary driver and cost winds up being sort of dragged along as sort of a consequence or a side effect. So since we're talking about cost, I've often heard that it can be economical to own the base and rent the spike. And with the recent price wars, how has that changed that model? So I've helped a lot of customers do the ROI analysis of typically they come to Redapt looking for a path out of Amazon, feeling the cost has been incredibly high. So after April 1st, that modeling that I'd often do kind of got a lot worse. That said, there's still a lot of valuable requirements for people who have performance needs, et cetera. But the concept of own the base, rent the spike, I think is great when you think of cloud just being the infrastructure. But if I think about one application living in that particular model, I get concerned because we actually did this at Zynga because we were the ones kind of broadcasting this capability, right? So most of what we were doing was actually taking workloads wholly off of Amazon and wholly into our Z Cloud. And the reasons why we're doing this is cost and performance. We had built something that was ideal for our workloads in our private cloud. So after we had done probably 80% or 75% of that shift in back into our private cloud, what we wanted to prove is that we could actually burst back. So we took one game, a very stable game, not a very risky game in terms of value to the company. So we proved that we could actually do it. And what we found was to do this with one application, you're kind of getting the worst of both worlds performance-wise. You're dealing, especially when you're dealing with, say, that web tier that's user-facing, that's what you want to be able to scale and burst, right? What we found was we had built something ideally performant for our workloads in the private cloud. And then when we kind of mixed that with a public equivalent, C1 Excel, for example, you kind of get all the noise that's in there and it kind of drags into your private cloud environment. And then there wasn't really much value for that. So I think it's valuable for, I want to base rent the spike in terms of being able to move whole workloads into different clouds. But to operate one distinct application across them has some disadvantages, depending on how your performance can be expressed through those layers. Mark and I like to disagree on this one. So the thing is, and I remember when the whole notion of hybrid cloud first came out and people were talking about it, and my problem with the way they were talking about it was they were like, we're just going to jam any two clouds together. OK, well, to Mark's point, it doesn't really work out very well. You got a big gigantic VMware vSphere cluster running on Cisco UCS with a fiber channel saying you connect down to Amazon and it's like apples and oranges. They just don't make a lot of sense together. But if you design a private cloud so that it looks a lot like the public cloud that you're going to interconnect into, and there's very low latency between them, it can look like an extension of your data center. Also, it's very use case dependent. There's a company called Teradata. They're a data warehousing company from the 70s. They're trying to move their model to a SaaS-based model. 90% of their traffic comes in during one month a year. For something like that, even if the performance is wildly different, even in that case, it still makes sense. Because why would you over provision 10x for the other 11 months of the year? It just doesn't make sense. Yeah, I think where you have users not experiencing those potential differences, and you have an operations team that isn't going to have so many layers of technology and difference between two environments to dissect when there's an observed problem, I think that actually does work for games in our particular use case. Because user experience mattered and the performance and responsiveness of the server mattered, that's where it broke down for us. And just at a steady state as an operations person, when there's a wobble, I want as few things to step through to eliminate as fault as possible. And you get a lot of noise in a blended environment for one app. Do you have an idea? We're both right. So you brought up one important point which comes down. You mentioned latency. And we keep bumping into Google internally in the way we run our workloads. It's always about data locality. And people build apps and expect them to behave a certain way based on where the data is. And so if you want to move the app, you tend to have to move all of the data. It's very difficult to build an app that is actually able to span. These days, even more than a couple milliseconds is enough to really throw off a lot of workloads. So even if you are able to have a hybrid cloud environment where you are on-prem in the same metro as a public cloud, you run into challenges where you have to build the app to be able to split that latency boundary. Yeah, I mean, I know developers are really keen to get to a nirvana where they don't have to worry about storage and network and processors and that stuff. And they just have a magical container that auto-scales. And they put their code on it. And everything's wonderful. But latency is a key problem. And it winds up showing up in the app. There was a company called Triton Containers in San Francisco. They built an app in their data center with zero millisecond latency.  And all of a sudden, it was 100 times 200 milliseconds worth of latency every time that they did a transaction. And they're like, why is it slow? Well, the network isn't always zero milliseconds, and it's not infinite bandwidth. And the applications actually need to be designed to be sensitive to latency for the disk, for the network, and so on. And that is totally key. So in which cases does it? What type of applications make sense to put on a hybrid cloud with the applications on? Mark, you were saying that it makes more sense to put a single workload. Oh, we have a surprise. Sorry about that. All right, Ariel, go ahead and introduce yourself. Yeah, sorry, guys, for some reason, I thought we were on an 1115 set of 11. My name's Ariel Taitlin. I'm a venture partner with Scale Venture Partners, which is a mid-stage venture investment firm in cloud, IT, infrastructure, mobile, SaaS. Before that, I was running operations and cloud platform tooling at Netflix, which I think is the reason for my participation. We value your opinion on investments, but yeah. So we were talking about what types of applications make sense to have on a private cloud, what types of applications make sense to put on public cloud. If there's any use case for having an application right both simultaneously in that sense, and what types of applications are contentious there? So some of the use cases I've seen on what I would consider successful or well-balanced. And certainly as a company who's never started doing anything with hybrid before, one easy path toward that is taking advantage of, say, S3-like or object storage-like functionality where you're already counting on latency being a factor in your application. So it's a very reliable type of service from Google and Amazon. The way you can build your application to not influence or impact user behavior or experience there is a low bar. And it's a lot harder to build your own object storage with those same capacities around geo-redundancy and multiple copies, et cetera. So that's one that I think works well. On the opposite side, where I've seen customers get into trouble is one particular customer I've worked with had a MongoDB cluster. They started everything in public cloud, and they ran into this just pain point around the performance of the instances supporting MongoDB. And I think it was mostly around EBS being kind of the bottleneck there. So they had a great problem, successful application. They had to move the MongoDB to their private data center and then have this direct connect between that. Great. Well, now what? When there's a failure, originally where they were failing a MongoDB primary master node into an Amazon one, everything just grinds to a halt because you're back to that performance state. So you have to be very careful with how you lay out your failover capabilities with your clusters, et cetera. We ran a hybrid cloud for probably about two years at Netflix, and really the reason was while we were going through the public cloud migration. And so while we were doing the migration, we would take small pieces of functionality, break it out from this monolithic Java app that we had in the data center, and push it out into the cloud. And like Mark was saying, we created a direct connect back, a tunnel back into the data center where it would go back and fetch data or push data for the source of truth. And then slowly, as we migrated more and more functionality, we deprecated everything that was in our data center. And we're running all public cloud. And then the other piece of that is I don't know if I would call it a hybrid cloud, but it was a multi-cloud. And that was for backup. And so we were running all of our Cassandra clusters, which was our source of truth for all the production data. In the cloud, in Amazon, we would back up all the SS tables into S3. We would back up all the S3 buckets into another Amazon region. And then we would back up the other Amazon region into Google Compute basically for business continuity purposes. So I think one pattern that shows up a lot is since backup is a common use for public cloud, there's also a strong case there for doing long-term data analysis, like places where you would want to apply compute to data, but you don't need it in immediate real time. And you can also use burst capacity there. So you already pushed the data over there. And again, this comes back to data locality. You've already pushed the data there. And you don't need to use it in real time. It doesn't need to be completely synchronized to what's in private cloud. And then you can find new workloads and to kind of chew the data a little bit more deeply. It's a double-edged sword, though, because on one hand, it's great to have it in a system like Amazon or Google, where you can have tons of computing power to apply it to large data sets. On the other hand, it creates data gravity there, so it's hard to get off if you need to. So if you've got multiple petabytes pushing that across the wire, it's very challenging, right? Yeah, my point is that because it's the backup use case, that's already part of the business plan, right? The data is already being pushed, so now there's an opportunity to apply compute to that data that you wouldn't have otherwise. Data analysis on the backup set. Yes. Are there any examples that are clearly a bad fit? Like, I've been hearing about cloud bursting quite a bit. And I personally don't believe in it. I've never seen it work. But what's your experience on the cloud bursting use case? I totally think it can work, but I think it's a technologically sophisticated thing to do. You can't just ram two arbitrary clouds together and burst from one to the other. Latency is going to matter. But as an example, an extreme example, if I had an AWS run private cloud in the same data center as the AWS public cloud with 0 milliseconds of latency and tons of bandwidth between them, why can't I burst between them? They're same software set, same general performance. It's technologically doable. I think the problem is that it's not trivial in all the issues that Mark and others brought up here come into play. And I think it's one of those places where it's dangerous for the uninitiated in cloud who just make a lot of assumptions about the way that infrastructure works. Sorry. So I have a little bit of tunnel vision having run Google's front-end serving infrastructure for a while. But I do find that large-scale web serving workloads are pretty amenable to hybrid cloud, especially if you've built it where you've got a lot of your back-end serving store and data sets in private cloud and you are used to serving from a web front-end setup in private cloud. You can move the user-facing component, the stuff that's actually terminating TCP out to public cloud pretty easily and often take advantage of geographic diversity, low latency termination to end user, and sometimes cashing. And so you can drastically increase your surface area via public cloud while keeping the actual back tiers and backends in private cloud. I mean, I think it's theoretically very possible. I haven't really seen anybody doing it. It's a great idea in practice, but it does take a certain degree of sophistication in the way that you architect your deployment pipeline and the way that you architect your application topology in order to abstract away all the underlying clouds. And unfortunately, we don't have clouds that are compatible to the same APIs and to the same behaviors. And that was a great talk that Randy gave yesterday about the things that we would need to see in order to have those capabilities. But today, you'd have to build a lot of that logic and a lot of that deployment tooling yourself. And that's a pretty major undertaking for, I think, a relatively small value. What would be some rough guidelines for the types of applications that could support those use cases and the types that absolutely can't? It's less about the application specifically, but it's more about the strict religious discipline of treating infrastructure as code and having no humans in the process, especially with the complexity of hybrid cloud. If you're really going to be vacillating workloads between multiple distinct entities like that, you don't want to be counting on humans. So I hope everybody in this room has already kind of wrapped around that conviction because you can't do it. It's not like I have a lot of customers who are just hearing the marketing of hybrid cloud and thinking it's this panacea of, oh, well, this just is going to solve all my problems, but they don't have these disciplines already. And I think those are necessary prerequisites regardless of what application you can do. And I've seen it done, I've seen Puppet on Windows work pretty well with some advanced customers, but that's a necessary prerequisite, in my opinion. Well, if you didn't see the presentation yesterday, you should go check it out because I cover a lot of this stuff about what I consider requirements for achieving interoperability with hybrid cloud in some reasonable detail. It's really hard to pin down specific workloads. I think that we've already identified that they need to be latency and sensitive as much as possible or built around making assumptions about latency. An example of that is that if you were using S3 in a public cloud and you wanted to have your front end here in another public cloud, if you're using the caching service, say like varnish caches in front of that object store, then that works out pretty well because you're going to have mostly be hitting the cache. Occasionally you're going to miss the cache, you're going to wind up reaching out to that other cloud. Maybe it's 100 milliseconds of latency. Maybe it's even another continent, but that's happening so infrequently that it's not really a problem. And so it does have to do with a combination of both the workloads and the application architecture. And if you are designing it right, if you're thinking about the fact that there's the nine laws of availability, I think they're called something like that, but the network is not always zero milliseconds. You don't always have infinite bandwidth. The other side is not necessarily going to always receive your connection requests. Like if you build that in, if you use like the circuit breaker pattern instead, like a Netflix did in their application, then I think that you can be very successful at doing it, but you should probably kind of put a toe in the water first and then a foot and then a leg, trying to like jump right to like very sophisticated cloud bursting is probably going to wind up in a lot of pain. Thanks for mentioning circuit breaker. Ariel, do you think you could talk a little bit about that pattern? Sure. I mean, I think the point that was being echoed was that if you design your application with the right architecture in mind, not even for doing hybrid cloud, but just in the way that a good distributed service-oriented architecture will be, then some of these use cases are enabled. The circuit breaker pattern, the way it works is think of a circuit breaker where every service invocation goes through the circuit breaker and the circuit breaker monitors the latency and the response codes, 200, 300, 400, 500 of the returning requests. And if everything goes well, it goes through and the request from the client goes to the service and it gets returned and it kind of follows the regular loop. Whenever there's a problem that's detected and a problem can be either it's configurable, but it can be either high rates, high error rates, or some kind of latency threshold that gets crossed, then the circuit is opened. And once the circuit is opened, then the downstream service is protected from further service requests and the client no longer actually makes the request to the service, but instead gets a fallback. And so you need to configure a reasonable fallback. One example that we use at Netflix, for example, was we had the personalization service, the moving personalization service. And so whenever a user logs in, you show them their personalized recommended listed movies. If for whatever reason that personalization service becomes unavailable, then the fallback is to display the top 10 listed movies or top whatever listed movie that gets cashed by the client service. And so what happens is, you know, you prevent these thundering herds problems because once a service gets restored, typically you get all this backlog of backed up requests that now gets flooded with the service, whereas instead now you've, you know, gracefully degraded the service while your downstream services are down. So since this is the OpenSAC summit, if you were to give advice to folks in the audience here that are building an OpenSAC cloud with the idea in mind that it's gonna connect to public clouds and they're going to have hybrid cloud workloads, what would be the requisites for that? Randy, you give some excellence on like the truck versus the car and can you start off with that? Yeah, so I had one slide yesterday where I was talking about sort of API compatibility and how that's not really enough. You can have clouds that have the same APIs on them but have, you know, fundamentally different behaviors. And the example I gave, you know, to kind of make an analogy was say like a sports car versus a semi truck.  So you're really clear, like out of the box, OpenSAC does not assign a public IP address when you spin up a virtual machine but if you go to Amazon, you spin up a virtual machine and every virtual machine gets an IP address right out of the box. Both of those have the EC2 APIs but there's a fundamental difference when you spin up a VM by default about what happens. And so the problem is that if you're an application developer and you develop on Amazon or you develop on a private cloud like OpenSAC, you bake assumptions into your, you know, application framework that are, you know, based around how that system operates and then when you try to port it or burst or whatever into another cloud, all those assumptions break because they're no longer valid. So, you know, what I have found and what I'm promoting and why I have my whole business is to focus on trying to make a version of OpenSAC that looks, smells, and tastes and acts like Amazon and is also provably, testably as similar as possible. Now you can't ever be 100% similar but you know, I just always remember the days of IPsec VPNs where you had 20 different vendors, none of them had any interoperability whatsoever even though they implemented the same standard because there was so much looseness in the standard. And the only way they fixed that was through testing, testing, testing, testing until they got to a point where they were able to make sure that everything worked together. So I feel like that's the only way forward is to have sort of like reference architectures that we agree on whether it's Amazon, Google, HPC, something else, and have testability around that so that we can make sure there's interrupt between those clouds. I think the other path to ensure that you're starting down is whether you're starting public and then adding private or vice versa recognize that, you know, you have your customer internally who's going to be consuming these infrastructures and that things are gonna be changing over time. Whether you're starting with one private cloud and adding multiple different private clouds or starting with one and needing to upgrade it that you want the user experience for how they interact with that infrastructure to be consistent. So that's that interoperability point. So something like a scaler really helps unify that and abstract away and offer a way to parameterize the differences between the different clouds that ultimately get instrumented. The other kind of caution I would have for those who may be already in public looking to go private is being very careful about the most evolved and more recent services like databases of service, ElastiCache, those kinds of things that are proprietary exclusive to those public clouds. You need to, and in our case we knew as we were going at Zynga from non-cloud private data center into EC2 that we only wanted to do instances and basic networking. We did not use ElastiCache load balancing. We didn't use anything that was kind of specifically proprietary for that known expectation that we wanted to be able to control our destiny and still do all the automation and configuration management through our interoperability tools but to not get stuck where we would have to replace a proprietary technology on the private cloud. Yeah, I mean, I'll echo all those points. I mean, I think the first thing you got to do is start with an open stack distro that is actually compatible with public clouds. Cloud scaling is a great one, another one. And, but that's kind of the base requirement because I don't think that's enough. But what you really need to do then is think about all the non-development related activities that you're gonna have to go through in order to actually deploy to and manage a multiple cloud environment. And so, you know, in Netflix, for example, we built a whole bunch of different open source tools that we released for how we operationalized all of our software. And so, you know, for example, we have Asgard, which is the tool that we use in order to deploy and manage software in the cloud. And, you know, I don't think to date there's an open stack implementation of it, but I know that Eucalyptus, for example, took it and implemented, ported it to Eucalyptus as well. So, you know, if you're in a Eucalyptus environment and using a public cloud environment, then you can use something like Asgard. And the different clouds really just show up as different regions. And you can deploy your code to different regions and they really just end up in different clouds. So, you know, whether... PayPal ported it. They did. PayPal did port it to open stack. Unfortunately, they didn't contribute it back. They just did it one way. So that doesn't help most of these people. But yes, you're right. But, you know, whether you use Asgard or whether you use any of the other tools out there that let you have that deployment and operational automation, monitoring, managing, auto-scaling, right? All of these different things that, you know, what Mark was mentioning, where you really don't want humans in the process of operating the software and you really want them to be managing exceptions rather than managing the normal operations. That's really what I think the key is to making a hybrid cloud successful or any kind of multi-cloud successful. Here's the GitHub repo. Here's the GitHub repo for the PayPal open-source Aurora project that is a port of Asgard to open stack. I just, I can't, I'm an open stack guy and we're in an open stack conference. I just want people to know Eucalyptus does not have any superiority in this area in any way. Sorry. It's there. It's there. You were talking about people need a framework, so one exists. Yeah, just like it comes up throughout all of this, but if you really want to maintain that flexibility and actually not accidentally lose it, it's going to come down to testing, right? You have to actually build systems that can run these workloads on public and private and multiple public clouds, multiple private clouds and actually be willing to move those around with some frequency, maybe not the production stack, but at least the dev stack, the staging stack and the more automated unit testing you have where you can just have a stack come up, give you a full green and then move that and constantly be running it on different environments. That's the only way you're going to catch regressions. Randy. I want to turn this around to you, Sebastian. I mean, you're the only one here who actually is regularly talking to a ton of different clouds. I mean, does it, I know that you can take a framework like Scalar and you can somewhat patch over some of the differences in the clouds, but how much grief and how painful is it for you, how much time do you spend sort of dealing with differences between clouds where if they were sort of somewhat more standardized, you could spend more time adding value to your management system.  And that's one of the challenges that we have when we build the Skater Management Platform on top of multiple clouds. And that's one of the reasons we don't want to work with the vSphere or like any of the things that are totally not cloud, is they just don't have the same objects, the same constructs. And so that makes it very difficult to build a common management layer and enable application portability because the highest common denominator is really, really low. We have a really distinguished panel here. At this point, I'd like to open it up for questions. We've got one coming. Hi, I'm Kamesh with Dell. So underlying all of the discussion that's going on here, there's this concept of application mobility, right? And it's not just the VMs that you're moving from one cloud to the other, but the underlying network and the data that's associated with it. It sounds like every one of you has solved this in a different way or in a customized way. Is there like a standard toolset or somebody building something that enables application mobility? Are you talking about how you kind of lift and shift in application from one place to another like this? Isn't that kind of one of the use cases where you want to move an entire workload with all its components, whether it's VMs or databases or networks or load balance or what have you. I mean, it's not just one VM, right? We all know that workload is a complex pieces that all glue together. So that's my question. Jen, I was talking about the data gravity. I don't know if you want to expand on that or? Well, I don't think it's moving workloads that we're talking about with this hybrid cloud use case. It's having multiple execution environments where a single application can run in. So you have one application or one service that might be talking to another service that's in a different place, in a different cloud and abstracting that away into something that basically looks like reusable pieces, reusable deployment environments that you can just deploy code into and having all the glue around the discovery of being able to find other services that you need to talk to, having all the operational tooling around it so that you can monitor when things go wrong and the deployment tool chain so that when you have a new piece of code, you can get it from your source control system out into these different execution environments. That's, I think, the use case that we're talking about. I understand that use case. I'm specifically talking about the Zynga's example for example, right? You had a whole bunch of stuff running on AWS. At some point, you said, this is not gonna be viable for us, I wanna bring everything back in-house. And maybe after some time you say, okay, I'm gonna put some stuff back in Amazon. It means to me that you have to move stuff. Right, so I don't believe at all in any lift and shift fundamental technology. I think that, and to describe more deeply exactly how Zynga was moving applications, it would be at a minimum about, and especially when we started, about a seven week project with all the developers that know the underpinnings of the game and then my operations team who are providing infrastructure as a service collaborating for seven weeks with a project manager and all that work. What it's, with a lamp stack in our particular case, what that involved, because this is a game, right? This is people expect their freemium, purple cows, et cetera, and you can't take that away from them without impacting customer service, right? So the way we did it is with the MySQL databases, you add another slave in the private cloud that's real-time streaming everything. You make sure that's stable. Then you add the memcache layer internal to the private cloud and then you need to pre-seed that memcache and then it would normally take us and then you can light up all the web servers and be ready for effectively a DNS cutover, right? Or we would re-register the game URL and Facebook. We got it down to taking seven minutes of downtime to actually flip that over. So it's not lift and shift technology that's a lot of hard work to pivot it. That's my point. Excellent question. I think I made my point, right? Yes. That's the difficulty of... Yeah, it's not migrating VMs ever in our case. And I don't think that's sustainable, repeatable. You want to have the recipes and you want to have... You need to think about that hard cutover and the fallback process as well. I mean... A wide area of migration is a fool's errand. I think about this in two ways, Kamesh. So the first is that there's a variety of frameworks and there's no way to get away from that because if you want a lot of control, you're going to use Chef and Puppet or build your own framework that gives you very low level control. If you want less control and sort of more containment, you're going to use a platform as a service system or possibly something like Scalar or RightScale so that you can have templatized deployments. And then those templatized deployments taking aside the problems with data synchronization should be the same when you deploy the application on Cloud A or Cloud B, you want the same thing to happen on both. And so you're going to use a variety of frameworks. There's no way to get away from that depending on what level of control or ease of use you are for that application on a spectrum. And then the second thing is that if we're moving towards a model where IT is more like the electricity model, we've never gotten to the point where the individual appliances at the edge of the electrical system don't have fundamental differences in how they plug into electrical system. They all have different power adapters because they all have different power needs depending on the application of that appliance. But what they're able to do is because all the electricity looks the same is they can all build that interface to that same standard. And that's really what I try to promote a lot is I think that Amazon and Google really have made a de facto standard. And if we all kind of hub off of that reference architecture, then a lot of the problems around dealing with the application interface to those start to go away because you just build one application interface for that one reference architecture. And then as you go from cloud to cloud to cloud, there's a lot less variance. Any other questions? Hey, it was Hamlin, Trevor from RMS. We're going down the hybrid cloud approach right now where we'll have a majority of our stuff in the private and then we'll be bursting out into public. And so I know that it sounds like there's a lot of intricacies in doing all this, but I would appreciate if some of you guys could explain some of your war stories from basically the failures that you've seen or maybe the successes where you've seen this has worked. Excellent question. I hope we didn't ruin your day, Trevor. So some of the war stories, not this particular one isn't mine personally, but I had briefly mentioned it before where an acute workload drove the data persistence layer into a private cloud so as to take advantage of high-end hardware. And still to preserve, in this particular case, the public web servers which are customer facing or user facing, you still needed to have the caching layer and reed slaves to the Mongo cluster there. Now, what became volatile and broke down on occasion is that especially with Mongo and then this particular customer was using kind of lock-enabled rights, they would lock the reed slaves and thus the rights had been persistent that if there is a wobble in the direct connect or anything that's feeding where the right master is getting its data in the private side and that kind of can clog up in the public side or if there's a genuine failure and a master needs to be promoted in the public side, the war story is that the performance difference is going to crush the right performance and the application is gonna go down and you have the thundering herd problems that Ariel mentioned. So there's a lot more moving parts there and just as an operations person responsible for availability, I always try to keep the environment as simple as possible so that doesn't always mean hybrid distribution of an application for me. Is this still true with like dedicated instances and being able to get all SSD instances as well? I mean shouldn't that get rid of a lot of the noise in the neighbor problem? This occurred right before those became available and I believe they have since solved their problem and with that technology, but still having it persistent beyond the SSD node. So yeah, I guess they've got clusters that have solved that. Junha? So leading the cloud networking group, obviously I have a particular gaze but I do see a reliance on specific network topologies to be a real impediment to being able to split the difference. Often also I see a strong reliance on using numbering as a grouping mechanism. So knowing that a given sitter is going to represent a given set of machines and when you move into cloud, even when you move into private cloud, you start to tear away those beliefs, right? They become extracted away, you define groups and new mechanisms. And so a lot of times I've seen that baked into applications and makes it very, very difficult to move even into private cloud. Yeah, I mean the one thing that I've seen that's proven so far is when you have multiple applications and you're just gonna pull one off at a time. So you're using hybrid cloud in the sense that you're not trying to stretch one application across two different clouds but you actually have different applications in different locations like Zynga or Ubisoft where they basically said, okay, this game is flat lined. Let's take it back to private cloud where we can cost optimize it and that works pretty well because we've got lots and lots of applications out there already that communicate over the internet so there's no, tends to be no hiccups. The latency issues tend to be really within kind of an application grouping. All right, we'll take one last question otherwise. Oh, are we out of time? All right, okay, they're throwing things at me. All right, let's give the panelists a warm thanks.


Hello, I'm Michelle Baxter, Safe Work Australia's Chief Executive Officer and a proud 2015 National Safe Work Month Ambassador. I'm excited to launch 2015 National Safe Work Month and the Virtual Seminar Series. National Safe Work Month is a time to focus on health and safety in your workplace. The 2015 theme is Be Safe, Be Healthy, Because. Now we've deliberately left the theme open so that you can think about your own reasons for being safe and healthy at work. There are many reasons for being safe and healthy at work. What's important to you? As the CEO of Safe Work Australia, I understand the benefits of supporting the mental and physical health of your employees and colleagues. So I want you to be safe, be healthy, because. Your actions, no matter how big or small, can support mental and physical health. The most recent estimates indicate that workplace injury and illness costs the Australian economy more than $60 billion annually. But more important is the human cost of injury to workers and their families, their friends and their colleagues. So I also want you to be safe, be healthy, because. Your family and loved ones are waiting to welcome you home. National Safe Work Month is all about participation and there are many ways you can do this. You can show us how your organisation is participating in National Safe Work Month for your chance to win a participation reward valued at up to $5,000. You can also get involved in the virtual seminar series, a series of free online seminars held throughout October. The seminars showcase the latest thinking, innovation and research in work, health and safety. I encourage you to look at the Safe Work Australia website for more information about these events and for other ways that you can participate. There's also a free smartphone app so you can access National Safe Work Month resources and events at your convenience. Please do take the time to focus on health and safety in your workplace, especially during October. Have a conversation about your reasons for keeping safe and healthy at work and how you can make your workplace safer. And remember, be safe, be healthy, because, well, for me it's so we can make a difference to people's lives. What's your reason?