Well, it's beginning to look a lot like Christmas everywhere you go. Only four days till Christmas. I hope you got your shopping done. You know, we have got one of our favorites coming up on today's program. We're going to discover how our God is the God of miracles. Our very own Sydney Goldman had a chance to sit down with Dr. David Chotka. He's one of our favorites. He is a pastor with the Christian Missionary Alliance of Canada. And he has a remarkable story to share with you about the birth of his son. And it was only through God that he and his wife were able to receive their miracle. It's an amazing story of prayer, faith and healing and proves that God can do anything. You know, this is a time of miracles. I mean, Jesus being Christ incarnate is a miracle, but we need a miracle in our daily lives to the miracle of knowing him of the new birth, but also miracles like what David, Dr. David and his wife went through. We need miracles like that as well, and you're going to hear all about that. We're also going to have a throwback Thursday featuring a favorite Christmas performance by Cindy Morgan. And I just want to talk a little bit about Christmas with you. You know, Christmas is one of our all of our favorite times a year. At least for me, but it's interesting how it changes a little bit. You know, we we've our kids are grown and there's grandkids now. Of course, we're sharing them with the other in-laws. And and so Christmas is going to be a little different for our family. We're going to all get together probably a couple of days after Christmas. We haven't set that date yet. But we are going to get together and have a great time. But that Christmas day, it changes. And maybe it's changing for you. Maybe you don't know Christmas like you used to.  But now you're the older one. You're the you're the grandpa or grandma. Maybe it's not quite the same. And it can be a time where we struggle a little bit with those memories. But listen, God has great and good things for you. If you need prayer, we have a prayer line to just help you to have that time where you can get together with another person of faith, another person with faith, another person who loves Jesus and just pray together and share together and encourage one another. So why don't you call a prayer partner? Again, the numbers there on your screen. You can call any time and you can be blessed and you can just rejoice with someone else who loves this time of year because this is the time when Jesus came. Well, right now we're going to go to Sidney's interview with Dr. David Chaka. Dr. David Chaka is a pastor for Christian Missionary Alliance of Canada for more than three decades, who was taught internationally on prayer principles. And Dr. David, we're so glad to have you with us again. Thank you. It's a joy to be here and it's an honor to be on your program. Yeah, well, it's an honor to have you with us. And you know, Dr. David, one thing that, you know, you have a book called Healing Prayer that you recently wrote with your friend and colleague, Maxie.  Can you share that with our audience? Yes. My wife and I got married and, of course, she's been the first year laughing and giggling, right? That we decided to try for her family. And then we conceived and well, I should say she conceived and the bottom line was about, about a month and a half and she started to get itchy. And so she went to see the doctor and he said it was colostasis of pregnancy, which is where your enzymes levels rise up and your skin gets kind of itchy and so on. But usually that happens in the third trimester and this was the first. And so she went to specialist after specialist after specialist to try and figure out what this was. And they determined that it was an extreme case of colostasis of pregnancy and we did not know. It had to do with liver function and actually we discovered that as her levels of estrogen rose during the pregnancy, her liver first of all shut down, then began to systematically destroy itself. And it was horrible to watch. Now, both she and I are pro-life, but it's one thing for a guy to be pro-life. It's another thing for a lady who's carrying a child to be pro-life and her life was endangered. So we prayed. We had friends pray. There was a, there were two ladies in the church, three ladies in the church who would pray with my wife and it was the only relief that she ever got. And it was an interesting thing because we had seen miraculous healings and we cried out to the Lord for a miraculous healing and in the middle of that, the only thing that happened was consolation and comfort in the middle of this. And when those three ladies would call her once a day, the stings, the bee stings, she was feeling like bee stings on her hands and bee stings between her breasts and on the flat of her feet. And she could get no sleep and the only thing that would stop the trouble was having an ice bath, if you can imagine that. She'd get an hour of numbness and she'd be able to rest for an hour and then she'd wake up with this terrible kind of horrible thing. Well, these three ladies felt commanded with God to pray for her and they would call, if they couldn't get in person, they would do the three-way call on the phone and they'd talk to her and pray with her. And the sense of stinging would diminish for about four to six hours and she could sleep. But there was no healing. Meanwhile, I'm an assistant pastor in a local church and it was a church plant in the city of Vancouver. I don't know what costs are like down here, but up there that's the most expensive city in Canada and it's one of the most expensive cities in the world. And we were renting an area to hold church services. We were renting an office and we were paying the secretary and the lead pastor and me. And we didn't have enough people to be able to pay the bills and so it became clear. I was the associate. I was going to have to go. I was looking for a new church. The lead guy was on sabbatical. Why was covering off this thing? He was praying about what to do with the whole church situation. So I was responsible for the congregation. I was responsible for the oversight of that and my wife was desperately ill and terrifically in pain. And she wound up in the perinatal unit of Vancouver Grace Hospital. Thank the Lord it was actually a Christian hospital. There was a Salvation Army chaplain who came and saw her and prayed with her and it was an amazing thing. But I mean she was diminishing and losing ground and it was getting worse and worse. And I knew that at this point this was going to be a very, very tough pregnancy. So they did an amniocentesis and they tried to draw the fluid out to see if their lung development of the child was sufficient. And after they did the amniocentesis the fluid came out dark green which meant that my son was swimming in poison. And the prognosis was that he would be blind, he would be deaf and he would have cerebral palsy. And you gotta know that changes the way you think and she was not going to take the child. She looked at me and she said, life is precious and if I have to die for the child to live, I will. And you have to know that's a very, very different way of starting out your life with pregnancy. Very different way. It changes you. She almost died. So here's what happens. I know I have to provide. The liver levels are getting higher and higher and higher. Her liver is shut down. Now it's destroying itself. And we have friends who are praying and there's this one lady named Margo who stood beside her and with her. And then an opportunity came to go 2,000 miles across the country to another church where I could become a lead pastor and pay the bills. And I thought, I'm gonna have to provide. And we prayed that I'd be given opportunity and the liver enzyme levels came down into high normal. It was the craziest thing. Not a healing, but you know, how do I say this? Improvement. And she stabilized and so Margo stayed with her. I got in a plane and I flew to Ontario. And I wound up having the interview and the interview went well on the Sunday night and the next day was Monday and they were saying, okay, you're on Vancouver time. That's three hours earlier. That's like Seattle. You're going to need to sleep in and then you need some time to pray. And then a guy here named Bob, he's going to give you a drive around the neighborhood. I said, oh, okay, that's fine. So Bob gets in the car and he drives around and it's the same weather as here. It was August and it's usually 90, 95 and it was like 68 Fahrenheit and he was cold and he had a short sleeve shirt on. He said, well, David, would you mind if I stopped at home and got myself a change of shirt, put a sweater on? I said, no problem, Bob.   Now the interesting thing was it was from a medical association. He was a sponsor of some sort of medical association. My wife was 31 weeks pregnant and they had just said this to us. They said, at 32 weeks, we're gonna put your wife on the drip and that's gonna induce labor and it's the best chance the child has of passing through the uterine canal, kick starting the lungs. We don't know if he has lung capacity to be able to breathe but if he goes through the uterine canal at 32 weeks, it might just give him the ability to be able to breathe. Okay, that's fine. 31 week old preemie is coming up on 32. So I'm in the living room and I look at this letter. It's an open form letter. It's not personal. I look at it and there were two paragraphs about 31 to 32 week old gestational preemies, males, who would pass through the birth canal. And the article said, it's the best chance the child has for breathing but there's one problem. There's a 35 to 40% higher incidence of cerebral palsy because the skull case is softer and when the child passes through the uterine canal, it puts pressure on the bones and it presses in on the brain and 35 to 40% of those male 32 week old preemies wind up with cerebral palsy. So I read these two paragraphs. I finished the candidates. They said they're going to recommend me to the church. In fact, I did go to serve there. I flew back across the country. I landed in Vancouver Airport. I got in the car. I drove straight to the hospital and there were nine attending physicians in the room, nine of them. They had pulled their hair out trying to figure out what was the best thing to do because they were aware of all the exigencies. She was getting weaker and her life was in danger. We're not going to give up the child. The child was in danger and this was bad. And they looked at us and they said, we've come to a decision. We are going to put Elizabeth on the drip. It's going to take three days. And at the end of three days, labor will be induced. The child will pass through the uterine canal. It will kick start the lungs. In those days, surfactant was experimental. Not too many people knew about the way to make lungs work if you're young. It was just a brand new drug. Anyway, they said, that's the best chance he has. And they said, do you have any questions? I said, only one. I said, what's that? I said, isn't it true that with 31 and 32 what we call gestational male premies when they pass through the uterine canal based upon the drip, there's a 35 to 40% higher incidence of cerebral palsy and Sidney, I sounded so smart. I mean, I'd read two paragraphs in a form letter here sitting because somebody changed his shirt in a church 2000 miles away. And everybody in the room, their jaws all dropped. And they looked at me and they said, we don't know what to say. And they left. Now, my wife will tell you this. The attending physician, he should have been on the cover of gentlemen's quarterly. He was a perinatologist, always helping pregnant ladies who were in distress. All of them liked being with him because he looked good.  And he looks at the two of us and he says, we're doing a C-section next Tuesday. So what are the odds? We're talking about a 35 to 40% higher incidence cerebral palsy. She was very weak. Her son was premie. They were expecting a child who was going to be one to two pounds. So I'm in the place and I'm there when the child is being born and I almost hyperventilate in the hallway. And this is funny how he stopped hyperventilating. I sang hymns in the hallway.  I said, there's all kinds of people and there's got to be 20 people in this room.  The case is so unusual. So I walked up to the two attending physicians and I said, there's one group missing. And he said, what? I said, the Salvation Army Brass Band. Everybody else is in the room. And then he looked around and he dismissed everybody. Then the child came out four pounds, five ounces, not two pounds. He wasn't blind. He wasn't deaf. He wasn't cerebral palsy. He got married about three months ago and he's well and he's healthy and I'm very thankful. But in that case, it was pathway to a remedy not miraculous healing. And Jesus was the one who sent me across the country to sit in a living room when somebody was changing his shirt so I could have two paragraphs of medical information to address the medical constituency about the thing that could be done to bring healing to my wife and my family. So I believe that medicine and miracle intertwine. They are not separate from each other. They overlap beautifully. Isn't that amazing? What are the odds? I just remember Dr. David reading that in the book where I was just like, oh my gosh, just how God orchestrated everything and it's through the prayers what he's like divine appointments and he's like, I'm going to situate things because he knew what he's doing. What's your son's name? My son's name is Benjamin. Benjamin, wow. We picked Ben because it was a gentle name and a strong name. Actually, his name came actually, I don't know if you know this one.  And my wife could have died. She was that close.   That's so beautiful. And you know, Dr. David, I just feel like can you just take a moment and just pray? Just pray for somebody because we have so many people that call in to Cornerstone and they are just awaiting a miracle. There's a medical diagnosis that they just can't even comprehend. Can you just take a moment to pray?  Okay.  Jesus, my prayer, first of all, is that anyone who is crying out to you, you are more than able to touch a physical body. You are more than able to heal. You are more than able to deliver. You are more than able to transform. And sometimes you lead people in a pathway to a remedy. Somebody has dedicated 30 years of their life to research and they have discovered a way to bring health and safety to someone. Father, I don't know what the vagaries of life are for the people watching this broadcast but I know this.  So heal by miracle, heal by medicine and when we do not understand why something is happening, intertwine that with an awareness of mystery around who you are and make us more like yourself in Jesus' name. Amen. Amen. Dr. David, it is such a joy.  Should I laugh now? Yeah, you just have like something I just really, I just really appreciate you.  That is so incredible and beautiful and I know that God is using your story. You know, I just think about we overcome by the blood of the Lamb and the word of our testimony and what you've shared. And loving not our life under the desk. Don't forget the last part. You're right. That's a really important part too. Yeah. Thank you, Sydney. Yeah. Thank you so much, Dr. David. Hope happens here at Cornerstone Television. All this month, we're offering a joy-filled DVD, Christmas with the Chosen, The Messengers for your best gift to the ministry. Gather around the manger with loved ones and experience the first Christmas through the eyes of Mary and Joseph. Follow the young couple as they take the long road to Bethlehem and prepare for Jesus' birth. Plus, enjoy an extraordinary lineup of musicians performing both new and classic Christmas songs from the set of The Chosen such as Phil Wickham, Brandon Lake, Maverick City Music, Cain and many others. Thank you for your generosity that makes the ministry of Cornerstone Television possible. Request your Christmas with the Chosen DVD when you give this month. Call<PHONE_NUMBER> or give online at ctvn.org slash donate. From all of us here, we wish you a very Merry Christmas. You make a difference every day. You share the Gospel. You inspire believers. You offer wholesome family entertainment and so much more. We're thankful for you. And as year-end approaches, we're sharing some strategic giving options that may give you a tax break, too. First is a rollover gift from your IRA. If you're 70 and a half years or older, you can give up to $100,000 from your IRA and not add a penny to your taxable income. If you're 73 or older, it counts toward your required minimum distribution. Second, consider donating appreciated assets. Donate stocks or property you've held for more than a year and you often get a tax deduction. Plus, avoid the capital gains tax, too. For more information about these or other strategic giving options, contact us at<PHONE_NUMBER> or email info at ctvn.org. Thank you for changing lives and sharing hope together with Cornerstone Television Network. Wasn't that a great story that Dr. David Chotka shared about his wife and the birth of their child and the hard things of faith of struggling through, but a medical answer.  Sometimes it comes with just David happening to see something that gave him the information he needed to pass along to the doctors, remind them really and cause a great outcome. That's, I love that because sometimes we're just waiting for the miraculous supernatural thing even though that is supernatural, too. But let God be God. Let him move in the way he wants. And I have a scripture about that at Acts 1038. And this is Peter sharing with those that don't know Christ. He is a, he is their Gentiles and he is sharing with them. But listen to what he says. You know of Jesus of Nazareth, how God anointed him with the Holy Spirit and with power. And how he went about doing good and healing all who were oppressed by the devil for God was with him. So let's talk about this a little bit. First, it says you know of, you know of Jesus Christ. See, they didn't know Jesus Christ yet. They didn't know Jesus Christ as Lord and Savior yet. That was about to change, but they didn't know him yet. And they were about to meet the one that had created them. They were about to meet the one who had came and lived and taught and died and rose again for them. But so far they just knew of Jesus Christ. Well, we know the difference, don't we? We know so much more about whether we know the president of the United States or we know of the president of the United States. Everybody knows of. So the question for you today is do you know Jesus Christ? Now we're here. We are four days before we celebrate his birth and that's an important question to answer. When you celebrate Christmas, are you going to know the one that Christmas is about? Are you going to really know him like a person, a relationship with him? Are you just going to know of him? Is he just going to be the star on the tree is because of him? Do you know the story, right? And if anybody asks you why are we celebrating, you know why we're celebrating. But do you know? I want to ask you, do you know Jesus Christ today? And then it says that he went about healing all who were oppressed and afflicted of the devil. Do you ever feel oppressed and afflicted now? He did a lot of amazing things. He healed people that were sick. He healed people that were demon oppressed. And he did all manner of miracles, including walking on the water, all kinds of things that he did because he had the power of God within him. He was God in the flesh. God incarnate. Okay. So again, somebody there like these people that Peter's preaching to could have heard those things. They could have heard, do you know what's happening down in Jerusalem there? Do you know what's happening in Galilee? Have you heard of this Jesus? And people say, wow, tell me more. But they made a decision in just a few verses later where they came to a saving knowledge of Jesus Christ. So I want to ask you, do you have that saving knowledge? Miracles are great. Miracles like what David experienced are fantastic. They're wonderful miracles. I hope you've experienced miracles like that in your life. But the biggest and most important miracle is a miracle of the new birth in Jesus Christ. So if you've never experienced that miracle, why don't you take that step today? All you have to do is open the door of your heart and say, Lord, come in and be my Savior and be my Lord. Forgive me of my sins. What are sins? Well, they're breaking God's command. Guess what? I've done that. You've done that. Everybody on earth has done that. We're all in the same boat. Nobody's holier than now. We are all sinners who need to be saved by grace. And so you just say, Father, forgive me of my sins through the blood of Jesus Christ. Come into my life and be my Lord and Savior. And if you want to do that, if you've prayed, maybe just you said that prayer with me, that brief prayer. Can that really make a difference? Well, yes, it can because all it matters is opening that door. And when you open the door, then the Holy Spirit rushes in and it's called regeneration. He changes us. So if you've just prayed that prayer or if you want to know more about it, go ahead and call our prayer partners. They're standing by just a few days before Christmas here that they might be able to pray with you and they might be able to introduce you to the one that Christmas is all about. So please don't let this Christmas season pass without knowing Jesus as Lord and Savior, without knowing the one who went about healing all who were oppressed and afflicted and the one who was God incarnate. Well, we have something we like to call throwback Thursday and we are going to have a throwback to Cindy Morgan who's going to sing a Christmas favorite. I know you know it. Go tell it on the mountain. Here's Cindy Morgan. Tell it on the mountain over the hills and go tell it on the mountain that Jesus Christ is born down in a blessed Christmas morn.


Between 1832 and 1844, Joseph Smith and some of his closest friends recorded at least nine accounts of Joseph's first vision experience, given on different occasions to different audiences. The most detailed of these accounts, written in 1838, has been published in a volume of scripture called The Pearl of Great Price. What you're about to see draws upon all of the written first vision accounts to provide additional perspectives and insights into this remarkable event. My name is Joseph Smith. When I was young, there was in the place where we lived an unusual excitement on the subject of religion. My mind became seriously impressed with regard to the welfare of my soul. I pondered many things in my heart. I felt too mourn for my own sins and the sins of the world, but upon inquiring about the plan of salvation I found that there was a great clash in religious sentiment. This was grief to my soul. I knew not who was right or who was wrong, and I considered it of the first importance that I should be right in matters that involved eternal consequences. I wanted to get religion too. I wanted to feel and shout like the rest, but could feel nothing. I was one day reading the epistle of James. If any of you lack wisdom, let him ask of God, and it shall be given him. It seemed to enter with great force into every feeling of my heart after I retired into the place where I had previously designed to go, having looked around me and finding myself alone. I kneeled down and began to offer up the desires of my heart to God. I cried unto the Lord for mercy, for there was none else to whom I could go to obtain mercy. It was the first time in my life that I had made the attempt to pray vocally. Immediately I was seized upon by some power which entirely overcame me and had such astonishing influence over me as to bind my tongue so I could not speak. My mind filled with doubts and all manner of inappropriate images. It seemed to me for a time as if I were doomed to sudden destruction, but exerting all my powers to call upon God to deliver me. Lord! My mouth was open. Have mercy on me. And my tongue live right. Forgive me. Joseph. Joseph. This is my beloved son. Hear him. Joseph, thy sins are forgiven thee. Behold, I am the Lord of glory. I was crucified for the world that all those who believe on my name may have eternal life. The world lies in sin. They have turned aside from the gospel and keep not my commandments. They draw near to me with their lips while their hearts are far from me. They teach for doctrines the commandments of men. Having a form of godliness, but they deny the power thereof. Go not after them. Wait patiently. At a future time the complete truth of the gospel will be revealed to you. When the light had departed, I had no strength. Peace and calm filled my mind. My soul was filled with love. For many days I could rejoice with great joy, and the Lord was with me. I had received a promise that the fullness of the gospel should at some future time be made known unto me. And I had actually seen a light. And in the midst of that light I saw two personages. And they did, in reality, speak to me. I knew it, and I knew God knew it. And I could not deny it.


We continue on today with Chapter 8, The Undivided Will of the Sonship. Can you be separated from your identification and be at peace? Dissociation is not a solution. It is a delusion. The delusion will believe that truth will assail them and they do not recognize it because they prefer the delusion. Judging truth is something they do not want. They perceive their illusions, which block knowledge. Help them by offering them your unified mind on their behalf, as I am offering you mine on behalf of yours. Alone we can do nothing, but together our minds fuse into something whose power is far beyond the power of its separate parts. By not being separate, the mind of God is established in ours and as ours. This mind is invincible because it is undivided. The undivided will of the Sonship is the perfect Creator being wholly in the likeness of God, whose will it is. You cannot be exempt from it if you are to understand what it is and what you are. By the belief that your will is separate from mine, you are exempting yourself from the will of God, which is yourself. Yet to heal is still to make whole. Therefore to heal is to unite with those who are like you, because perceiving this likeness is to recognize the Father. If your perfection is in Him and only in Him, how can you know it without recognizing Him? The recognition of God is the recognition of yourself. There is no separation of God and His creation. You will realize this when you understand that there is no separation between your will and mine. Let the love of God shine upon you by your acceptance of me. My reality is yours and His. By joining your mind with mine, you are signifying your awareness that the will of God is one. God's oneness and ours are not separate because His oneness encompasses ours. To join with me is to restore His power to you because we are sharing it. I offer you only the recognition of His power in you, but in that lies all truth. As we unite, we unite with Him. Glory be to the union of God and His holy sons. All glory lies in them because they are united. The miracles we do bear witness to the will of the Father for His Son and to our joy in uniting with His will for us. When you unite with me, you are uniting without the ego because I have renounced the ego in myself and therefore cannot unite with yours. Our union is therefore the way to renounce the ego in you. The truth in both of us is beyond the ego. Our success in transcending the ego is guaranteed by God and I share this confidence for both of us and all of us. I bring God's peace back to all His children because I received it of Him for us all. Nothing can prevail against our united wills because nothing can prevail against God's. Would you know the will of God for you? Ask it of me who know it for you and you will find it.  Ours is simply the journey back to God who is our home. Whenever fear intrudes anywhere along the road to peace, it is because the ego has attempted to join the journey with us and cannot do so. Sensing defeat and angered by it, the ego regards itself as rejected and becomes retaliative. You are invulnerable to its retaliation because I am with you. On this journey you have chosen me as your companion instead of the ego. Do not attempt to hold on to both or you will try to go in different directions and will lose the way. The ego's way is not mine but it is also not yours. The Holy Spirit has one direction for all minds and the one He taught me is yours. Let us not lose sight of His direction through illusions for only illusions of another direction can obscure the one for which God's voice speaks in all of us. Never accord the ego the power to interfere with the journey. It has none because the journey is the way to what is true. Leave all illusions behind and reach beyond all attempts of the ego to hold you back. I go before you because I am beyond the ego. Reach therefore for my hand because you want to transcend the ego. My strength will never be wanting and if you choose to share it you will do so. I give it willingly and gladly because I need you as much as you need me. And from the workbook, Lesson 60. These ideas are for today's review. God is the love in which I forgive. God does not forgive because He has never condemned. The blameless cannot blame and those who have accepted their innocence see nothing to forgive. Yet forgiveness is the means by which I will recognize my innocence. It is the reflection of God's love on earth. It will bring me near enough to heaven that the love of God can reach down to me and raise me up to Him. God is the strength in which I trust. It is not my own strength through which I forgive. It is through the strength of God in me which I am remembering as I forgive. As I begin to see, I recognize His reflection on earth. I forgive all things because I feel the stirring of His strength in me. And I begin to remember the love I chose to forget but which has not forgotten me. There is nothing to fear. How safe the world will look to me when I can see it. It will not look anything like what I imagine I see now. Everyone and everything I see will lean toward me to bless me. I will recognize in everyone my dearest friend. What could there be to fear in a world that I have forgiven and that has forgiven me? God's voice speaks to me all through the day. There is not a moment in which God's voice ceases to call on my forgiveness to save me. There is not a moment in which His voice fails to direct my thoughts, guide my actions, and lead my feet. I am walking steadily on toward truth. There is nowhere else I can go because God's voice is the only voice and the only guide that has been given to His Son. I am sustained by the love of God. As I listen to God's voice, I am sustained by His love. As I open my eyes, His love lights up the world for me to see. As I forgive, His love reminds me that His Son is sinless. And as I look upon the world with the vision He has given me, I remember that I am His Son. So today we open our hearts, our minds to the undivided will of the Sonship. Today we offer unity of mind to all our brothers and sisters, offering it and receiving it ourselves, offering and receiving it in the name of Christ. Alone we can do nothing, but together our minds fuse into something whose power is far beyond the power of its separate parts. This mind is invincible because it is undivided. The undivided will of the Sonship is the perfect Creator, being holy in the likeness of God, whose will it is. Today we accept this will. The recognition of God is the recognition of our self as the Christ. There is no separation of God and His creation. God's oneness and ours are not separate because His oneness encompasses ours. Our union is therefore the way to renounce the ego except the Christ, to know the will of God is my will. God is our home. God is our source. We cannot see apart from God. We will see another world when it is our desire to truly see. We are reminded of this today as we review these precious ideas. God is the love in which I forgive. God is the strength in which I trust. There is nothing to fear. God's voice speaks to me all through the day. I am sustained by the love of God.


They were cheering when he jumped, but he did not come up again. Turnpike Troubadours Before the devil and those were dead, I remember one time growing up when my best friend David asked if my grandparents were still living. His own grandmother had passed away a few days earlier, and I think he was just looking for someone to talk to about it. Do you still have your grandparents around? David asked from the sidewalk, pausing for a moment and looking over at me. Yes, I said, leaving it at that. Of course they were around, everyone was, but I could never tell, that's what mom and dad had said. No one in my family has died for quite some time, at least not in the traditional sense.  Like my uncle Carl for instance, he had developed bone cancer in his late 70s, and day by day it wore down his body until by the end he could nearly pass for a skeleton. One day as he lie in bed with his family circled around, his heart finally gave in and stopped fighting. He took one last breath, held his necklace tightly in his right hand, and then he was gone. My mother held me close and watched as I stared over at his vacant body. It's okay dear, he'll be back tomorrow. And he was. Sure enough the next morning uncle Carl was the first one at the table for breakfast. And what's more, he was young again. Uncle Carl, I yelled happily, running over into his arms.  Is it gone? It's gone buddy, he replied warmly. I didn't ask, and I'm not sure that even he knew the exact age he now was, but he appeared to be in his early 30s, it's different every time. I like to think that maybe you come back at the age you were most happy, but that's just speculation really. None of us know why you come back the age you do, all we know is that you always come back. That is until recently. My great grandfather Isaac was the strongest of us. He had been a coal miner and had seemingly come back in his best physical condition. His shoulders were broad and wide, and his arms thick and muscular. He still wore the same dusty pants and cloudy gray jacket that no doubt came from the old days. He would even wake up wearing a cracked mining helmet that looked to be over 100 years old.  He would yell through the house, throwing his helmet angrily against the wall on the other side of mine. I always chuckled. That was one of the peculiarities with the family. Once you died and came back, you always woke up every morning wearing the same set of clothes. It seemed to be chosen at random, as far as we could tell. I was particularly embarrassed one day when upon entering my great aunt Natalie's room, I found her wearing a bright red negligee and a black top hat. Needless to say, I left the room immediately. I'm not going to ask about that one. But I digress and back to the main story. I'm 18 years old now and was due to leave for college this week when Grandpa Isaac went missing. Maybe more accurately still, he never showed up. That morning when our family sat around at the breakfast table, I noticed his seat was empty. I wouldn't have thought much of it had it not been for the surprised look of fear from my remaining family. I'll go, his wife had said, quickly rising from her chair and leading three or four family members upstairs to his room. He wasn't there either. We held the funeral three days later. I had never been to one before. His wife, my grandma gave the eulogy. And I wish I could remember more of what she said but I was too shaken up. I had never experienced death before and as she stood up in front of us, I only recall her saying one thing, death seems to have found us again. After it was all over she took my hand. Come with me, she said softly, leading me up the old creaky stairs that led to the attic. She unlocked a dark wooden chest I had never seen before and pulled out something small that was wrapped tightly. You need to know. That night I walked with her from door to door and window to window. At each she would recite a set of words. I'm still learning Latin but upon asking she told me the approximate translation. Death is not welcome here. The chest had contained an ancient vial of liquid. Guy blues, forest greens and taxicab yellows seemed to swirl endlessly and with purpose.  She would recite the words and then close her eyes, dip her index finger into the old vial and use a small portion of the liquid to draw a small symbol. The liquid seemed to glow more brightly upon contact with her skin and when the symbol was drawn completely it would shine as well. We double checked that every door on window was covered. They were. Upon our completion the rest of the family was then in charge of taping up the windows with dense black paper that made it impossible to see through to the outside. That night as the last of the sunlight began to fall over the mountains we locked the front door and checked it twice. My grandmother sat down next to me. There's one more thing. What is it grandma? She looked towards the closest window and then back at me. Whatever happens tonight, whatever you hear, never look outside. It's better you don't think of such things. Just promise me. I looked back at her. I promise. Good. She smiled. It's going to be alright dear. That evening as my family and I stayed up to keep watch, the night seemed to pass like any other. We played cards and made poor attempts at joking, trying our best to keep our minds off the current situation. And then the knocking started. First there were three knocks at the front door. My family grew silent. Then there were three more on the kitchen window close by. And then suddenly there was a rapid more powerful knocking that seemed to strike every window and doorway in the house at once. Something wanted desperately to get inside. After a time it stopped. There was quiet again. A minute or two passed and the calm was interrupted by a great howling. Our heads looked up almost in unison towards the ceiling as we strained to hear the distant noise. It came from directly above us, far off in the sky. The howl could be heard echoing throughout the house and seemed to shake the very foundation. And then as quickly as it had started, it was over. It was like none of it had ever happened and a sense of relief came over my family. We had survived. But there was a problem. Later that night I awoke to the sight of two blue eyes staring down onto me. It was my nephew, Iwan.  I whispered angrily over at him. And then he began to sob, first softly and then uncontrollably. I know I shouldn't have, I know I shouldn't have. He cried, bearing his head into my chest. It's okay, I said warmly. It's gone now. The crying stopped for a moment as he looked up at me innocently. When the thing knocked on my window, the paper must have not been taped on good enough and my heart filled with dread as he continued. And I saw its face looking back. What's going to happen now? He asked. I looked into the bright blue eyes that belonged to my eight-year-old nephew Iwan. And I answered him truthfully, I don't know.


No, here, I'll do this, we'll make it so that I'm, yeah, hey, how you doing, guys? This is, there's a lot of you in here. This is fantastic, thank you all for coming. So, if you can't tell from the CalSE terminal window up there, this is the DevOps panel. We have with us representatives from Chef and Puppet and DevStack. What are people using DevStack? They are using DevStack, so we were supposed to have also Juju represented, but I think it's gotten too much traction and so they're all off running, off doing amazing things. So, unfortunately we won't be able to hear that story. So, I'm Monty and you have the benefit of having me moderate this thing. So, any complaints you can direct them my way afterwards. You can do that in the form of buying me beers. It's the best way to complain about any misuse of this forum that I might do. So, you just buy the beer that you don't like or? No, buying me beer is, I hate it. So, that's the best way to get back at me for doing a bad job here. All right, anyway, this is going well. Well, hopefully we actually don't have that occur. No, no, hopefully I get no beer from anybody because it's going to be lovely. You actually down there Dan seem like you're not as much in the light as the other two guys. I am actually loving it. You can see people. DevOps stand up comedy. Yeah, that's dough and then now, sorry about that. Anyway, so, yes, that having been said, why don't we start with you, Dan, on a quick introduce yourself and your project. Why don't you? Tell me what you do. Sure, so my name is Dan Bodie and I'm actually a long time employee of Puppet Labs. I actually started when it was just a few guys in a closet. Wait, you were all in the same closet? Wow, we actually had two closets. We had two closets that we worked out of. His and hers. Exactly, and then it slowly expanded out into the hall and then we had to get a proper office and it's been quite a ride, but we do, we write automation software called Puppet, so we're the company behind the open source project, Puppet, and yes, and I do stuff. And I do integration work for the business development department. So I write a lot of code, but I really report to marketing, so it's kind of a strange job. That is very weird. Yeah, excellent. What, just because for out of curiosity, what city are you guys based out of? We are based out of the fantastic city of Portland, Oregon. Excellent, excellent. All right, same question right down the row. So my name's Matt Ray. I'm a senior technical evangelist with OpsCode. We're the company behind Chef, and as long as we're talking about fantastic cities, I'm from Austin, Texas. Love it. I'm not getting trolled for Seattle. Ah, see, I was going to have Chef versus Puppet, we should also have Seattle versus Portland. OpsCode is based out of Seattle. I'm actually from Austin, Texas as well. See? That's really kind of... We reached the same university, didn't we? UT Austin? Yeah. Awesome. Great. You guys did really, really well in football recently out here. Next topic. Wow, I did not expect to get trolled on that. So, yeah. Boomer something, I think, is the appropriate thing to say? I don't remember. So yeah, we're the company behind Chef, and most of my time is spent working on a project called Chef for OpenStack, and I just had a session on that about two hours ago, if you may state it was great. And I'll post my slides later. It's just a community project around deploying Chef. I have to deploy an OpenStack with Chef. Cool, cool. Jesse? So my name's Jesse Andrews, I'm with Nebula, and the reason I'm on the stage, I guess, is that I started a project called DevStack, which basically it doesn't use either of these things, and although you can use both Puppet and Chef to deploy DevStack, the problem that tries to solve is there are hundreds, I think 180 or something contributed in the last six months or something that, of people doing development on OpenStack, and what we want to do is on every single commit, run as much of a test as we can get before we, does everything still work together, all the different projects, so not just unit tests, integration tests, does it actually work together before we actually do the merge? And so we had originally started working on, you know, doing the packaging, and then after you get it packaged using, we actually had both Puppet and Chef recipes, but what we ran into was the problem that when any of those steps broke because developers made changes in how things need to be installed, they would run away and not actually help us because they aren't experts at Debian packaging or RPM packaging or Chef or Puppet, and so not every developer knew every single tool, and so we're like, what does everybody hate? So half the people hate Chef, half the people hate Puppet, I don't know if they hate, but anyway, everybody hates Bash, so we use Bash. So it's a Bash script that's a thousand or so lines of code now, and you could just set some options and it will download everything and install it. The goal though is that it helps, like Matt on his job, by a developer saying that we changed something and you need to be aware of it, so hopefully package maintainers and people building products and recipes on top of it can look at what we're doing and then actually update their recipes or packages based on it. I mean, if probably the people here have already been to devstack.org, it's great, it has comments for all the, what it's doing, it's a well-documented Bash script. I've actually been looking at using the same software that's used and I forget what it's called for comments for inline documentation for puppet modules. Yeah, there's, we use something, the ideas Shoko or Doko or Roko, it's different, there's one for each language, but yeah, it's pretty nice, literate programming for Bash. So one thing that I forgot to mention is I lead up the development efforts and the community efforts around puppet labs and the puppet modules for OpenStack, probably important or relevant. Okay, relevant. Great, really, you can use puppet to install OpenStack? Yes, you can absolutely use puppet to install OpenStack. Wow, that's fantastic. How's that working? It works, it's reliable, repeatable, and yeah, a lot of people are definitely using it and contributing to it. There's no question it could be better. Wait, no question it could be what? Better. And how? So I would like for it to live a little bit closer to the development process. I think, and that's a lot of what I'm doing here and asking people about is kind of where does it make sense for it to live in the actual development ecosystem? Which I think is one of the big questions. Do you want it closer to DevStack? Or do you prefer, sorry. No, no, go for it. Or do you prefer to work with releases? So that's a really interesting question. I had assumed for the longest time that it was something that consumed packages, but I think there's a lot of value in having configuration tools that can not only do deployments based on packages, but also be used for development. But I think it's gonna be a progression to get from where we are now, which is kind of something chasing releases to getting from there to chasing trunk, and then from getting there to gating, or sorry, post gating, and then the big question will be once we're at the post gating step, how much sense does it make for it to also be gating? Yeah, which is an interesting question when, as Jesse said, if you've got Chef and Puppet and DevStack and Juju and CF Engine, anybody hear from CF Engine? No? Okay, weird. But yeah, how do your developers deal with that? Well, I think that that's where identifying the minimal core that happens as you're going in, and I think you actually helped set up DevStack with the CI gating project. But as OpenStack continues to grow, there's so many options for what backend you use, what storage, even how do you want your storage and backend to be related? And I think Nova has 600 flags. So having it such that we could chain the gating, so like once something lands in trunk, then it could go ahead and deploy with your recipes that currently exist and build packages as they currently exist. So at least you know if there's regressions occurring and then you can go in and fix them. So within Chef for OpenStack, there are many parallel forks and several of them have CI already applied to them and my goal is eventually to take kind of the main trunk and offer it up to fit it into the other CI branches because there are forks that are working with source already instead of just packages because some providers need a certain feature that is not stable yet or not released yet. But let's get in there. So it's actually, you've both touched on something that I've actually been having some conversations with people about and it's interesting that you brought up the installing from source thing because it seems like I know when we started off OpenStack originally, we spent a lot of effort to make sure that we had sort of packaging thoughts going on and it seems like over time, oddly enough, as the project gets more mature, we are doing less and less with that and we see more and more people going from a source and I'm sort of curious where you guys are seeing that go. I see that it's actually a good thing because what was happening originally is we had developers going in and doing a crappy job of packaging and so what we're having now is the actual distros and people who know about building RPMs and devs and Chef recipes and all this go in and first of all, package it correctly and then secondly, sending back requirements and here's things you could do to make it easier to actually implement so. So one of the questions that I kind of have for the OpenStack community is OpenStack isn't really in the business of packaging right now and everything's running continuous integration from source. Does it make sense to actually have package deployment be part of the gating process and when I think about should we be using packages versus source for the puppet modules, that's kind of my reverse of the question is should packages be part of the continuous integration process? Interesting question, do you want to jump on that? It's a hard thing, there's history, definitely. At one time, a lot of different people working on lots of different packages and nobody knew what to do. It's certainly, I think, hard from the user community, the fact that we don't really have official packages that we recommend, we have tar balls basically. Then again, the recommended way that is to go get whatever your favorite distros is, version or crowbar or what have you. So would it be better for us to have official packages or should we be having it such that we have a really easy way for all these different people who are building both open and closed source editions to chain their Jenkinses together and then run the, they run the packaging and then they run that testing afterwards and tell us when it's breaking because some of those configuration options require hardware that we don't necessarily even, would you like to really set that up? No, in fact, I'm not going to. It's just not going to, somebody asked me earlier if they could mail me some hardware and I was like, sure, I'll put it in my living room? I don't know, we don't have a data center. So yeah, that's not gonna, but you're exactly right. Getting that, we've had that, and this is sort of the reason I asked a question about the source thing is we've been having a lot, when we stopped doing packaging in the project, it was a big question mark, but part of that did come from the distro saying, hey, we'd like to, we can do a better job of this than you're doing. Have the distro's been engaged to be part of the gating, F for CI?  This isn't supposed to be the panel discussion on me now. Dude. But it always becomes the discussion on you. If right now we're splitting our tar balls. Anybody, anybody, great. No, no, actually. We've asked for that several times, for that to be more, to be, to do more of what Jesse's talking about.  And I hear that and that's fine. But it, the distros have been fantastic at doing a lot of actual work inside of OpenSec and have been thrilled to have every single person from SUSE and Red Hat and Canonical that have been in there doing Debian, even not a company behind that. But like the folks that are doing work have been doing great work. But I think that they're, the problem is they've all got processes that are centered around their release of their thing. And so we can, whereas we can talk, it'd be the same thing as Dell or HP or Rackspace. They've got their clouds they're deploying and they've got developers working on the core software. But the releases of those things aren't necessarily in lockstep. And so I think we've got that same sort of impedance mismatch. But a lot of the developers probably, at least in the companies or projects that do have a lot of development resources, as you said, probably actually, if there was, would benefit from that sort of integration. Definitely. I'm wondering, like again, this may be a sidetrack from what, this entire thing may have been a sidetrack so far, but I think that there's, both from a performance testing, the Tempest project and so on and so forth, getting that sort of gating up there and maybe just having a few sample test stacks or something that are really like, here's what you do, get the script or chef recipe or puppet recipe and it deploys Jenkins that chains to us and then just uses the official Ubuntu and the official Red Hat, whoever, just do two or three of them and then say, hey community, take over. Well, I guess sort of my incendiary, the way I was trying to make this be a little bit more incendiary is if you're already working on a puppet module that uses things from source, do injecting packages back into the mix, does that help? Or are you doing just fine pulling it directly from the Git repo? So I could, I mean, I have a pretty clear vision of what puppet modules, of how to make the puppet modules configurable for either doing source versus package installation. I mean, you can, you know, the way that actually compile and parsing is staged in puppet, you could collect, you know, you could tag all the services and packages as being open stack and then collect and disable those from another scope. So you could use that to just kind of toggle a global switch to go from package and service to some special source versus service. But I think from my perspective, the question is, is that worth spending time on? Is that, you know, how do I prioritize that? Is that one of the more important things that can be worked on? Yeah, which I think is the, what sort of question I've been thinking about recently too is like, should we, how much is this actually, where's the, is it the thing we, is it helpful? Might be, I'm just saying, is it worth thinking about? I think it would be valuable from the, like, I don't know of developers who willingly want to break, you know, other people's packaging and things like that and tests. Totally. So adding longer and longerness to the gate or adding complexity to gating to have those tests kicked off, like if we all of a sudden have the matrix of like 80 different configurations that we actually test, such as my SQL head, do we, we may not want to gate on that, but we, I think we want to know when regressions are occurring. So. That's something that I'm thinking about and I've spoken with Dan Prince about maybe looking at using Smoke Stack and having that be master that goes along with Grizzly and do post-gating, like Smoke Stack is doing post-gating. But I think not really being an active developer, yeah, what do you guys think about Smoke Stack? Are you, are you guys pretty content with the post-gating model? I guess maybe they're drifting off. And they all look at me. Well, I guess both of you guys are involved in the development process. I mean, it's hard to say everybody agrees one way or another, but I think people are pretty happy with the gating that we have right now unless their feature is in the, that gating test. Yeah, which, and then we, there have been just to follow up on that, there have been some decent discussions this week and I think this is pulling back around to what you're talking about with, with checking out the puppet or the, or the chef things is, is ways that we can, we can hook in other people's testing work, I think is how we start to expand that because everybody has their pet feature. And so how about you test, you hook up a thing and then, then you don't have to convince me, because that's, I mean, it involves beer again, but that's, there's only so much beer I can consume in a given week. So we possibly have beaten that one to death. Yep. So, only three or four people left. What's that? I know. Only three or four people left. I'm surprised that more of you are still here. So to go back into the, that's a very good point. Yeah, there's internet in here. So why, cause I'm pretty sure I know where they want this panel to go. Why, why should I use a chef rather than a puppet? Chef versus puppet. I've, I've never been asked that. So, I was going to bring up an anecdote of an actual, there was a session with Mark Burgess from CF engine and Adam Jacob from Chef, where they traded each other's slide decks and gave each other slide decks.  Yeah. Well, actually, yeah. What if you said, tell me why I should use puppet? So, you know, puppet is, I'm in trouble. Yeah. Puppet is, it's embedded in the new Cisco open stack stuff. Yeah, it is. That's cool. Yeah. What's that? And red hat stuff. And red hat stuff. Yeah, I do. Yeah. Got the puppet guy giving me leads. You know, you know, Dan has been involved in open stacks since I saw you at Austin, I guess, or Bayer. Yeah. I guess I started sneaking around Austin. Yeah. So I made a year ago. We've both been at, you know, the last six summits. You know, and, We've had six of these things. Wow. And so, you know, both projects have been involved for a long time. And so, you know, you are going to get access to people who, you know, know the rough spots have been, have seen a lot of deployments already. And, you know, hopefully have got stuff that works and is reproducible and other people are finding useful. So, you know, both tools have their place. Well, so that leads to a bit there. So you get access to the people that have worked on these things before and you get access to their knowledge. To speak for our absent JuJu, I know that one of the things that they put out as a feature is sort of the reusable nuggets of, you know, app to get install WordPress server, right? The equivalent of that. What's the, but that reusability seems to be something I've heard from both camps, both you guys' camps from day one. Like what's the, is there a, you guys working on that same thing? You got reusable, like how is, how is me getting your knowledge through your system working out? So I think just in terms of, you know, general knowledge, and of course one of the reasons that Puppet and Puppet Labs is so interested in OpenStack is that the easier that we can make it to get to those self-service APIs, the more interesting kinds of automation things we can do on top of those APIs. And just talking about reusable content going beyond OpenStack, forge.puppetlabs.com is a great place to go. I think we have around 600 contributions, a lot from the community. And we have a lot of content, but one of the challenges now that we have with the forge is trying to filter that and make it easier for people to not just find lots of content, but find the content that they need to deploy their applications. Right. And so, you know, going back to the idea of reusable content, and you know, embedded kind of wisdom of that content, you know, one of the things that we have going with the Chef Cookbooks is, you know, these are used by real deployments at scale. You know, this is, it's parallel to the work being done by Rackspace. It's used at Dreamhost. It's used at AT&T, you know, HP, a lot of the work, you know, that I did with HP early on, you know, has filtered back in. And I said, Dell, right?  We had six or seven companies where people were like, I'll do that. I'll do that, you know. And so we are watching each other's forks and branches. And then, you know, I'm just kind of a ringleader in the middle making sure that if I come in to this community, I'm actually getting something of value. You know, if I go to, you know, let's say I go to Dreamhost, for example, they're not really incented to make your OpenStack deployment work. You know, I mean, they're busy running a public cloud. You know, and so what, you know, what you get by having, you know, lots of people working on the same code and someone trying to make it useful in the middle is, you know, someone that is actually incented to help you succeed. So that's what you get with, you know, the Chef or OpenStack side of things. Cool. And I think it's similar. I've recently called myself kind of the hub and the wheel with all the sporks of sporks. Sporks. You should see my bicycle. I am from Portland. Oh, that's a good point. Yeah. The old hub and spork.  No, actually I have a bicycle that's just a hub with all the sporks. Of course you do. It's amazing. It's actually a unicycle. Yeah. I'm sad there wasn't more of a Seattle, Portland thing because now it's going to be you and me and that's not nearly as fun. But I was saying a lot of what I do is really, you know, all these companies that are deploying OpenStack have the best practices and the knowledge and it's all about working with them and being the person that leads, making sure that those changes go upstream so that everyone can take advantage of them. So I want to do a follow up on that. So one of the challenges in DevStack is the fact that everybody has different requirements and people want to do Chef back ends or Fedora or Ubuntu or how well does your various recipes and cookbooks do at providing flexibility for like all sorts of scenarios or at what point do you have to say like here's my cookbooks for this scenario and here's my cookbooks for this scenario or recipes? Thank you first. I think a couple of things there. One is that most of the differences between platforms are really data differences. It's almost an internationalization problem of everything's the same, only the names have changed. Well, how about things like Zen server versus KVM and things like that? So right now, libvert is the thing that's mainly supported but a great thing about the community is there's someone from Microsoft working on Hyper-V support. Zen support existed originally because the puppet stuff was done for Rackspace Cloud but they've definitely forked off and I'm not sure how supported the Zen stuff is but I think in a lot of ways that's due to demand. The people that are using the stuff will add the extensions that they need. And so it wouldn't be that complex if the people put in the time to upstream those and you'd have one recipe that would support Zen and Hyper-V and KVM or is that? Everything's been designed to have directory layouts where it's obvious if you wanna add an extension where that extension goes. And it's also been designed so there's really kind of three layers. It's kind of three layers of code where at the very back end you have very specific classes and very specific configuration interfaces so you might wanna do something like this is a glance file back end or a glance Swift back end. And it's maybe there's a hundred of those that are composed into kind of the roles that you may wanna deploy, things like a glance, which is a registering API or a Keystone or a database which is a database with the six databases, or sorry, a MySQL server with the six databases that have to be added. And then at the very top level there's something called controller and something called compute. And the interesting thing is I designed it like that thinking that people with different amounts of complexity, people for getting started would say I wanna controller, I wanna compute, but that really most big companies would wind up customizing using all the little bits. And what I found is that's actually not true that people are actually fairly happy to say if you can either give me a controller and a compute or if you can just give me the seven or whatever roles I want, that's actually what people prefer is that as the interface and then hide all the details of the implementation behind it. And so, Chef? Yeah, so the way Chef OpenStack is organized is around roles. And so currently the N plus one model of end computes and one controllers is definitely there. All the services could be decomposed to run on separate machines. So if my SQL is clustered over here, Rabbit's clustered over there and then computes are each on different boxes. So that's the approach we've taken as the roles are all separable. So as people need to replace an individual component, maybe Rabbit isn't what they want, they want something else. We'll put in a different, we'll use that role to search against just plug in whatever the message area is. So then as far as like, again, I think Zen server versus KVM is the largest difference just because in Zen server, the recommended way to deploy is using a DOMU to run most of the things. Whereas KVM, we're running directly on the host. I know that the crowbar uses it in a certain way, so that's KVM if I'm not mistaken, but is there a good Zen server? I don't support it. There is, over in the SmokeStack cookbooks, they've got a Zen server. No one has really asked for Zen specifically. What kind of happens with the cookbooks is if somebody wants a feature and we talk about, hey, how are we going to get that supported? So right now we have KVM and LXC and talking about Hyper-V, they all follow the same model words. You fork off an attribute and it says, I'm using compute. Compute needs to include a recipe. Oh, what is my hypervisor? It's KVM, include these. If it's Hyper-V, include this. So if Dan or whoever is working on SmokeStack put the time into upstream that change and others or others were interested, that's something that would fit in again. Absolutely, absolutely. The model we've espoused is if there's a feature you want to support where we will make, if it's not already pluggable, we'll make it pluggable. And how much is that magic where you have to be experts at your various tools to be able to have that degree of flexibility? Or is that pretty low? It's fairly standard, like if you're a chef user already, it's not hard to see this pattern where it's like you set an attribute here in the role and magically I'm using all these things. Well, I don't know how I'm building the recipes themselves. Yeah, so the recipes... Are the recipes magical or are they pretty... That's actually not that much magic. You know, because a chef does a pretty good job of making reusable components. And so, you know, I think Chef for OpenStack uses 20-some community cookbooks that, you know, we didn't have to go and like rewrite whatever features, just grabbed it out of community and started using it. And so we're, you know, the OpenStack cookbooks are getting committed back to the community and we're trying to just keep, you know, I try to get whatever comes in, we commit it back to the community, document it. And so when somebody says, well, I want to run LXC, but I want to Chef back end and I want to use Postgres, we're like, set this attribute, this attribute, and this attribute and you're good to go. So you've both mentioned people that are using the modules and cookbooks and stuff like that. And then question marks about getting some of their work upstreamed or their barriers to getting that stuff in or questions like how are you seeing that going? I can say that it seems like time is really the number one barriers, everyone's really racing for to have something every six months. And I think part of it is time, but I think also part of it is culture for companies to understand open source and definitely a lot of the companies that are joining OpenStack are not traditionally open source companies. It's very true. And I think it'll just take some time for people to really understand the benefits so they can justify to spend the time to get things upstreamed. So if the release was, let's say every two months, would that be better or worse? Worse. What if it was every day? Are you providing packages? Do you want me to provide packages? So you're gonna go double A in just a few days? I was sort of mildly leading towards, if like off of that every two months is, a lot of the companies that are doing this are looking at more of a continual deployment model rather than a, I'm gonna go grab the latest release, not that the Shuttleworth demo wasn't fantastic because it was actually really cool, but like doing an app to get upgrade from one release to the next release, I'm guessing is not how most public clouds are running themselves at least. Or maybe you guys are seeing more than am I. I only work for one company. But I'm guessing that's not how it's working. What I have seen and heard is usually that no, people are not app-get upgrading there, decommissioning racks and switching them, slowly migrating things over. So that being said, there's sort of a two prong thing here. One is the question of what if we weren't releasing every six months and or what if we were releasing more continually, would that help or hurt? And the other one is how's upgradability rolling upgrades and stuff like that working out for you? So I think one thing that's important from my perspective is if OpenStack was gonna release every day, then for sure you're gonna have, I mean, the APIs have version one, version two. And I think you need to make the same kind of commitment for the command line interfaces and also for the configuration interfaces. You know, that they need to support some kind of semantic versioning. So you can have expectations about when things may break. That would be nice. Yeah. That's partially what, you know, maybe we could get into with, I know Vish did some work for XML interfaces to from the public side, now that anytime a commit is made, we actually make sure that the XML is still generated the same way because none of the tools out of the box, like all the CLI tools, all the libraries actually just use JSON. So it was just lucky if XML was working or not. Whereas now we're actually testing that. So whether we get there soon or late, I think we're adding more tools in place because of the fact that there are a lot, public clouds actually doing continuous deployment with OpenStack. And there are individual companies, I know that are doing continuous deployment on their own products, even though there's then a barrier between when they are done internally versus shipping. But can you guys, can you think of certain priorities that you would add there? If you had three things to ask for, as far as helping with the upgradeability, is the CLIs interfaces on the top? JSON, I mean print JSON. And I heard that's a feature that's coming in Grizzly, that there'll be an option for everything flags. So as opposed to tables, you can just dump JSON. Okay. I mean, stability and configuration would be amazing. The CLI, we don't use a lot in actual configuration and deployment, but it's just, you know, things are mostly in the same places, but the config files seem to be moving around. Maybe this is then a chicken and egg in the sense that by having sort of the post commits where we're deploying with the recipes, maybe, and then running Tempest against them, which I think is very similar to what, you know, the public clouds are doing. Maybe if we have that resources for the community, we could capture those. But I don't know if, like, you guys are not having your own CI infrastructure for these projects, correct? Or- I have some, I mean, right now, I'm running unit tests, but not integration tests. That's something I'm hoping I'm getting soon. But just like Matt said with his project, there are people who have built products on top of this stuff who are running continuous integration on their distributions up there.  Excellent. So just as a general note, because we can do this for a while, honestly, it turns out, we all like to talk. But there is a microphone in the middle of the room, so if at any point anybody takes offense to anything that we've said, or wants to ask a question or whatever, feel free to sort of jump in. I just want to jump up and shut the thing. Oh, wow, that, that was quick. We gotta jump up. You know, there are a couple more sessions starting right now, so I thought it would be a good idea to ask it. Guys, do you have recipes for deploying highly available OpenStack? In particular, let me rephrase it. Can a regular Ops guy take your chef or puppet recipes and deploy full production environment with DRBD or Peacemaker or something on top of it? Not yet.  You know, now that there is a semi-official HA support for Folsom, we'll be implementing it. And I would say that we have a repository which shows an example of how you can wrap an HA layer on top of what I've created and actually all the components have an enabled flag which can work for active passive even though I kind of had implemented the capability to allow active passive modes for everything as iteration one, because really what I want is active active. And I'm actually working with members of the community who have a fork that supports active active for all the components that I'm currently in the process of merging. What do you find the most complex to make highly available from the queuing or database or like what is the challenge from having the recipes just work in HA mode out of the gate or is it different environments require different things? I know that for the HA mode that I'm working with a partner on that it was that they actually needed to change some of the configuration options to allow multiple rabbit hosts, for example. But I know that that stuff's actually going into Grizzly. So that'll be one of the challenges. When you say active passively doing just like HA pairs of the services? That's what I had done. I'd implemented a prototype basically using Pacemaker and yeah, exactly. But going forward, the thing that I'm really looking at is using Galera and it's really a partner has been working on a lot more than I have. So I know that I am looking at the code as it comes in and merging it. Yeah, so the Florian, us? Florian is Florian here. David, it's your talk and it's Pacemaker for the various Nova services, Galera for my sequel, Rabbit MQ's clustering. So trying to move away from pairing and I think tomorrow CloudStack has an HA talk to. Yes, so. Right, right. I'm just giving a shout out for more HA. Yeah, HA's good. Oh, I've done that again. Oh, it got darker. My monitor was dimming again. So we've touched on it a couple, I think Jesse and I've actually both mildly asked the question but I don't know, went somewhere else about upgrading and that sort of path. Upgrading. Yeah. Let me preface this, but us and OpenStack developers haven't, up until maybe a few months ago, hadn't really put much effort into really providing, like what we are doing in DevStack was trying to do proto packaging and proto recipes. We hadn't been doing the work to show how to move between upgrades. In fact, Jesse, do you have a project similar to DevStack that has something to do with upgrades? We're looking forward to how we can do that and actually Vish and Dean who work on DevStack worked on a project called Grenade that the idea was that it would deploy the old one and then try to run a set of scripts and then deploy the new one and then make sure the instances and volumes and all that are still there. Now, we caught some bugs and it helped with catching bugs of how upgrades need to occur and making sure that the migrations actually work but I don't think it was in time to help you guys with actually knowing how you move from one release to another. So I think that you guys are currently are facing two problems of which is you have to chase developers and try to figure out what actually changed and then actually get the recipes to work. Yeah, so on the Chef for OpenStack mailing list we had a thread about what are we gonna do about upgrades and it kind of got turfed because it seems like all along no one has given enough thought to upgrades and said, so the next it became, we'll do it with Falsal. Yeah. And so it is kind of an ongoing problem because what really happens is you stand up Diablo or Cactus, I mean in the Mercado Liberations talk, they stood up Cactus and then over here they stood up Essex because there was no upgrade path. And so until we are followers, we are going to be consuming what is available. So if there is a grizzly upgrade guide that says this changes to this and this changes to this, we can recreate it. I mean we can programmatically do technology. Yeah, we're good at scripting. Yeah, if there's a document that says how to do it with some assurance that it will work and not blow everything up, then for sure we can automate it. What if that document is a shell script? It's fine. I read Bash. Great. Yeah, I can read Bash. It'd be interesting to see if Puppet should model a shell script versus if Puppet should get, it depends on what the shell script looks like. Well, the shell script is again just meant to be a proto, like the DevStack and Skrinade, the goal is that you guys are actually the audience and deployers, it's not actually meant to be used as deployment. Jesse, are you saying that people shouldn't use DevStack for production deployments? Burn. No. If you know what you're doing. Yeah, sure. So it should be on purpose. It destroys everything every time you run it. Would you recommend that people who write tools for production employment should read DevStack and check diffs on DevStack to understand what's going on? The hope is yes. And so if it sucks, I would like to know. We've got a question here. So does Grinade only test the default DevStack changes? Or does it test all the config options that DevStack doesn't use? It doesn't check. So this was just the first stab at it. And so the question is, first of all, is it the right even tool? Because we are, of course, just taking DevStack and then installing the previous version, then running a set of configuration changes, and then running the new version. But it did enough where we could check Nova volumes to sender. And the idea is maybe we could help with the Nova network to quantum. So the big changes. Yeah. But again, this was just a month or two of work right at the end because we realized, while during Folsom, a lot of people said that, hey, let's work on upgrades. A lot of work did happen behind the scenes with a versioned RPC and other things in the code to help that so that you could actually do live upgrades. There wasn't necessarily that much communication between the people doing the development and the people who actually are asked, how do I do this? And I guess I can say too that that's definitely one of the advantages of Puppet and these configuration management tools is that they can really check to see what the current state of the system is and manage state transitions. But the bigger question is, is there actually a set of state transitions that can do a live upgrade? And I think for certain use cases, you could get really close to a live upgrade, maybe with control plane downtime. But not necessarily data plane. So your VMs will continue to operate. And everything, but we recommend currently turning off your control plane while you actually do these operations. Moving forward, we're actually adding versioned API, versioned RPCs, which are how all the components actually communicate, which will allow you to then upgrade each individual things as a rolling basis, like upgrade some of the workers, make sure it still works, and then continue on. We have added challenge though of the fact that we have the HPs and rack spaces who are doing continuous deployment who can do those baby steps to do larger changes. And they could support the version today and the version tomorrow and not have to do that gradual change out if you're doing something large. But when we talk about six month increments of work, it becomes potentially much harder to do that sort of rolling upgrade in a single deployment. And so that's kind of why I was getting to, at what stage should we look at more rapid releases, as well as, for instance, Nova is becoming calmer in the sense that we're removing features and it's becoming just about VMs. If it becomes mostly about drivers, like improving the Hyper-V or a Zen server, having more rapid releases, but it's mostly drivers and interactions with other tools, is it better to have more rapid releases? The rapid releases doesn't become a problem if APIs have all stabilized. I know the conversation comes up every summit. Is OpenStack the APIs or is OpenStack the projects? Both. It's the people. OpenStack is the people. It's the love. If those APIs aren't stable enough to switch out pieces or have them versioned, then we can't handle it. Well, I think that in order to even do upgrades, you had to have them versioned and or at least the ability to have the old thing and the new thing running at the same time. And we have that problem though of the six month release versus the every day or every hour or how often the continuous deployment guys do it. So it's an added challenge that maybe the recipes that help with upgrades for the real releases don't actually help with the people who are doing continuous deployment because they need to chase different goals, maybe. That's where I was trying to get to, how much magic is there in these things that you had to be an expert to be able to talk to both.  Would it be relatively easy to have packages in your recipes be able to support moving them with all the coordination between all the services? I can say, yeah, that it's possible you might have to write your own coordination layer, but we're working on using the resource model to manage coordination between things. But in terms of a pretty typical workflow for upgrades with Puppet would be using environments which allow you to have multiple versions of code and then targeting in NOOP, which means don't run but tell me what would happen if I run against the new environment to determine change impact to make a decision if you wanna do a live run. And then run live while using for orchestration you could use ImCollective which is another Puppet Labs tool which is essentially a message bus that sits between the person controlling their environment and all the Puppet agents. So you can use that for staggered runs to say maybe run one, check it manually and then say, well, let's run the next 30%. Let's run the next 30%. Yeah, so that's the model that you're looking at in terms of how to deal with the fact that you don't really wanna just update the module, change the version and have all thousand machines splat out and upgrade at the same time. No, you would definitely wanna maintain just like you were talking about maintaining multiple simultaneous API versions. It's very similar to that concept. Same thing. Yeah.  You can use ImCollective, but I mean, we have environments as well. We have the idea of rolling upgrades, switching between environments, that's how people do stuff for real. Yeah, because these things are getting pretty big. How's the scaling picture looking for you? And this is honestly, I'm just curious. Like as people are doing thousands and thousands and thousands and thousands of nodes, is that within the capabilities of what you guys are doing and stuff? I can't speak for the size of all the users. You could ask three most. They're very public about their deployment. You could ask HP who's not very public. I don't know if I can ask anything. That's gonna be very difficult for me to do. Right. If I know any of the works there. Definitely substantial production environments that are being deployed with Chef. But I'll let those guys start. I've got four boxes. Yeah. Wow, that's so many more boxes than I have. And I can say the same for Puppet. There's so many different ways to run Puppet to achieve different kinds of performance. Of course Puppet apply just running, synchronizing just code and then running everything locally is gonna scale forever. The Puppet master itself, if you're running in client server mode is by default, stateless, except for CSRs, which whatever. But it's basically stateless. They can be scaled horizontally. But there are definitely a lot of feature add-ons. If you're taking advantage of all the features like maintaining a configuration management database and really using the entire feature set of the Puppet server, then it definitely as you add state, it does have limits on scale. Yeah. Fair. Any other questions up there? That's we're actually at the end of the list of questions that I've got. Any questions? Any more questions from folks in the room? Or any other things that you guys would like to delve into? We've got a thing. There's a thing. Do you guys maintain different branches for the different OpenStack versions? Yes. We currently have an Essex branch and a master branch. And the master is when Grizzly milestones start to come around, we'll turn master into Folsom. And then have master will be Grizzly, I guess. And then there's lots of community forks. So somebody like Dreamhost has, they've got their Folsom branch, but in their branch they have things like sources. They're pulling source for quantum, I guess, because of stuff they wanted. And so there's lots of forks and branches. Yeah, I'm also using, I have a branch for, I don't even think I have a branch for Diablo. I think I just gave up. I have one for Essex for Folsom. The state that things should be in next week is an Essex branch, a Folsom branch, and then master is gonna be for Grizzly. And I had spent some time briefly last week thinking about trying to simultaneously support all versions, and pretty quickly decided against it, just because. I asked the same question in the chef cookbooks session yesterday, like who wants to keep working on Essex? And it was kind of like, Grizzly. Yeah, I mean people are ready to move to Folsom. Well for me it was more just like, it becomes accumulation of technical debt to try to maintain a single branch for this stuff. It's tagged. Yeah. Cool. Any else from anybody out there? I think, yeah. I was gonna say, I think we've made it. There's a handbag there. Oh. There is a, there is one feature in Chef or Puppet that you do not in the other schools that you think is like a killer feature? I mean, both products are pushed by each other. And so they had a no op mode. And so we have a Y run mode. And then one of their devs is like, now we have to make another no op mode, have the same features that Y run has. So we have had search. You guys now have search. Yeah, we do have search. You know, it's. Good idea. Yeah. And so, we use Ruby, they have a Ruby DSL. We did that one first though. Well, you existed first. But I mean, Chef was written kind of in a response to things that, you know, like philosophical differences about how Puppet worked. And over the years, those differences have kind of simmered down a little bit. And so both projects, you know, are doing a lot of the same things. It really comes down to, you know, what matches your engineer's minds, you know. What are you thinking about, you know, on architecting a big system, or are you thinking about it as individual servers? I'm curious, out of those two, like which would you say is, is it related to those two? So on your website, does it talk about systems, systems managing individual systems? Because on ours, we're talking about continuous delivery. We're talking about, you know, your business moving everything from code to production.  I guess there's no question that that's a lot of what we're working on with our customers. In terms of branding stuff on the website for direction, I can't really speak very intelligently about it. Nobody can, it turns out. It's marketing. It's not just marketing. But I would say that, you know, I think Matt really hit on one of the major differences is do you want a Ruby API or a Ruby-based DSL, or do you want a declarative syntax, which is a little bit simpler, but simpler, which has pros and cons for different use cases. And I think the other thing in Puppet is, you know, the entire reason that you take Puppet code and kind of compile it is it all compiles down to a data structure. And I think that's really one of the differences of Puppet is that everything in Puppet is data and everything's exchanged as a well-known data format, which is actually a directed acyclic graph. And then Chef is built on the idea of your infrastructure as a code base. And so that's, you know, at the end of the day, you have a DSL, you have a Ruby, the programming language. Is it data or is it code? Data. Yeah. That was your cue. You had a pull-up? Okay, cool. Dying here. Cool. Anybody else? Cool. I think that's about it. Great. Thanks, guys.


You know, over the years, I've met a lot of people. I'd like to think that I surround myself with good people, with good souls, with good intentions, and with good senses of humor. So today, I thought that I would put that to the test. I asked all of my Twitch mods. I said, Hey, content. I asked my mods to send me TikToks to see if they can make me laugh. So let's see if they can make me laugh. And if they do, they get nothing. Let's just hop up for a sec. Just hop up for a second. I love these guys. So much.  It's awesome. So I just want to check something. So, what do you want to say? Hey, I think you want to... Shit.  I gotta get me one of those fuckers 16 inch mega fart. Yeah. I am actually going to buy that.  I was recommended. Don't give the mods any credit for that one. OK, one hip watchers chicken salad for you. What was that, sweetie? He said he wants the chicken burger with rooster sauce. OK, chicken burger with rooster sauce. You gotta game at it. Can't you hear him? He said he wants the chicken nuggets. Those guys are cute. It is kind of ironic that I have this face and I've always wanted to be a comedian since I was a kid. But, you know, like, what else am I going to do? Like, imagine if I was a doctor with this face, you know, just walk it in like, bad news. It's cancer. Good luck. What else we got here? Come on, give me something good, innit?  You got a knee? What is that? What are you doing up there? Oh, that was sick, actually. It's me, Spencer. I'm here to tell you about today's wonderful sponsor, Sunday's Dog Food. It's food for dogs. Like me, I'm a dog. In the past, I've been a picky eater. But that's only because my human, Ethan, gave me trash to eat. Like, literal trash. Like garbage from a trash can or dumpster. It was terrible. But I digress. Now, thanks to Sundays that get to eat real, human-grade dog food with natural ingredients. Not only is it healthy for me by containing zero synthetic nutrients, but it tastes great. I really prefer this over the garbage I was served before. Not only is Sundays good for me, but it's convenient for my human too. Sundays makes dry dog food that's easy to dish out and easy to store. And if you're forgetful like Ethan is, it's easy to order Sundays on an automatic schedule so you can make sure your pet always has food at the ready. I love Sundays. It's healthy, delicious, and makes me feel nice. The garbage I ate before could never do that. But like, actually, it was real garbage. Like plastic and sheet metal and misery bar. It was awful, but not like Sundays. Sundays is incredible. If you want to try out Sundays for yourself, why not head over to SundaysforDogs.com slash Ethan. That's SundaysforDogs.com, and use code Ethan at checkout for 35% off your first order. Well, that's all for me. Thanks for watching.  Ah, here we go. Here's one from Shira. Shira, I will give credit. This is one of the funniest mods we've got. I just wanna, I don't wanna offend any of the other rods. The journey is usually the part that you remember anyways. That was pretty good. Another one from Shira, oh god. This is so hard. I couldn't look away. I couldn't look away. You seem like the kind of guy who laughs at kids getting hurt.  Let me see it. Get caked. Oh my god, that kid's gonna get held back a couple of times. I don't know. I don't know. I don't know. I don't know.    I'm gonna prove this message.  I love this guy so much. I'm really sad. Pardon. Good night. Susan Bennett's pain, pain, too. I saw this on the other day. It's so funny. I'm parked, but I would do this trend. F, country, Florida, city, frankest. Animal, frog, body part, female, p-jar, friend, brand, Fridays, TGF Fridays. Famous person, Frank Sinitra. Sinitra. Clothing, very cooked. Chemical element. Furion. Sport. This one was hard. One more. Can't see what we get. G, country, Ghana, city, grapevine, Texas. Animal, giraffe. I actually have a phobia of those.  Job. All right, Jordan, let's see what you got. Rwinda. This is this, this is this, Jordan. This is this. Throwback to when I came out to my brother. I mean, you have two boyfriends. I do. That's disgusting. Bro, you can't be homophobic. What does that mean?  I've seen this. This is so well done.  This is so well done. Bring that back up to the boil. Like just like the rotoscoping and the tracking of it, like getting rid of the background, it's so smooth and then the color grading and like the matching of the shadows and the highlights and everything. It's really well done. It's so good. It's wild. Shit's wild. Okay, so I saw this.  No. I can't remember if I reacted to this in a video already. This video is wild. No. Is this all because, dude, this is getting dangerous, bro.    Dude, don't go under there, actually. Maybe. No, do that.  The video like getting all fucking. It's crazy. It's crazy stuff. I don't know if it's real or not. It's gross. My mom just sent me this picture for my first birthday. That looked like little seals. Did you find a body? Jesus. Oh, gross. This one's from Amy. It's not going to be funny. Why are you mad? What's up with this kid's haircut?  Don't do it. You're going to fall. You're going to... You're going to fall. No, this is good. Oh, I've seen this man.  You're okay. You're okay, man.  Ah, it's so good. It's up a button in your jacket. There's a way. It doesn't say such a place just for you. Jesus.  It's been a while since we saw each other. A lot of people are saying some things about me that aren't quite true.