John chapter 2 verse 1 till 5. On the third day there was a wedding in Cana of Galilee and the mother of Jesus was there. Now both Jesus and his disciples were invited to the wedding. When you get married invite Jesus.  His mother said to the servants as though she didn't hear what he said. Whatever he says to you do it. This is where the Nike took it from just do it. A wedding a wedding is when Jesus' first miracle happened. It wasn't in the church. It wasn't in the crusade. It was in the wedding. But the peculiar thing is the fact that in the wedding he did not heal someone. His first miracle was not delivering someone from the devil. Sadly his actually first first miracle wasn't even saving a soul. His first miracle was providing to a group of people something that they could live without. He didn't give them what they needed. He gave them something they could easily go without. A wine in those days it was a luxury. A water was a necessity. Now imagine on that time there were people children in the world dying because there was no good nourishment and no water. And here is Jesus begins his ministry on earth. Almost it seems like wasting his power creating something that is not a necessity. When there are people who don't have a necessity. I want to talk to you about faith today. And the first thing I want us to learn about faith is that it's as important to meet our needs as it is to fulfill our dreams. Many people will have faith to meet their need. But the Jesus we see in the Bible does not only meet your needs Jesus Christ also wants to get through your needs into the area of his mission his purpose and his dream for your life. Can somebody say amen any dreamers we have in the house or am I just speaking okay okay I just wasn't sure if anybody here who actually had a dream. Jesus is interested in not just meeting your needs. If he would only be interested in meeting your needs on this wedding this is what would have happened. He would have turned wine into water. The Jesus I grew up knowing as a child only turns wine into water. He takes all the good and gives you just enough to make it through. He's not interested in blessing you. He's interested in helping you get through life because it's all about heaven. The Jesus I was taught as a child he's not Jesus that makes luxury. That not just meets the necessities because wine was a luxury and not only that but he makes it in abundance. But the Jesus that I was told about maybe you heard about that Jesus he's only interested in giving you water. He's only interested in just letting you get through life. But the Jesus we see here does a first miracle that was not a necessity it was a luxury. That means our God is as interested in meeting our needs as he is in fulfilling our dreams. If your idea of God the closer I get to him the less dreams I will have I want to tell you something today. That is not our God. Because when people got close to God he gave them dreams and freaked them out. Moses said I can't do this too big. Gideon said God who did you think I am? Every person God called and God came to talk to. They said didn't say God my dreams are bigger. They were so blown out by God's dream they say God I can't take it.  Our God births dreams when we come close to him. Can somebody say amen? God is interested not just in fulfilling our needs and this is where our faith comes from. You have to have faith not only for your healing. You have to have faith not only just to get by. You have to have faith not just Lord I just want to get through this very difficult time. You have to begin to foster flourish faith inside of you. God you have something even more for me than this. God is on a different planet. God is not Baptist. He's not Pentecostal. He's not Russian. He's not Mexican and he's not American. God is God. Amen. He's not limited to the box of your upbringing your traditions or your paradigm. God is not limited to none of these things. He is all by himself. Like there's a story about that prince of one of the one of the Middle Eastern countries who came and picked up one of the golf players very professional golf players from United States and they took him to his hometown and he wanted to learn how to play golf better. And so he took this professional golf player and they played for three days and then he brought him back on his private jet to America. And upon arriving back in America he asked this golf player he says I just very thankful to you that you gave me three days of your life and that you spend time with me. Is there anything I can give you. And the guy said no I have everything I make good money. I am I am well off. And the prince said well it's disrespectful to say no to a prince. So please tell me what can I give you. The guy said I don't know.  The guy said OK thank you I will get your golf club a few weeks passed by and he didn't get anything from the prince. And then he got a letter from the prince. It was it was sealed. It was a specific special letter. So he was trying to think in his mind how can a golf club fit into a letter. Until he opened it and he saw a title due to a largest golf club in the United States. See when the king heard golf club he wasn't thinking about a stick. He was thinking about real estate.  God is bigger than that. See your God is not a president. Your God is not a senator. Your God is not just religious. Your God is the king of kings and Lord of lords. He has no luxury of thinking small. That's why when Jesus shows up he shows off his glory.  He is bigger than your needs. He wants to help you fulfill your desires. Can somebody say amen. You know when when King Saul was once approaching a Samuel. He came to Samuel and Saul had donkey problem. He couldn't find donkeys. His daddy sent him to find donkeys as he approaches Samuel. The Bible says that Samuel approaches him and says Saul I will tell you tomorrow what's on your heart. Oh as about the donkeys they are found and now your daddy is worried about you. And the next day the prophet tells Saul you're gonna be a king and of course Saul does exactly the same thing. He says not me. Who do you think I am. I'm too small. It's interesting that Saul donkeys were on his mind but destiny was inside of his heart. He didn't even want to admit that destiny was inside. Saul dreamed of being a king when there was no king in Israel.  That's all you're living and he says God wants to get through your donkeys but really God's goal and the mission of the prophetic is not just to fix the donkey problems. It's to get through the donkey problems into the destiny issue. You were destined for more than just to fix donkey issues. Can somebody say amen. You were destined more than just paying rent. Working eight hours a day. You were destined more than just getting married. Just having children. You were destined for more than just getting your papers. You were destined for more than just paying off your car. You were destined for more than listen. They're just simply getting through this thing.  You don't even want to admit to. A destiny is so large. A calling so big that you are saying no no no no. It's there but God this is so big. I can't even comprehend it. That's why it takes the prophetic to get through your donkeys into your destiny. Because somebody say amen. You know when I was younger the devil used a lot of donkeys to wrap around my life. Just for one thing. Because he wanted me to be busy. Chasing just the things in this world. Busy. Trying to be insecure. Busy. Trying to be this and that. As long as I do not take any time to look into the fact that God of heaven placed something bigger inside of my chest. And that is a vision and a dream. Not just to survive. Not just to live.  But to make a difference in this world for His kingdom. Because somebody say amen. Have faith. Come on let's put our hands together for the Lord. Have faith. Have faith please. It's not just about finishing your diploma. It's not just about getting through school. It's not just about getting married. And it's not just about having a million dollar in your retirement. It's about fulfilling the destiny God has placed you for. Amen. The second key of faith is trust in what God said in the face of what God has not done yet. When Mary approaches Jesus during this problem. She tells him that they have no wine. And Jesus does not give her a talk on drunkenness as a sin. She's asking Jesus to make something that is on the borderline sinful. She's asking Jesus to do something that if people take too much they are gonna slip into another side where the Bible calls it sin. She's asking Jesus to make a miracle on the wedding. Now later on we see down that this was the first miracle. Verse 11 beginning of His signs Jesus did in Cana of Galilee which tells us this was the first time Jesus did a miracle. There are agnostic Gospels that talk about the fact that Jesus when he was a child like he walked on water and he would go to school and just like you know make things appear and disappear and he make these like really cool things and while I love to entertain that idea it could make a great fiction movie it's not scriptural. The Bible tells us Jesus as a child as a teenager as a young adult did not do anything that made him stand out from any other person. Which tells me begs the question how could Mary have the audacity to ask Jesus to do a miracle at the wedding when she has not seen him do one miracle for 30 years. How can you ask him first of all how that thought even circulates within your mind. What makes you think he will do it now if he has never done it for 30 years? Has anybody here has some things God has not done?  I think every single person in this room this is where the challenging of faith is is when you are walking with God and there are things God doesn't do. What it does to our faith typically it destroys your faith. You ask once or maybe you pray for someone or maybe you believe for something and you don't see it happen or maybe you tithe and you see all these miracles you're like not happening in my life it's just not getting better. You see the testimony someone's saying that they got healed and and your situation maybe progresses. You see you know a testimony of someone else being you know depressed and they got free and you recognizing that for you this is just not happening and so what happens is that many times when we are surrounded by what God has not done what it tends to do is it tends to cripple and it tends to shake and break our faith to the foundations where we just give up and we just live with God and we sing you know what Lord Jesus does not do any miracles. People go as far as to create a doctrine that miracles ended with the death of apostles but Mary has faith. Jesus is going to do something. Where did she get that faith? Because a long time ago 30 years before the wedding Gabriel came into the room and Gabriel said this that child you are gonna have will be super natural and he is the Messiah and she experienced something in her body that wasn't natural and after that she believed in the word Gabriel spoke that even she did not see Jesus do a one miracle she went back to the word spoken by the archangel and she said to herself even if I will live 70 years and I will not see one miracle this boy is not normal Jesus this is Messiah and he will do wonders and he will change things in this world. Do not let your faith be destroyed by what God has not done. Let your faith be strengthened by what God said. God said that he heals people. We believe in that. Do you know why we believe in healing? Not because healings happen at the revival with Wiseman Harry or because healings happen in TB Joshua's church or because healings happen in Benny Hins crusade or because healings happen in Rayhart Bonkies or Dr. Young It shows it's not because healings happen in my family we don't believe in healing because healings happen at the race to deliver conference we believe in healing because in Isaiah it says by his stripes we were not will be one day hopefully maybe we were healed and as long as that word is true my faith stands on that word but what if my mom passed away what if my brother what if there's a pain in my back this word holds the universe and one upon which I walk this word never changes my situation can always change God's word will never change and I trust God's word trust God's word even if you don't see certain things don't happen maybe you're you prayed for something and it didn't get answered and your faith gets shaken how can I pray for something else now remember your faith is not based on the answer to your prayer it gets strengthened by that your faith always has to be rooted in the promise and in the word of God how could Smith Wigginsworth who raised over 18 people from the dead have kidney stones for most of his ministry time and had to run from the service to the bathroom to urinate with blood how come his own daughter was sitting deaf most of the service you can say how can you pray and raise people from the dead and at the same time go to the bathroom and have kidney stones that are killing you see a man like Smith Wigginsworth was a man like Mary he didn't see some of that activity in his body but he saw it in God's word and he stood it both feet and before he passed away both the kidney stones and the deafness of his daughter were completely healed but even if they wouldn't God's word does not change put your faith in God's word when you don't see God's act see God's word and that's what Mary did didn't see Jesus act do you think this is a first time that Mary asked Jesus to do a miracle I don't think so come on let's let's think realistically here you have a Messiah living in your house and one of the neighbor's kids broke their leg wouldn't you ask your Messiah could you do something you're running low on rent Jesus you have any tricks in your under your sleeve come on you're the Messiah you look oh you would never a David killed a Goliath as a teenager wouldn't you come to your teenager when the Roman Empire is invading your village Jesus do you have anything of that of course you would Mary asked Jesus problem number of times and it's interesting that every time Jesus no not gonna happen Mary you can't do that we can't talk about it that's a secret we can do it and she keeps coming and now it's the wedding it's the worst time of all this is when God doesn't do miracles weddings you know especially the problem with wine he definitely doesn't do those miracles and and Mary comes to Jesus has the audacity to ask him to do something he has never done for 30 years and Jesus did it don't trip over what God has not done step over it step over it that devil will put that in every person who steps into what God is doing first had the possibility of tripping over what God has not done don't trip even if you prayed for someone even if maybe it's in your family maybe it happened in your own body maybe you went through a divorce maybe you went through a death in the family maybe you went through a bankruptcy maybe you couldn't finish the school maybe you trusted in God something and you completely missed it and after that you're saying I won't hear God again I won't try anything again I tried doing a home group I tried to serve God I tried morning prayers and it's just I couldn't fall through and what you do is you trip and the devil laughs over it but little did you know you were one step away from your miracle don't trip over what God has not done step over it it hurts you won't make sense there will be doubts there will be confusion but listen God told to Samuel stop crying over the fact that Saul didn't turn out what you thought it would be fill your horn with oil go to a town of Bethlehem and I have prepared a king means when you've met with the disappointment God already has an appointment and a calendar your next miracle but don't trip over Saul step over it kindly with respect step over it and go to your Bethlehem and begin again that's why God says mercies are new every morning every morning God resets the clock and says let's begin again that's why Jesus told disciples when you go into a city and they say get away from us Jesus says don't accumulate rejection on your feet he says the moment you leave that city no matter how painful it is no matter what they threw at your face shake it off step into a new city as though you're coming for the first time with a smile on your face step with the joy with an expectation at this city will have revival forget about the things of the past why because I'm making things new somebody say man or restore your faith today if it got damaged because you haven't seen God do something step over it and you will see God do something because somebody say amen number three God's will never changes even when God's timing does God's will never changes but God's timing does you know what Jesus's response was to not wanting to do a miracle Jesus said this you know how many people say when when they ask you something and we say this religious thing sometimes we don't mean it I'll pray about it which means don't talk about it to me again it's over it almost seemed like Jesus's response was not my time yet it's like what is that supposed to mean when is your time but it's interesting Jesus says sir not my time and if that will be I or you you will a great I'll check with with with you what in 24 hours 2 hours 2 days the moment Mary heard not my time Mary turns around as though she did not hear anything he just said and called the servants he says are you guys ready for what for a miracle why because his time changes and in the next verse Jesus changed the time do you know how fast God's timing can change like this one moment God says no and next second he says yes one moment God says hold off and the next second God says here is the open door God's will never changes God's clock always does don't let God's clock blur your view of God's will don't let God's clock confuse you concerning God's character and God's faithfulness just because certain thing has not manifested yet certain thing has not made real certain thing didn't look right to you yet that does not mean God has changed the season the clock you are in is changing is changing but God's character is unchanging somebody say amen you know the story of Joseph is very peculiar because Joseph when he was a little boy he got this nice garment from his dad it was a call a coat of many colors and he this code symbolized that Joseph was the favorite Joseph is like getting a car on your you know 16 birthday brand new and the rest of your siblings are driving a bicycle that's kind of what it was it was an insult to the rest of the kids and and this kid he got a coat and the Bible says when he got the coat he started to get dreams and in his dreams he saw everybody bowing to him it's kind of hard to dream bad dreams when you got nice code when you have people speaking over your life and saying you're awesome you're beautiful you're handsome you're strong you're gonna make it people like that tend to have dreams not nightmares dreams not fears and Joseph gets this beautiful code and one day his brothers they ripped that code from him and they dip it in blood and then Joseph goes into Porifar's home and they give him another coat it was a code of a slave and we know Porifar's wife she didn't waste any time she ripped that code from Joseph as well Joseph goes into the prison where he is now a prisoner he is a code of a prisoner and the moment Pharaoh had a dream the Bible says when they called Joseph into the Pharaoh's court they ripped that close from him as well and then we see last time when Pharaoh puts a close of a prime minister on Joseph that's a lot of changing of clothes four changes of clothes it's very interesting because in every season when his clothes changed Bible use this word and the Lord was with Joseph in a dry pit and the Lord was with Joseph when Joseph is in jail and the Lord was with Joseph when Joseph is a servant he has a death certificate laying in the house and the Lord was with Joseph when Joseph was a prime minister and the Lord was with Joseph your seasons will change God will never change God will never leave you and God will never abandon you God will never forsake you can somebody say And that's why I want to challenge every person tonight do not become attached to your season your season may be today a season of being on the top of the mountain don't think God loves you more because you're happy your season may be today in the low valley don't think God left you because you can sense him or feel him or see him no more and the Lord was with Joseph God's character God's will it never changes it stays permanent that's what makes him God that's why he's trustworthy that's why he beckons faith in him because he is unmovable unshakable but his clock is always ticking that means that your situation today tomorrow it could be completely different that means that your problem today in the next five minutes that could be completely shifted that means that whatever you are in today in the next minute God can completely shift the situation around God's clock is on the move because God stands still can somebody say man have your faith anchored in God's character and always know because God is immovable my situation is gonna move joblessness is gonna move the fact that you're single ready to mingle and you can't find nobody that could move the fact that maybe your children are serving God and they completely have nothing to do with God listen that is able to move what because God does not move hallelujah God's clock can move amen lastly ridiculous obedience brings radical miracles Jesus did not call his disciples for a first miracle to assist him because they would question him to death probably Jesus did not ask other people who were important to do what he just asked only one category of people who blindly sometimes without asking questions and foolishly would obey some of the most ridiculous orders given by people he chose servants because servants don't ask questions they do what they're told you know what he asked them he said to put water into the vessels that's not complicated that's simple that's done before it would be done after it's what he said next that was so ridiculous and honestly could cost them their jobs is when he said take some of this water take it to the man in charge and tell him try the new wine now that would be you but like Jesus that's your business you go do that I can bring the water I can mix it I can do whatever I'll dance around it but you go tell him that that's your wine cuz you can't be doing this and you're hiding in a background and sending me to get embarrassed but see servants don't give Jesus opinions they said yes sir ridiculous obedience unleashes radical miracle painful obedience releases radical miracles when you're a single person and you're in high school and it seems like everyone is sleeping around everyone is using drugs and your obedience to God makes you not fit into certain circles your obedience to God makes some people look at you with that weird look your obedience to God simply means that that had boyfriend or that hot guy who everybody wants to be and he's winking at you there's nothing between you and him can happen because he doesn't walk with Christ and your obedience maybe cost you some emotional pain but remember it's that kind of obedience that turns your boring water life into God's exciting miraculous wine it's when you are dating and you have a wedding plans and Jesus asks you to abstain purity wise and disobedience doesn't make sense we're gonna be married five months from now with the same spouse what difference does it make it's all love and you have all of these reasons but this is where you have to take the obedience even if it makes you a little bit uncomfortable even if it complicates things and even if it brings little bit discomfort and pain and so Lord ridiculous obedience brings a radical miracles in my marriage it's when you're married and your spouse is not there for you your spouse doesn't share same passion with you it's when you're married and they don't support you as much and they're not there for the children or maybe they're completely ignoring you and you feel like you're alone over there and there's someone old fleeing on Facebook and this is where God wants you to honor your husband love your wife when you honestly feel like loading a rifle and doing something completely different and honoring them is painful loving that is painful it's when you were a child and your parents tell you you can't go to church and their life is falling apart and they're turning against you for lame reasons and you're saying how can I honor someone who hurts me like that who doesn't lift me up who beats me down emotionally and my obedience to God is so painful it makes me look like a victim it makes me look so weak but listen that's exactly what happens before a radical miracle radical miracles don't happen to anybody they happen to service who put on obedience and even if it doesn't make it look good there's somebody same as somebody same as it's when you come to church and you could have really really bought those new shoes with that money that you put on tithing and the shoes went on sale on Amazon and it's exactly is exactly same as your tithe and you said I deserve these shoes I haven't been doing anything bad and and you're putting it in the basket and you're looking someone else and they're walking with exactly same shoes and the devil is sending lies and said listen that's a sign from heaven don't give tithe don't give look it's so painful who in the world even gives nowadays you don't know what they spend money did you see the car of lad drives he probably steals that money and let the devil lies to you and says don't don't don't don't give don't give when your obedience is painful your reward is gonna be pleasant you know what encourages me when my obedience is painful what encourages me is the fact that if I don't obey God I will experience pain just a little bit later because you will play now pay later or you will pay now and play later but you will pay Jesus obeyed God and he was on the cross but can I remind you he wasn't the only one on the cross there was other two guys who disobeyed God and were there the only difference between Jesus and them is they went to the grave and Jesus had Sunday morning yeah it looked like Joseph obeyed God and up in prison and Samson flirted with anything that moved Samson also wasn't prison he got to prison as well the only difference is Samson came out of his prison to his funeral and Joseph came out his prison to his promotion every one of us will go through hard times even if you obey God you will go to hard times first can I say it again every one of us will go through hard times if you obey God you will go through hard times first and promotion later the Bible says the pleasure in God's hands is evermore but the pleasures of sin are passing I want to challenge every person today don't just obey God when it's convenient don't just obey God when it's easy it's the obedience the hurts it's the obedience where you have doubts with the prince the most results the Bible says God exalted Jesus's name above every name but it's not because he was his son it's not because Jesus did miracles it's not because Jesus had a virgin birth the Bible says because he obeyed till the point of death God doesn't give exaltation to people he likes he gives exaltation to people who obey amen God will see you through a miracle is on the way to your life but your faith will be shaken so that it will be unshakable and your miracle will be permanent in Jesus name


So, 601st Aviation Support Battalion is doing Victory Action Week this week, and it's modeled after our 1st Infantry Division Initiative of Victory Wellness, and we're calling it Sustainer Games. Throughout the week, we've invited the other sustainment elements, the FSC from 2-1 GSAP, to compete against our sustainers, and overall it's a great way to do the Victory Wellness Initiative. We're taking a whole week to conduct this with the ASB and to integrate it with the Army values of personal courage and also team building. The Aviation Support Battalion is made up of an entirely sustainer-maintainer MOSes. It's a very complex, diverse group of soldiers, and so we designed the Sustainer Games to fit into Victory Action Week in a way where each of those MOSes would be represented. So for example, for the 25-series MOSes, the Commo Guys, we did a Retrans and an OE254 competition. Our Charlie Company hosted that, and we invited the Brigade S6 and also another Battalion S6 shop to come down and compete and set up and do a Retrans site. My event in the Sustainer Game is the Lockware Challenge, and the purpose of it is to see how well our maintainers can perform under time to complete a Lockware that is 2-to-2-4 standards and is approved by a technical instructor who is the TI. My favorite competition for the Sustainer Games was a scamp competition where we had two cranes facing off towards each other, and it was a timed competition where they had to move a water bucket from one X to another X without spilling any water, of course. Well, I'm a competitive person, a very physical person, and I love to compete and I love to win, which means I hate to lose. So, you know, doing this without a doubt, if I'm going to come out here, I'm going to give it 100%. Even though the weather is bad for today, we're doing the FARP, the 4-point FARP we really didn't know.  What you see here, we're set up for a 4-point FARP competition. Two 8-man teams, one from the 601st Battalion, one from the 2-1 Battalion, are competing by time and to standard to set up a 4-point FARP. The obstacle course here is designed to test soldiers physical, and their confidence as they conduct each individual obstacle. With the risk mitigation due to the weather, we did have to take out some of the higher obstacles, if you will, and we incorporated some calisthenics along with the nine-line mediback. The Division back and forth, Riley, executed Victory Week right around the time of the Army birthday and that gave us several ideas and we asked the question, how can we do this as a support battalion while we're deployed? And so we looked at what different sustainer competitions we could do to nest a competition where we build a screed-a-core team building and also personal courage amongst our troops using the tasks that they're expected to do every day. So that's how we modeled it and it turned into a series of competitions. We invited another battalion and our Brigade headquarters to come and compete against us, which kind of upped the competition and the motivation a little bit and got after several different events that we integrated into Victory Action Week. And really it's part of a bigger plan that we have while we're deployed to ensure the wellness of our soldiers through several touch points each month that we dedicate and take time and set it aside to focus on the wellness of our soldiers and taking care of them.


So my name is Roman, this is James, and we both work at Cruise. Cruise is a self-driving car technology company that works on a lot of different things, but one of the main ones is trying to make a car drive by itself. So today we're going to talk to you about our journey through authentication and authorization at Cruise. So James. All right, thanks Roman. So just to sort of set the stage for the problem space that we'll be talking about here, I'll just start by saying that Cruise currently has about 2,000 employees. Of those employees, approximately 1,500 of them are engineers, and we've got about 150 services running at any given time. The majority of our services are running in Kubernetes with a mix of both microservice and monolith architectures that span multiple clusters, and most of the traffic that's passed around is HTTP-based. As for storing and managing secrets, we use HashiCorp Vault, and we use a typical cloud-based identity provider for managing employee and contractor identity. And so when talking about Authent and AuthZ at Cruise, we really have three different types of callers or clients that we have to take into consideration, and of those callers are browser clients, CLI clients, and service to service or application clients. And so I'll just start at the beginning. This is more of handling Authent and AuthZ at Cruise, to be honest. In the early days, it was sort of like the wild west, and this was really more, our talk is called crawl, walk, run. I'll say this is more of the embryonic stage, but it's worth mentioning, nonetheless. So in this early stage, we had, for example, engineers that are building services, for example, on top of our PaaS GKE clusters, and then in terms of how they handled authentication and authorization, they're basically on their own. The security team may have implemented policies or standards, which required service owners to implement these controls, but we didn't really have a whole lot in the way of tooling or standardization. So I'll affectionately refer to these as the bad old days. And despite my initial impression, the span sounds absolutely nothing like kiss, but nonetheless. I'll say that at this early stage, we had authentication and authorization sprawl. Teams were handling Authent and AuthZ entirely differently. For example, one might be doing OIDC with IDP1, and another might be doing OIDC or SAML with a different identity provider altogether. We also had naive implementations using static or shared credentials, and there may have been one or two services that had no authentication at all. And so we realized that that was problematic, and we had to start somewhere to fix it, and that's where we reached the crawl phase. In the crawl phase, the first thing we did was we just built a SAML authenticating proxy sidecar and just sort of hoped that people would use it. In this case, the service owner would add the sidecar to their pod, and then every HTTP request that hits the service will eventually, or we would enforce authentication via a SAML flow before proxying the request. And so to give you an idea of what that might look like, we'll take a look at this diagram. And we'll actually use a similar diagram throughout the rest of the talk as we introduce new systems and concepts. And by the way, this person isn't praying, they're crawling, but it was the best crawl icon I could find. But in this case, we have these different types of clients that I mentioned earlier. They're making requests from outside of our cluster, and they come in and pass through this nginx ingress gateway, and eventually we'll end up hitting the service. In the case of this early stage where we're using a SAML authenticating proxy, once the request would reach the service, the authenticating proxy would check to see if the user is authenticated, and if not, just bounce them over to the identity provider to complete authentication. And then once the user is logged in, the SAML sidecar would set an HTTP header that contained the subject, which was a user application, and then it would be up to the service owner to figure out what to do with that in terms of authorization. This also required a fair deal of setup for our users. First of all, they needed to set up the sidecar. They also needed to talk to IT to get a SAML integration setup with our identity provider or to have groups created, for example. And another problem here is that SAML as a protocol for the most part isn't really friendly to non-browser clients. So when people needed to provide API access, for example, they just passed a bypass header to circumvent authentication entirely, which obviously isn't good. And so obviously there were a lot of problems at this crawl stage. As I mentioned earlier, creating these authentication integrations was still a request-driven process that involved users reaching out to IT or an opening a ticket with them to have that done. Also, creating and modifying groups in our identity provider was still like a request-driven process, as well as assigning access to these authentication integrations. So now I'll talk about the next stage, which I'll say is the walk phase. And since I mentioned that we realized there were a lot of problems in the crawl stage, to solve them, we set out to do a couple of things. And starting by choosing an authentication and authorization standard. So for humans, like human to service authentication, we kind of settled on OIDC, which is the open ID connect, which is like an authentication extension to OAuth 2. We also built a custom entitlement-based authorization broker that we'll call C-Auth-Z. We also set out to build self-service tools to promote use of this standard. One of those such tools is something called identity orchestration platform, or IOP. And IOP has a lot of functionality, but one thing in particular that's worth noting here is that rather than users having to create a SAML or OIDC integration directly with an identity provider or ask IT to do it, we built a web app that allowed them to do that as a self-service option. We also wanted to build frameworks and libraries to foster implementation of these standards. This included things like opinionated auth-in and auth-Z libraries and commonly used languages at cruise. Another auth proxy that used OIDC rather than SAML and also was able to validate tokens coming from C-Auth-Z, as well as an authentication CLI tool that was used for retrieving and storing non-browser client credentials for humans. We also sort of settled on offline tokens or JWTs that were minted by C-Auth-Z. And for those that aren't already familiar with JWTs or Jots, they're basically usually the payload of a JSON web signature, which is like a JSON blob containing structured data or claims that can be cryptographically verified. Okay, so, oh boy, now we're at this diagram. There's a lot going on, and I can also tell that Roman made it in my case because he worked for Microsoft for a long time and he put the DLL icon on here, so none less. Moving on, so yeah, we'll go ahead and break it down. So we'll start with C-Auth-Z, which I mentioned earlier, and I also said that it has to mint tokens, but how does this authorization broker mint these tokens and what do they contain? So firstly, C-Auth-Z would need to know about or be able to reason about what users and groups exist at crews and what groups users are members of. Once the C-Auth-Z system is able to talk to the IDP in order to figure out what users and groups are present, it's then able, or service owners at that point are able to create applications in this system, and that allows this authorization broker to assign or allow service owners, rather, the ability to assign various permissions, I mentioned entitlements earlier. They can assign permissions to various users groups or even other applications, so it's sort of like a record of internal applications at crews that is used for authorization. With this information, C-Auth-Z is able to create these tokens, which contain claims that can be verified. Speaking of verification, the tokens themselves are cryptographically signed, like I mentioned, but how do you go about verifying them? In this case, we gave service owners two options, basically. In one case, they could use a library, like the opinionated library that I mentioned earlier, to validate tokens as well as to handle the OIDC authentication with our identity provider. The other option is the sidecar, which also does OIDC authentication and is able to validate these tokens directly from C-Auth-Z. The reason why some service owners may choose to use a library over a sidecar is, for example, if they needed more finely-grained authorization controls, for example, to be able to authorize based on application state, which is something that the sidecar would have no context of and only be able to authorize things based on the HTTP request or the path. Also, astute observers may notice that, hey, what about OIDC? How do you get a C-Auth-Z token then? In this case, we don't actually get a C-Auth-Z, like a JWT token. What happens in that case is, after the user authenticates, we do a real-time API call with C-Auth-Z to see what actions or operations are entitled to perform. Now, I'm going to pass it off to Roman to actually talk about some of the problems with this approach and then how we built the next stage of it to address them. All right. So, James talked about the walk stage, which was definitely an improvement over the crawl stage. No doubt about that, but it's not some problems that we didn't like so much, especially in security. So, the first most obvious problem is there's no enforcement of any of this stuff. Like, yeah, we're providing some libraries and some sidecars and people can use them, but they don't have to. We recommend that they do. But coming from kind of a security side, we definitely don't want to be in a position where this is an optional thing. So, if you're going to choose the library that, frankly, a lot of folks have, you need to change your code. You need to pull in this library.  You need to call it. You need to put in the right path. You need to also, as this library is changing, as we fix it up, if I'm bugged, fix them, you have to go and update it. So, there's a whole cycle there. And James mentioned that your permissions or entitlements are actually encoded into this JWT, which is great. But as these applications evolve, they become bigger. And as they become bigger, a user has more permissions to various aspects of this application. And all those permissions need to be shoehorned into the token. So, now this token is growing and growing. And these tokens, JWTs, are sent as headers, right? And then that means the header size is growing. And as that grows more and more, right, then you'll hit the dreaded, I think it's HTTP 431 or something like that. For 29, sorry. And, you know, that's really hard to debug, right? Because depending on the state of your headers at any given time, this will fail or succeed. So, it's just not great. You know, for us in security, right, that's great that we're providing all these libraries and sidecars for folks. But, you know, maintaining them is really hard. You know, at the end of the day, we're kind of a security team. We're not just like a development team. But we have to not only support these libraries, but support them in like four or five different languages, right? With their own, like weird gotchas, without really a common core. And, you know, as these languages change, you know, the libraries that they require change, and we have to like figure all that out. And then, you know, build them for every different, like every language has its own package manager. So, it's pretty difficult. But even then, you know, when we build it, let's say we figure all that out, and we kind of did for a while. To get folks to use the latest version is very difficult, as, you know, you might imagine. So, you know, what ends up happening to us a lot of the time is, you know, we'll release a new version. And, you know, maybe it'll have a bug in it. But nobody uses it until like four or five months pass. And then somebody starts using it, and they're like, oh my god, everything's broken. And we have to remember what we did five months ago, which, you know, we're really bad at.  So, in order to like figure that out, you would have to go to the repo of the application, understand the code, figure out the language they're using, you know, dig around, figure out the call paths. Super difficult to do, especially in a scenario where security team maybe needs to audit, you know, hundreds of these, right, not really scalable. And lastly, you know, kind of a high level problem here is that we, we be in the security team in this case, you know, we don't really control both ends of this equation. What I mean by that is we control, you know, we have the broker that's minting the tokens, but we don't control the code that's validating them. And the reason that's a problem is that it doesn't really allow us to iterate and make this end-to-end experience better by using, you know, maybe different tokens, using maybe tokens that don't have some of these limitations. The only way we could do that is then we have to go to every service and say, hey, you need a bunch of code changes.  That's, again, not scalable. So, to move on to the run phase, well, I know we initially said run phase, but it's more of a speed walking phase for us. And we'll presumptuous to say that we're at the run phase. But there's a bunch of things that we kind of want to improve here, right? So, the first thing is we want to be able to move authentication and authorization to an opt-out as opposed to an opt-in. What that means, kind of in simple terms, is that we want to just make this secure by default, so that if you have a service, you deploy a service as a developer, you know, it has authentication, it has authorization, right? Just from the get-go.  That's cool too, but then you have to opt out of that, right? Whereas, you know, kind of in this state, you have to specifically add authentication authorization to your application. So, the next thing is, you know, we want to, you notice we built a lot of libraries and sidecars and stuff, you know, which is great, you know, give us something to do for a while. But ideally, you know, we don't really want to support all that stuff going forward, as I mentioned, all the problems around it. So, we want to, you know, participate in this amazing open-source ecosystem and, you know, both use the fruits of other people's labor, obviously, but also, you know, contribute our own, right? So that we submit, when we solve solutions for our company, we're actually solving it for others as well. And, you know, also we want to move this policy code from just being very deep within the application. We want to move it out. Move it somewhere where it's a standardized place, where security teams can review it, other people can review it, and, you know, verify it and make sure it's okay without having to dive into the application itself.  Because, you know, frankly, anybody got time for that. So, next diagram, right? So, we have a bunch of new components here. I'll go through them. Don't worry. So, let's start here, right? So, the first move that we wanted to make is from using the NGINX ingress to using an Istir ingress. Now, why would we want to do that? Well, the Istir ingress, it is actually powered by Envoy. And Envoy has a lot of cool things outside of this. But one cool thing that we wanted to leverage was its extensibility mechanisms.  One of them is called external authorization. And what external authorization does is it basically lets you define another service that Envoy will call for every request. And it will basically let that service either approve the request or deny the request. So, that's very helpful. But in addition, if you deny the request, you can actually mutate it as well, right? So, we leverage that by having another service live alongside the ingress and hooking into the ingress via this external authorization mechanism that I described. And for every request, this other service called CAPI basically takes a request and says, hey, is this request authenticated? And if it's not, it denies the request, but it actually denies it by overriding the response and sends the user a 302 to the IDP. Now, the user completes the IDP flow, logs in. And because when we do that 302, we actually set a redirect URL for the IDP. The user gets bounced all the way back to where they came from to their original request. This time, CAPI looks at the request and says, okay, you're already authenticated with the IDP. I'm happy with that. I'm going to actually just set a cookie just so you don't have to go through this flow again. And then it lets the request through, right? So what this allows us to ensure is that any request that is coming from outside the cluster to inside the cluster has been authenticated, right? Just there's no way to do it otherwise, right? Doesn't matter which service you deploy. So the request has been authenticated, that's great. But, you know, we also want authorization, obviously. So how do we do that? Well, we decided to move to a different sidecar. You'll notice they're different from the one on the right. And this is an Istio proxy sidecar, right? Because now we're living in the Istio ecosystem. Now, Istio proxy, what's really cool about the Istio proxy sidecar is, guess what, it's also envoy. It's the same envoy. But this time, instead of living kind of globally in the cluster, it's living inside of the context of the service itself, basically, but doing the same thing, right? So it takes every request and proxies it through. Now, because it's envoy, guess what extensibility mechanism it has, external authorization, right? So let's use that again. But this time, slightly more appropriately by actually allowing or disallowing the request, right? So we need something else to hook into this envoy, right? To allow or disallow the request. We have CAPI globally, right? So what are we going to do here? Well, there's another really cool project called Open Policy Agent, right? So Open Policy Agent, what it allows you to do is it allows you to hook it up to your sidecar via external authorization. And then for every request, Open Policy Agent basically lets you write a policy in this language called Rego. And basically, the policy, if you think about it as implementing a function that always returns true or false, right? So you can have kind of a big flow of things that you can do. But at the end of the day, you're returning true or false. You're either letting the request through or you're not. And in our case, what we did was we actually used the same exact policy for every service in our cluster. We were able to do that because Open Policy Agent also lets this policy be augmented with additional data. You can think of it as like parameter store function, right? So the actual implementation of the Rego policy, really all it does is it says, okay, like, give me the routes that, give me the route that the request is trying to access, like the path, right? Let's say the request is trying to access my application slash admin. So it says, okay, you're trying to access slash admin. So it gets stuff from the request. And then it says, okay, this is additional data that it has specific to the application says to access slash admin, which permissions do you need? So it looks at that information. And then it looks at the request again to look at the incoming token, the JWT, right? That's attached as a header, as we said. And says, okay, does this token have those permissions, right? And that operation of matching the routes and their permissions to the token permissions is the same across all applications, right? The only thing that's different is that those two lists, right? The lists of, you know, which routes need which permissions and, you know, the actual token itself, right? So you might be wondering, okay, that's cool and everything. But, you know, as I mentioned, right, like the open policy agent actually needs to have some data that's specific to every application, like these extra files, right? So where does that come from? Well, the other cool thing that it has is it supports something called a bundle API. So bundle API is really just a polling mechanism using HTTP that it can pull various cloud providers. So it already supports polling, like a three buckets, you know, GCS buckets, like all that kind of stuff. So we leverage that. So we basically tell it to pull a bucket that's specific to the application. And in its polling mechanism, it already supports caching as well, right? So it knows just uses basic HTTP caching using intags. So it knows if there's a change. And if there is, it downloads it, loads the new policy, the new Brego, in our case, it's always the same, just loads the additional files, and it's good to go. Now, how does this bucket get populated? Well, remember, James talked about CODZ having all this context, right?  Well, all we did was we took CODZ, right? We serialize all that data on a regular cadence, and we write it to each application's bucket, basically.  So it wasn't, and it isn't. So, you know, let's kind of go step by step to see what the challenges were. So kind of how we dealt with some of them. So basically, moving from NGINX to Envoy, you know, that pretty much involves, for the most part, taking an ingress resource as defined in each service and converting it to kind of a matching virtual service resource. They're kind of analogous, not exactly, but kind of. And Istio has a utility that lets you kind of convert from one to the other, but it doesn't always work, you know, depending on the complexity of your ingress resource and the routing rules that you set up. So, you know, that required some hand-holding and kind of doing some things by hand. One thing that I want to mention that we did there, which was a good idea, and I think it helped folks, is we actually ran both side-by-side for every cluster. The way we decided, you know, which, like when you do a request to a service, which ingress to use is just simply by the DNS. So the DNS of every application was mapped with CNAME to the DNS of whichever ingress that we're using. So in the beginning, they were all CNAME to the NGINX ingress. And as folks, you know, made changes, right, they could actually create a new DNS that CNAME to the Envoy ingress, to the Istio ingress, and then just test that out on their own, you know, mucking around with a virtual service. But while they're doing all that, nobody, you know, the ingress, the NGINX ingress isn't looking at that data, so it doesn't break anything. So they can iterate, they can make sure it works. And then when they're happy with it, they just switch over the DNS to the Envoy ingress, and off they go. So that's kind of how we are handling the NGINX Envoy ingress migration. So with CAPI, you know, we obviously want to enable it globally in the whole cluster, but you know, we got to start with still an opt-in model, make sure, you know, things don't break. And the way we did that is we basically had people add an annotation in the virtual service. And then we have CAPI basically scanning the cluster and saying, hey, if I see a service with this annotation, I'm going to authenticate traffic routing to it. If I don't see that annotation, I'm just going to step out of the way. And this is kind of like a solution for the migration. Now that code is going to get completely removed, you know, once everybody's migrated. So you might be wondering, like a lot of these applications, you know, as services, as James said, you know, are using these libraries, these legacy site cars, legacy libraries, maybe doing things on their own. So how do we handle that? Well, the good news is that's totally cool. Like we can have multiple, we can do authentication multiple times, and that still achieves our goals. So for example, what can happen is you'll get a request into the cluster. CAPI will do that whole magic of authenticating the request. The request will go to the actual service, and the service will authenticate it again, right? Sorry. Authenticate it again. Security people don't install their updates. Anyways, but the good news is that our IDP already has a session for that user. And so really, it'll just be an additional redirect. So from a user perspective, it's not a perceivable difference, even though authentication is technically happening twice, right? And that like kind of latency of that additional redirect is only present while you have this two authentication mechanism. So once that's removed, you know, you don't have to do that. But the cool thing is, you know, we can still enable it and say the entire cluster has authentication. Some folks have it twice, but none have it zero times, right? So that's the goal. And then we can also integrate with some of these libraries and site cars that we provided previously because, you know, we do own them. So we can, you know, add some code there to make CAPI kind of more intelligently reason about some of these services. Yes, it requires folks to upgrade to the latest versions, and I mentioned some of the problems with that, but that's kind of another option. So moving on to enable the STO proxy. STO makes it relatively simple. You just need to set this annotation and STO proxy will be injected into the pod. You can set this annotation at, you know, the namespace level, the pod level, a bunch of different levels. Once you have that, you can actually start working with the OPA agent, the open policy agent, and that's again opt-in. So you set this OPA agent enabled annotation, and then you get this additional site car. So now we're up to two. I'm going to see how many site cars we can shove into every pod. And as I mentioned, you know, there's a single Rego implementation of the policy in every, for every service. And really that's doing what we'd like to call stateless authorization. So basically it's just looking at the request, just looking at the token, just looking at the local data that it has to decide if that request should be authorized or not. Now, if a service needs to do finer grained authorization, so for example, it's maybe has a matching database that it needs to look up a user and kind of augment this authorization data, it can still do that, right? So the idea is that, you know, the OPA agent does one authorization step, right? And then it can be authorized further by the application itself if they so choose to do that, right? So again, you're doing kind of authorization twice, but one is coarse grain, one is fine grain. And then lastly, our CLZ broker, right, needs some new UI so that you can go and define some of these policies. You can define, you know, different routes, as I mentioned, like you have an admin route, and these are the permissions that are required to access it. And, you know, obviously, folks don't just want to use the UI, they want to use the CI, you know, the current CI pipeline to deploy some of this stuff. So we also have that as well for them. And then lastly, you know, as I mentioned, we want to in the background serialize all this information, including the group memberships and push all that to those buckets that then the OPA agents can read. So kind of, what have we achieved here at the end of the day? Well, I think we can finally make this statement that all services in our cluster have authentication authorization. That's not something we've been able to say before. Before it was kind of like, maybe they do, maybe they don't. Authorization policy now does not live in the depths of every service. It's actually lives in our broker, right? That can be reviewed, can be exported, or it can be audited centrally, right? We're no longer using, you know, custom tools that we've built and have to support, right? We're using open source tooling. And the best part of this is actually, you know, we're not proxying the traffic ourselves, right? We're using Envoy, which is, you know, kind of a battle tested piece of software to do all the proxying and traffic management and all the cool stuff that it does. We're just kind of hooking into it to provide, to augment some of its capabilities a little bit, right? But we're not doing it ourselves. And best of all, you know, now we kind of control both the minting part of this equation, right? The creation of these tokens and also the validation, right? And all the way through the entire channel. This means that, you know, if tomorrow we like say, hey, some new standard comes out, we don't want to use JWTs anymore, or we want to sign them differently or something, right? We can actually change all that without every service having to modify a bunch of code. So that's pretty cool. But there's more, right? There's more, that's the image I should use. So there's more that we can do here. We, because we have these Istio sidecars, we can actually do transparent MTLS between services, right? Why can we do that all of a sudden? Well, remember that those Istio proxy sidecars are doing both ingress and egress filtering for each service. So now, really, you can have these two sidecars establish an MTLS channel between themselves. And so whenever an application wants to talk, a service wants to talk to another service, it actually funnels through those sidecars, right? So it goes egresses out of its internal sidecar and ingresses into the destination services sidecar. And that's even cooler because now you can actually have MTLS between the ingress itself and the service, right? So that channel is usually not secured, right? So like, for example, if the ingress were to add any headers, right? As part of that request, technically speaking, you can't trust those headers, right? Because that channel is not secured. But if you have MTLS between those two endpoints, those headers can now be trusted. Also, MTLS, you know, brings you certificates, certificates usually bringing you some form of identity that can be used here. And, you know, we have that now. And so now instead of using something like API keys for services to authenticate and authorize themselves to, like, say the COG Z broker or other other systems, they actually can use their own service identity that is actually rooted in the cloud provider's identity, right? In terms of the rego, you know, we've had, I'll give you one example of a request that we had. So, you know, somebody wanted to authorize based on the method of the HTTP request. So as I mentioned, you know, you can only right now authorize based on the route, right? So if you slash admin, right? Sorry. But if you wanted to say, hey, you can access this route, but only as a get and this other route only if it's a post, you know, that's not possible in our the rego that everybody uses today, right? But it'd be really cool. It'd be a trivial to add this functionality, right? So we want to move to a world where this rego is actually can be modified by services service owners themselves to fit their needs. But we don't really want to just unleash rego on them, as is very complicated. So what we're thinking is we want to provide some libraries, as well as kind of like a CI pipeline with some linting validation and various, you know, approvals and various other things, so that when they make a change, you know, we can have some confidence that it is doing what they're intending that it would do. So, you know, if this is interesting to you, come join us. We're hiring. Yeah, we also work on self driving cars. Okay. And so I'd like to a special thanks to Natesh, the top there that he couldn't be here today. And it's also actually his birthday today. So happy birthday, Natesh. Cool. All right. Yeah. So the question was, how do you do service-to-service authentication and authorization, I guess, and do we use JWTs or MTLS? So right now, we're still using JWTs, because we're not fully in the STO model of having MTLS available. Now, we do when we do have MTLS available, what we're thinking is we would use MTLS is again, use the same core screen fine grained method, right? So use core screened authorization to say like, Hey, this service can maybe talk to these three other services, but nothing else. But then layer the JWT on top as the user identity that's actually making that request. So kind of use both. Yeah. Yeah, that's a really good point. You know, right now, we're just simply scale it and use Kubernetes kind of to handle that, you know, if it blows up, we have more instances of it available. And we also have multiple clusters. So we have CAPI kind of one per cluster. So we don't have one like globally, right? So one is kind of matched up within Istio ingress and you have, you know, one Istio ingress per cluster. So I think we just have maybe more clusters in that scenario. Maybe you folks have less. But yeah, that's possibly something that we'll have to consider in the future for sure. That's a good question. It was sort of the predate in both of us, but yeah, I think sort of like the idea was like to Oh, sorry. Yeah. So the question was basically that CLZ broker, you know, why did we decide to build it and why, you know, we didn't use something off the shelf? Yeah. So yeah, I think it did like predate both Roman and I at cruise, but I think like it was originally something that was built by like a product team for things to do with with like our product and the car and sort of like we inherited it because it was responsible for like dealing with, you know, making authorization decisions. I think it also like provided, you know, quite a bit of flexibility in terms of giving these like service owners that are like registering their applications with this CLZ system, the ability to like define like, I mentioned the entitlements. So basically like being able to express like what operations are able to be performed by different human and non human subjects in like a really flexible way. I don't really know of anything like there are a handful of other, like I guess maybe my follow up question would be like, like what alternatives are there? Like we did also like at a later point kind of an investigate using like OAuth for authorization, but it really isn't the same type of, you know, it didn't really work very well for us in terms of, you know, it's like handling like a delegated authorization. But yeah, I don't know, we'll go, I think you in the back or first, yeah. So the question was if you have like automated testing, is that so like if you have like you're trying to do like load testing or something like that? Yeah, yeah. So we right now, honestly, we don't have a great solution for that right now. We use, but that's a problem that's existed in the past with basically all our other solutions as well. And so, you know, sometimes people will have like test accounts that they'll use in a different dev environment as opposed to using the production environment. And then have a cluster that's specific for that use case. If they need complete end to end, right? There's nothing like if they're already in the cluster, right, then they can do that without ever interacting with Cappy. We're at the end of this session. So if anyone has questions, you could bring them to the speakers. The live stream is going to end now. There are additional questions from our virtual attendees that will be placed in the Slack channel for y'all to answer. Thank you.