Thank you very much for coming So I know deployment of rails is not an interesting or exciting or remotely as cool as Kubernetes and containers and Docker nowadays So I'm gonna get that out of the way But today I thought I'm gonna show you what we've built what we've got and how we can help you and your team Do rails deployment? But before I do that There's a little bit of kind of history around why we did this why why we ended up doing the Non-interesting solved problem of or we already have Capitano. What's wrong with that? Kind of problem and how we got to where we are so Classic six is started as a as a different company with a different name doing a different thing and we were a rail shop We're still a rail shop and we did this Like everybody else we started with a pass provided our friends over there heroku. We used to be their customers so like everybody else we started at heroku and The So I've been doing this for a long time not not classic She's but I've been you know like you you guys been doing rails for a long time And I've worked in large and small companies and every time I started heroku and I leave her Now the question for me was why why why does why do we all love Paz? like engineer or heroku You know we really like it, but why do we all leave and and We started asking this question as a diff has a different company that we wear when we started thinking about leaving Paz We started asking this question from other people like why do you leave do you start? I'm just gonna risk my whole career right now. I'm gonna ask you who has a started something at heroku in this just trust your pants You guys done heroku before Okay, and how many projects like say, you know, you start with heroku and you end up not moving out Have you guys are you still on heroku? Or what you started? You are Okay, so this audience might be a slightly biased because you are here I'm guessing that you want to answer that answer that question, but what we see is kind of what I call a Paz cliff We all start with Paz and we fall off of that cliff pretty soon And the reasons are kind of different. A lot of people say cost gets expensive very quickly Which is an odd one because you know if you're doing really well Then we should be able to pay the bills and but somehow it's not sometimes it could be something around flexibility Oh vendor lock-in Regardless of what that is I've even had cases that I used to work in a bank in large bank and you know I wanted to put a proof of concept or like something you know Just just a feasibility study sort of thing project together and I did that at heroku and just like a pass I just went out to my colleagues and showed it off and I really loved it and said great Tell me have it. I just want to put sales people on you're gonna sell the hell out of this thing I'm like It's kind of running on a past thing. It's like whoa. No, no, no So ops guys coming and they take over and they want to run it in like in a kosher way and on all sorts of things I'm just like, okay. I'm never gonna start with Paz again so we jumped and The jump turned out to be a different business. It actually turned out to be our business So this is kind of what it looks like you jump And then you realize there's a lot of duct tape and that kind of stuff involved So today I wanted to instead of talking about this I wanted to show you What we've done and what this jump looked like what we had to build ourselves To be able to deploy a rails application in a way that gives us the same experience that we really love about a pass provider But we are in charge. We are in control of the cost We benefit from the cost-cutting or the price wars that are going on between say Amazon and Google and they keep cutting The cost of every CPU cycle and we don't get the benefits of when we're running on a pass. We also wanted to Be able to experiment with new technologies like you know We have time-series data and then we want to go and use a time-series database or use boats or some other cool technology That our developers want to use and all of a sudden we can't find another pass provider So we have to go and add it as an external entity into the whole deployment and then it creates these things called snowflakes and you know when you get an email about Amazon having an issue with heart bleed or Meltdown or whatever else and they say you know what just shut down your ec2 instance and restart it and it moves into an improved thing And you go everybody goes oh That's there. They want to shut it down. We have some stuff on that and you know It's supposed to be just shut it down started run the puppet or a chef script again And it should work, but we all know it doesn't so let me show you a quick demo of What we've got what we built So this is class 6 6 dashboard what it looks like and On the left I have one container based Kubernetes ramp all the cool stuff Application that I use for other demos when we are talking about our container products We have four products two of them are framework rails and node and the other two are skycap and maestro Which are about Building a Kubernetes cluster and also deploying your application into an existing Kubernetes cluster So Maestro builds your Kubernetes cluster for you and skycap Deploys your application on to any Kubernetes cluster that you might have where there is Maestro or you get it from say gke From Google from Amazon or whoever else, but the but the focus of what I wanted to show today is on On the rails product So How do we start what we wanted was we wanted to build something that Starts from the code and ends up in production. So if you think about You know your your typical paths experience on one hand You have your code and it then goes to production. Well, that's great And every time we commit into git it builds, you know does the tar balling and all that sort of stuff and then rolls it out What we wanted was to just slightly modify that image that model and say on one side We connect your git repository and we get the code on the other side You connect your cloud provider of your own choice under your own account We fire up the servers that we need We then configure them provision those servers to run your application and we do the deployment and the catch here was How do we know what kind of service we need? It's not like a generic server that runs everything, right? So if I go and read my rails application, it says I have Dali So it means that I have memcached D and I have some sort of redis gem So I need redis and I postgres as well Then what kind of servers are we going to create? I'm going to create all different servers like one each or put all of them into one. How do we go about doing that? So this is how a flow kind of looks like and these are the different products that I mentioned so I go to rails and I'm gonna Choose one of our sample projects. So this is a simple rails application as you can see It's typical rails app that we put together It uses my sequel but You know, you can see there's a docker file here, but this is this is not used in this demo You can also see there's a classic six folder here with some Extensions around like you know environment variables that you want to put in you don't need to use any of that This is not necessarily use use part of this app, but we'll get to that later So I'm gonna just start deploying this take the git repository URL Go here Use that. It's a public repository. So I'm gonna use the HTTP Use the branch master call it rails com Three and I'm running a live demo so demo gods are gonna help hopefully But I have some that I've already prepared earlier I Don't have a video of it. So if there's a Wi-Fi I'm toast you're gonna have to improvise So what have what's happening now? We connect into the git repository. We pull the code out and we look at some, you know pointers Jem file Jem file lock database YAML Also some other configuration files that we all put there So the great thing about rails is we all know it's convention based so you can you know Use a lot of things from those conventions. So what did we find out? We are using rails in production Which we are specified as the environment. This is using rails for two eight and the ruby two one five and we've Flipped everything to run on fusion passenger here. You can choose to run unicorn Puma if you want But that's the default setting which means that it runs engine X with fusion pat passenger on top of it The back end is my sequel obviously wasn't like a super genius thing to find out. It was in my gem file and Deploy hooks are what we have In the dot cal 66 folder which basically says I have this custom package I want to install on a certain type of servers So you kind of use it that way and the reason for that is instead of you know jumping on a server with a bash And just installing the thing so the next time you don't remember what you've done on the one specific We want to kind of try and nudge people into doing it the right way and scripting it out instead of doing it post deployment We also do support proc files, which we all know if we have used Heroku, but it's basically back-end jobs and think that one So if you think about classic six we do a lot of dog fooding we saw we installed a deploy classic six with classic six It's kind of an inception sometimes but The idea is that If you think about our process and how we manage these things so we run rails God six is that itself is rails and the back-end jobs are on a sidekick Now if you think about how long does it take to get a server from say AWS about a minute Maybe two then you have to install packages on it and reboot it and all sorts of things So end-to-end your deployment might get from five minutes to 20 25 minutes We have customers in about 110 countries There are about 12,000 developers on a daily basis about 600 of them use classic six to deploy about on a five six thousand times You do the maths you realize that there's no time that any sidekick is going to be quiet So how do we deploy and that's the kind of the thing that we had to build and it turned out to be a good product as well to build the business around and Kind of solving God's problems. How do you do those things? So when I mentioned that around proc files I wanted to tell you that those are the things that are we try to take care of automatically for you because rails is such a great Framework that's got lots of opinion baked into it Here you can add environment variables if you want I've locked in my Ruby version here in my gem file But if it wasn't locked, I would be given a drop-down that I could choose what I want As you can see, this is a very rails specific Or has a deep understanding of what rails is so you can you know understand things like raise acid pipeline Or debu schema loads and things like that now on to adding a cloud provider. We do support About I think nine providers from the second as well as bringing your own service So if you're running on any of these cloud providers, you are all good You can just connect it to your account depending on the authentication method of that cloud provider If there's AWS you can just get the API keys if the solution is off and all sorts of things But if you happen to run on say head there or VH or some bare metal provider Apart from packet, which we do support here natively. You can then bring them in with something We call registered service, which is drop a bash file a bash command Onto the server it calls back home registers itself You need to have a bunto on it and that's basically it and it becomes part of your what we call a cluster So I have added my dissolution account here as you can see and I get to choose what region up to Dislotion I want to deploy to and each region has whatever server types that it supports I get to choose those So here's a part that I was I was I was telling you about how do you kind of thought what's a topology to use a very big word But how do you actually distribute this app? So for us the rails part sits on a a server at this point and On the other part my sequel in this case, which is the only other component that I have in this example You can choose to say share with the rails, which is not good practice for production, but for the purpose of this demo I think it's fine You're gonna forgive me or fire up a new server on the same cloud same account same data center same availability zone if you want to or Use an external server from like, you know registered server somewhere else Which brings me to hybrid you can combine multiple plans with this as well as that What you can do here is that You start with a small skeleton of the application We don't want to say tell me everything about this up front you have production You don't know how many servers you're gonna need you don't know how it's going to actually you know turn out to be So we start with one server of each type if you want and then you can scale out as you want all right So let's just go ahead And click the appointment now what's gonna happen now? So we're gonna connect to the solution if I have my dissolution log in still log it in so in a minute That guy is going to fire up a dissolution Server here There we go So that started that we give it an animal name you can see gazelle in color sometimes they come out like compassionate pigs and some weird names, but I think you're lucky today I'm just gonna give this a Different name I can rename it here. I'm just gonna put along here and the reason for that is we have a process in the background As you can imagine we fire up a lot of servers on a lot of clouds So we call it janitor and comes and cleans up all the servers that be fired up If you don't put along in there in the middle of the demo the server is going to disappear Which is a cool thing you'll see how it how it works But um so what's gonna happen now we fire up the server or server is that we we agreed Then we check for the compatibility based on the base best image that's available on that cloud provider Then we check for the kernel if there's a need for a kernel update. We do that Any packages that need update is all sorts of things setting up automatic updates for security You know installation everything or anything that's needed. So for example in your gem part Let's say you have image magic then image magic requires a bunch of things that you need to install We install all those all those things if you have any custom packages that you need to install that's also done and All of that then ends up being a server with an IP address that then called back calls back and then gives us the IP address If your cloud provider supports private public networks, for example a private Nick and a public Nick we do get those IP addresses back the IPv6 and all sorts of things around that we get it back You get a chance of kind of drilling down into all the locks What preparing for example of the server is then here you can see you know we install a ufw as a firewall manager All sorts of Actually in activities that are happening right now. So let me go back to one that I created with the same stack So this is what it looks like at the end. I've got my server created At the top when was it deployed just earlier today? I have a rail server and SQL server and a process server which holds my Proc files if I go in there So there's a worker in a schedule. I think they are Sidekick that are running here and my my SQL server Yeah, I Can I can see my my SQL server with my SQL things that are running on it? But it's sharing a server. It's obviously sharing a server with with my rails one Okay, so let's just see What the sites like It's just a simple rails after dump whatever it gets in the header If you notice here, it's a DNS record that's created for this and this is quite an interesting thing. So Obviously every server that you fire up has a public IP address You can access that and you can block that and say I don't want this application to be accessible publicly You can do that We also create a DNS record for you the same way that for example for oku does and that is pointing to the head of that Stack if you have a single server it points to that one if you add a load balancer Then it points to the load balancer it changes that and it's a short response in a short TTL to point to that server As well as that you can have you can have a DNS server that points to every single of one of those servers And you can change your DNS record to point to that So, okay, so now I have my stack great now that was a that was easy now What else can I do? So now at this point you have your application up and running? It's working But it looks like this Now here you can add a load balancer or add your backup backup to your databases so As you notice I deployed this on digital ocean and I choose to deploy it without the digital ocean load balance I want to have a chip proxy I can do that if I were to deploy that on to say Amazon Then here the load balance option would be an ELB or an ALP that you can deploy and that Decouples you from the intricacies of that cloud provider which means that you can get the application and move it from one cloud Poverter to another one with the same topology you can say I have a unit might you know a load balance under my sequel I can just move it from one place to another one So you can do that you can also create database backups for my database So here I can say I want to manage backup which means the backups taken up your database Sizz all databases So if you have like say my sequel postgres and red is all of them as a snapshot are taken at the same time The files are then compressed encrypted with your passwords in the ship the closest s3 Region that you want or the one that you specifically want if you're for example running it in say Germany There are data resins and residency requirements means that you have to move it only to the Frankfurt s3 All those things are taken care of how frequently you want it What do you want it to binary one which means it's much faster or you want to text one which you can then inspect manually And you can create a backup here. So this is stack the one that I showed you earlier This one has both databases back backed up and load balance And here I can download it or restore it And when I say I want to restore it It basically goes to all the databases that I have postgres red is everything that I have and it just restores that back So if you combine that with the code and get ref that is deployed you have a time machine You can go back and forth back and forth in there one note is that you can use this with Your own database if you want so if you're using you know aroku postgres or cloud sequel postgres or RDS from it Amazon you can do that if you want to You don't have to use the databases that come here and the same with load balances and the last one around here Is that as a certificates if you choose if you have your own as a certificate? You can add it here And it will add it to the load balancer or to the server engine x depending on the topology So you just say I want to have a cell and it will add it to the load balance as a load SSL termination if you have a load balancer if you don't then it will be added to the engine x and if you If you say I want to start to small and I would just want to add an SSL certificate to my only server that I have at the engine x level and it just works and sometimes if your colleague comes and says Why is this thing running without any load balancer? They will go and add a load balancer. We then detect that we move the load balancer We move the SSL certificate from engine x all the way to back to the to the load balancer So it doesn't disrupt your service. So that is not going to break anything And if you choose to go with let's encrypt we call let's encrypt API we get the SSL certificate We put the reminders in there and we do it automatically every three months So you have that we are in the process of supporting version two which has wild card as well So that gives me the kind of the basic running of the rails how do I set it up and all sorts of things now Here you can see that I have two processes and they're running the both green all good Now database is a little bit more on that so Here we have a we have a database a single database. It's my sequel What I can do here is I can create replication. I can simply say I want another database server That's essentially scaling up the server database server for me Now here what we do We natively support replication from my sequel postgres redis memcashd elasticsearch And I am MongoDB So we do the replication for each one of them the native way they actually support So you talk about my sequel we do a master slave. You want to do postgres We know what write to head files and how to sync them all sorts of things and Not only do we do the replication but the system also supports replication verification across different data centers So you oftentimes when you set up replication You see that you have like two different availability zones and you want to have like high availability So you put the master on one side and then the breed only a slave on another data center and the connectivity sometimes not great If you have a lot of throughput in terms of writing a lot of inserts the slave might need time to catch up And they go out of sync So when you set up a replication with classic six rails stack Repetitions set up is verified and it's constantly being checked For whether it's in sync or not and we keep it up to date or if we cannot for example You just lose connection between the two servers will let you know these are all the things that we had to build to just get back to where we were on a path provider and On the other side if I go to my source code Now in this case, I had a backup verifier as well. So This is a as you can see it's in my sequel query what happens is this is a backup verifier So if we find that file Called under that backup verifier for my sequel for production that specific environment It means that every time I run a backup I want you to put that ship that backup to another server unzip it decompress it unencrypted Encrypted and then put it into Into that specific database then run this query and this query should return a true or false Which means we verify the backups every time and you get a green tick for the verification of the backup as well So it's not only about You know whether you have a backup but the backup is going to work when you need it because that's the worst thing That could happen if it didn't Now onto the deployments here's my deployment and This is the one that's live and every deployment that I do is going to show up there So I can see the commits that have gone into this as that gives me a visual history of what's happened on Configuration files now, that's an interesting part. So This is not a you know pass obviously so it's not lots of things are hidden But we don't didn't want them to be hidden like for example, we install engine x We want you to be able to shell into the server that you whether we've set up for you And you look at things you look around and kind of feel that you've done it yourself It's just done for you. It's not an alien black box as a result. For example, if you think about deploying engine x What would you do if you were to deploy engine x yourself? You would write the configuration file for engine x deploy engine x and apply that configuration plan and this is what we do here so if you're familiar with engine x configuration you recognize that that as a Engine x configuration with a small difference of these double curly brackets around here Now if you've used Shopify You know about Shopify liquid for example, which is a rendering way of the safe way of rendering or markup language If you will for for files and this supports that so you can say you know number of working processes Comes from this variable or you can have if statement for example if you want Around this what happens here is that then we get those base template That we have here and then we render it with the values that you have and then we generate a Real executable engine x configuration and push it to all the servers that we've even installed engine x on What does that give you one it gives you consistency across all the components that are installed So there's you don't have these snowballs and snowflakes that have one of them is engine x is slightly different Sometimes it happens and this happens to us that you know something goes wrong I'm doing the weekend some up sky jumps on a server shells into it and fixes that and nobody knows what's happened One of the servers go out of sync so we detect those differences and we bring it up and we show you that there is an issue There is one of them is different this not just again will not just into Doing it the right way, but if you don't you still get that notification The second thing that it gives you is that it gives you kind of a constant update so We did this for a long time ago. I think 2013 13 if I'm not mistaken and Right after that The Snowden thing happened and they said oh not only NSA reads whatever you say, but they record the encrypted traffic So if they won they can then run something afterwards which meant all of a sudden all of our customers were asking for perfect for secrecy Which will prevent that from happening So what we did with this was this is back by git as you can see there is a git repository backing this So you don't need to use the web UI if you're familiar if you if you're more comfortable with using git Which like I am I go to command line clone that git was there and do my thing But cloud 66 acts as a yet another developer in your team So let's say tomorrow Snowden says something else and then we're gonna have to add that you're gonna have to go and say You know I need to add perfect for secrecy number two to this or there's hard bleed or engine x patch We pull that repository We make the patch and we try to merge it like you're you would do and If it goes in great, you get to choose to say I want to deploy that if there's a conflict You see the diff file use you can resolve the conflict and then push it in that means you get to benefit from All the things that happen in our network now as I said That's a six product rails product power is about Two thousand five hundred unique workloads something about five thousand to six thousand different applications with different sizes So oftentimes we find ourselves in the position of running in front of our customers finding out issues like a canary in a Kind of a coal mine and then we bring it back to our customers before they before they face it They see that there's a patch for some contribution file. That's there But if you want to manage that yourself you know feel free you can do that and if what you do Doesn't conflict with any patches that are coming in great You're going to benefit from that and as you can see this is installed This is available to you for all components that I installed. So it's not just the engine expert, but also You know any my sequel for example has its own native one As well as my hf boxing so I can customize everything to my heart content if I want to Now onto the network settings Here is the firewall settings that we have so From cloud 66 which we have a publicly available set of IP addresses We open 22 if you want to have a bastion server in the middle to lock us out when you don't want to when there's no Deployment happening or you want to have a bastion server that we do have access to but your ops guys has to give Permission to your devs guys to deploy during a certain window of time you can do that But these are the kind of statically created Dynamically created burst that if you applied a Set of fun configuration files Sorry firewall rules and here I can create the ones that I want for example from anywhere To my rail servers. I want to have TCP on a specific port that I have now The best thing here that I find is this We had a case of a customer who had a CRM application managed by one team The CRM application had a database the database had a list of customers the marketing guys have another thing that would generate some spam email And send it to some of the customers the issue would come up that two teams were taking care of those two stacks Sometimes there was a need for a load to because there's a campaign for example going on They would scale up the servers on the marketing side that will fire up new servers will give them new IP addresses Then those IP addresses don't have access to the sequel server on the other stack Which had the list of customers So every time that would scale that would have to go through the other ups up to the other The other team and say can you add this IP address list and worse than that?  And that's just an security nightmare So what we had to do was to say you can't create dynamic firewall rules You can say from any rail server of this stack that I don't have access to I don't know what it is To all my my sequel servers on this side and these are dynamically calculated. So as you go and scale up your my sequel They inherit the right firewall rules as the other team goes and scales up their worker servers on the other stack Then whatever rule dynamically is created this generator. So this is we call it kind of an application century firewall as well as that I can I can I can be naughty and Access the servers directly if I want to with having a lease We just opens up with 10 to 20 minutes access and it shuts it down based on the IP address That I have if I want to really get into the you know the servers themselves traffic redirection rules and And traffic access rules, you know basic things that you have there One of the final things is active protect which is Luckily, I don't have anything there What we do is that we monitor your servers for you at different levels at the OS level at the patch package and pack patch level if there Is any known vulnerabilities that are there as well as that For every deployment that you do for every component that you add to that There are certain files that you want to monitor if you add Docker for example There are some files that you want to monitor if there's my sequel you have some files that you want to monitor for change Whether it's change inadvertently or maliciously you want to be sure that you know about that change and those things will show up here As well as that if there's an attempt For brute force access to your SSH or hitting your web servers repeatedly, they will be blocked and shown here And that's the change files that I was I was talking about environment variables played pretty much, you know self-explanatory with a small Kind of caveat that you can refer to different stacks environment variables as well So for example if you get in the example of the two stacks that I want to talk to each other If you need the access credentials to my sequel of another stack But you don't know the in the password and it might change in the future because of the password rotation You can use the environment variable coming from another stack that you don't have access to and it gets rendered and Sent over to your servers without your developers having access to the real password itself and every time it changes on one stack it gets changed on the other side and This is a basic kind of notifications that you can get on slack email web hooks about pretty much anything that happens in the system and Lastly live logs So we didn't want to build a lock shipping and storage facility There are really good solutions out there from you know, log lead to others that did that do this But we found that oftentimes we wanted to take care of just be sure what's going on right there as a live Logging mechanism so every time you deploy something new or your application the log files for it will be added here And you can just look at them here. You can have a look at the context around them If you want or you can have kind of a basic search around it and this is real time We don't store it It just gets shipped as you enable it and it will be shut down after 30 minutes It's a very good facility that we found for debugging Now This this this stack, I guess I'll have a connectivity issue But on the ones that I've created as you can see I have this tiny thing called stack score here That gives me an a to be a to F green to red Scale and if I go there gives me some hints about what's what's wrong So code we do some code quality if you want to connect it to cloud code climate You can do it as well. We don't have any backups on this production. So that's bad. That gives me an F score Connectivity I'm sharing on production. I'm sharing like a database on the front end So that's also bad and you can fix all those or ignore it if you want and These kind of show you and you get emails if you the stack scores drop or improve as things kind of change So that's what I wanted to show you Very quick run on everything that we do support obviously you have the basics of You know account management as well as the teams which are very popular with with agencies that we work with If you look at the organizations, you can create multiple organizations that you that you have and you can switch between different organizations So if you if I switch my organization to another one, I see a completely different set of stacks that are there and And you have full access control on every single one of the actions that you can do Give it to different team members and we see that in especially in agencies You have people are assigned to projects and are taken off the project Or you want to get you hand over the keys to one project when it's done to your client And the client wants to get one of your engineers onto their stack again because of some maintenance or improvements And they can invite your engineers into this and then take it and take the rights away So it's got all those kind of features in their built-in as well as that you see you have a you have audits if you want Of every action that's happened Where did it happen? What was the action for example to change in the environment variable to keep the history of all this is basically kind of very specifically built around rails, but with some Good ops practices around it that will allow you to run your run your operations the way you used to having a pass But it just works much more smoothly Now I've got about two two or so minutes if any any questions, please Let me know Yes, okay, so the question is how do we manage memory spikes or manage your memory memory usage of a process that That's running on the server. There are three things here one is that Based on the heuristics that we find so we fire up on a summer Thousand-something servers a day not all of them keep running But you know people fire him up and they fold it down and based on those We collect a lot of metrics about the behavior of the cloud provider with a specific component So if you run redis on this solution Netherlands data center am s1 with this kind of server This is the kind of a footprint that we see and based on those those configuration files that you saw the custom config We try to map to change those variables the variables that get ended end up in the rendering based on those heuristics So if you deploy my sequel on to and I'm talking about these kind of prepackage processes like my sequel if it deployed to a Ws and then you take that stack and move it to this lotion It might be that those variables change you can influence it you can fix it to whatever you want But we apply those heuristics that we've learned on to that to optimize the performance of that That's the first one. The second one is about the spikes and everything else so we do collect the vital signs of the server by the memory the processes and disk and CPU usage Those are kind of collected and you can see it for the past I think six months You can you can have a look at it and the disk you can also had a threshold and a trigger that gives you an email Something goes wrong. You can get it Or you run out of this because operating systems don't really gracefully handle this just But it's and it's based on processes So you can get the process metrics out so that's kind of on the metric side We also allow you to run side cars and side processes as well to shut down something So for example, say you have a huge you add a huge asset to your rails application and the npm Asset pipeline compilation just blows up You know gives you an o.m. And just blows up the whole thing We can take care of those things as well some of them are rails specific like we see a lot of that around npm I'm sorry note asset pipeline compilation that we take care of we know what it means for your rails and stack some Of the word generic you might be a stack You know memory leak in your sidekick that you have and we take care of that by either killing it Or you are reverting back to the kind of a native OS behavior around memory management. Does it answer your question? Cool. Any other question? Sure. Yes So the question Absolutely, so the question is what's our process for taking a workload from my from Heroku for example on to classic six There are signs of something You know we can see as you know as you know we would detect for example my sequel if you also detect something's been running on Heroku our signs to tell and if we detect those things we actually are more proactive We show you something in heroku per pole that says it seems that you've been running this. Wow. That was efficient You've been running this on heroku here are the steps that you can take if we cannot find those tellful signs Then still there's documentation around how to move each one of those things around code database and traffic All right. Well, thank you very much for your time. I hope you enjoyed it You can check us out at classic six comm we also have a booth at the exhibition You can come and you know ask more questions or you know see the project in the demo for the things as well. Thank you


Let me say how very pleased I am to join you this afternoon at this commissioning of the Westlink iconic villa. And there are those who may ask why this particular private housing development is one that I needed to be here personally to commission. The reason is more than one and I'm sure you've heard already some of it. But the first is that this is a project that demonstrates what we as an administration, the federal government has preached since 2015. It's a project that tells two important stories. One is the story of success of local entrepreneurship and forward-looking development finance. Local entrepreneurship, of course, these are diaspora friends and forward-looking development finance in the body of the Odua Investments Limited. The first part of that story is that the Nigerian diaspora has a crucial role to play in investments in our economy and that the investment prospects in our economy are great. And though it may require some sacrifice, the rewards are well worth it. As you've heard from the CEO of Chapter 4 investments, in March 2015, just after the APC won the general elections, I was in the United Kingdom meeting with Nigerians in diaspora. Several business persons were present at that meeting at the Marriott Hotel in Kilburn in London. Mr. Shams Obumwe says that I may have forgotten that meeting, but that he definitely remembered. But I will show you a video where it tells you that I not only remember the meeting, I also recorded the meeting. So now this is that meeting. If you look at this video, this is that meeting and that, I think, is you right there. As you can see, this is a much younger looking me as well. And this happened in Kilburn on that occasion where we met at the Marriott Hotel. And there after the meeting, we had a discussion which is what you're looking at up there. And after that discussion is what I believe led to your decision to come to Nigeria to invest. On that occasion, I think we encouraged ourselves that it was important for them to bring back their skills and capital back home and to partner with the incoming APC government to develop Nigeria. After that event, I had a discussion again with the event, of course, was the one that I referred to. We had that discussion at the Marriott Hotel. And he expressed a very keen interest, Mr. Shams Obumwe, expressed a very keen interest in coming back home to invest in real estate. I believe it was some time after that, going by the story, that he and others in the Chapter 4 investment came together to develop this project. They kept me informed through a mutual contact, another Nigerian from the diaspora, Mr. Muiwa Uduni, who, by the way, returned to Nigeria shortly after this meeting and is now working as a director at the Bureau of Public Enterprises back there in Abuja. The road to raising capital, as you've heard, was rough, but they kept starting through the year. But the breakthrough appears to have been by their coming across Odua Investments Limited. And Odua Investments started itself, as you know, we've all heard the story of Odua Investments Limited. It started as a development company in the golden era of development in the Western region. At that time, of course, as you know, Chief of Bafemiaholoa was Premier of the Western region. Odua Investments Limited at the time built several industries, real estate investments, et cetera. But in later years, it's fortunes deteriorated, especially during military rule, until in recent years, when the governors of the Southwest takes inheritors of the illustrious legacy of the Western region then, under the premiership of Bafemiaholoa, decided to turn over the management of the company to seasoned and tested professionals and business persons who were given a completely free hand to run their fears of the company. So it is the new and greatly improved Odua Investments, which we see today and which brought the vision of our diaspora investors, Chapter 4, Estates Migrant Limited, UK, to build what we have here today. Also, the story of the West Link iconic housing units here in Alakia, by the way, is a story of the triumph of ideas and a vision of private enterprise development and capital working seamlessly together. There's a lot that we can do in housing in Nigeria. There's a significant deficit somewhere between 17 and 20 million houses. This is the deficit that we are told we have, which should ordinarily be a major economic opportunity, not just for house ownership, but also for the many jobs and manufacturing opportunities that housing brings. The missing link is not the market. There is a market for housing, as we've heard from the Secretary of Government of Ogun State. 1500 housing units snapped up very quickly and this is not necessarily low-cost housing. The missing link is also not the number of ideas for private development. There are so many ideas for private development. The missing link is housing finance. It is affordable mortgages. That is the missing link. Social housing and affordable mortgages is what we need today. Every developed economy has had to develop usually with government support and mortgage schemes that are affordable. So what needs to be done is that government must provide the capital to the risk single-digit mortgage lending by the banks. And when I say government, not all the federal government, but also state governments. The United States, with all its free market protestations, in 1968 established the Federal National Mortgage Association, which is commonly called the Fannie Mae. This is a U.S. government-sponsored enterprise. And since 1968, it has become a publicly-traded company. So the U.S. government had to intervene to derisk lending, housing finance, even in the U.S., despite all the talk about private capital, about free markets, etc. There's also Freddie Mac in the U.S., the federal home loan mortgage corporation. It's also a publicly-traded government enterprise, created in 1970 to expand the secondary market for mortgages in the U.S. So the role of government is crucial. There is no way we can escape government putting in significant funding and putting it in smartly in order to derisk the kind of capital that then becomes lendable to those who are doing housing projects such as this today. We started that process in our Housing for Jobs program under the Economic Sustainability Plan. The Economic Sustainability Plan was a federal government response to the COVID-19 pandemic and the aftermath. The Family Homes Fund was the alloy head for the first phase. So the CPN, in that arrangement, was to provide the Intervention Fund for Mortgage Finance through the Family Homes Fund. Part of that fund was provided, but not what was adequate, to get the fund to the state where we thought it would get to. And we're still working on that program. But there must also be affordable, private sector funding for housing. So we must commend Odua for providing the support for the project. The future is bright. It will stick to the vision of providing the risking funds supported with development finance funding, such as from Odua. Private capital is also going to be very important. But I think that it's very crucial that governments and private sector see this as an important investment opportunity and also as an important development opportunity. Mass housing is crucial. What cannot do without mass housing. We started the Mass Housing Project also under the Family Homes Fund. Some of the governments that have done exceptionally well in mass housing, and I'm not talking about affordable housing but mass housing. The Bono State government, for example, partnered with us on the mass housing program through the Family Homes Fund. And today they've done 10,000 units of mass housing, 10,000 units of mass housing. So it is entirely possible for us to do mass housing on the scale that we need to do it in order to make up the deficit. It's also entirely possible for us to do affordable housing, housing that's a bit more middle class, and make it affordable for the middle classes, for people in the civil cells, for teachers, and for private business people through the de-risking facilities that the federal government and state governments can provide. So today we congratulate the management and staff of Chapter 4 on your investment limited on this landmark investment. But not just for this landmark investment, but for the hope that this project gives us for the future of housing in Nigeria. I commend you all, and I thank you very, very much for seeing to it that not only did we talk about this, but that we actually made it happen. Thank you very much. God bless you.


Welcome to today's masterclass. Our topic is wildfires as a people problem and wildfires as a utility finance problem. My name is Katie Kaplan-Sternakka. I'm a program manager at the Precourt Institute for Energy. And I'm just gonna quickly introduce our speakers and then we'll take a quick gallery screenshot and then get started. So today we have with us Bruce Kane. He's a professor of political science and director of the Bill Lane Center for the American West. He has a BA from Bowdoin College, a B-Phil from Oxford as a Rhodes Scholar and a PhD from Harvard. And his areas of expertise include political regulation, applied democratic theory, representation and state politics. And then we also have Michael Wara with us. He's a lawyer focused on climate and energy policy. And he is the director of the Climate and Energy Policy Program and a senior research scholar at the Stanford Woods Institute for the Environment. And his legal and policy scholarship focuses on carbon pricing, energy innovation and regulated industries. And he has a JD from Stanford and a PhD in Ocean Sciences from UC Santa Cruz. So I will now hand it over to Bruce to share his screen. All right. So let's get started. I'm happy to be here to talk about this topic and it couldn't be a more timely topic as you are here at the tail end of our most recent set of fires. And I've titled this, it's not just a nature problem but also a people problem. So as we speak, we're finishing up. This is from today's Cal Fire site. The two pink areas here and here are the two fires that are closest to where we are. We're up here in Palo Alto and the two fires are to both the east and west of us. There are actually fires to the north of us as well. And according to Michael, that's what's generating a lot of the smoke. That's what he told us yesterday. And it could be because a lot of depends on which way the wind is blowing. But at any rate, these fires are almost 74% contained, which is why we still are not past the whole event. But it's pretty much getting close to being totally contained. And if you go to the site, you can get a list of all the fires that major fires in the state that are still either still burning or the ones that have been put out completely. So by the way, this is just the beginning of the season. And for reasons I'll explain in a minute, the fires that we'll get later are the ones that in many ways are scarier because of the nature of the wind. So this is a social scientist explaining science. So you want to take it with a grain of salt. I co-teach with a number of hydrologists. And over the years, I've learned about the very distinctiveness of the climate in the US West. So I want to talk a little bit about that. I want to talk about eridity, seasonality, eridity being the lack of precipitation and the high rates of a Vapo transpiration, seasonality being the fact that we have periods, long periods, months that go by without any precipitation and months where we do get precipitation, which causes the vegetation to grow and then dry out and become the source of fires. We have a lot of variability across the year. Some years are very wet years. Some years are not so wet. We have the Diablo Santa Ana winds. And these produce three interrelated phenomena, the drought, which you'll see in a second, the wildfires and the mudslides. The mudslides will come after the wildfires because a denuded, if you like, denuded from vegetation, hillside can easily develop into mudslides when we have heavy rains in the winter. And all of this is going to be exacerbated in ways by climate change. And that's not a topic I'm going to be able to get into, but we can certainly bring it up in Q&A if you'd like to. So you're in the West. And the West is defined in many different ways, but the most frequent way it's defined and the way we do it for the purposes of the Bill Lane Center, is to take the demarcation at the 100th Meridian, which was basically the line that was defined by an expedition in the mid to late 19th century, which determined that the line of aridity, that is where the precipitation patterns changed dramatically started somewhere along this line, the 100th Meridian that goes from North Dakota down into Texas. So parts of Oklahoma, Texas, Nebraska, and North and South Dakota are all in the West. Parts are on the other side of the aridity line. But aside from the aridity and the evapotranspiration, that is the fact that the heat in, particularly in the Southwest causes the vegetation to dry out quite rapidly and thoroughly. We have the fact that we have distinctive seasonality. So here I've taken two shots of the precipitation and you can see that in Sacramento, which is way here in California, you can see the precipitation is pretty much very minor to trivial from about May all the way to about October, okay? There's just not much rain. And those of you who grew up in the East or in other countries will recognize that actually in the summer, there's often a lot of rain in the East. And here I took the other side, which is Oklahoma City, which is just on the other side of the 100th Meridian, although by the way, now people believe that the line of precipitation really starts the 98th Meridian. So we'll see how that pattern changes in Oklahoma City. Which you can see the precipitation in the summer is still pretty heavy, quite heavy in even in Oklahoma, let alone on the East Coast. So it's the seasonality that is extremely distinctive of the Western climate. We also have a lot of year-to-year variation in precipitation. So you can see the darker colors represent where the ratio of the, if you like the variants or standard deviation of precipitation is highest. And you can see that that's particularly true as you get into the Southwest. Those are areas down there, San Diego, Imperial County, which rely very heavily on water from Colorado River. Now, what this produces, of course, is a pattern of droughts. And this is one drought index that you can find. And the severity of the drought ranges from sort of just dry periods, that's the yellow to the exceptional drought. And you can see that just recently in 2011, 12, we started the drought. And then for a period between 2012 and 2017, substantial parts of the state. So the fraction of the state is on the vertical axis and on the horizontal is the year. You can see that for a period of about four or five years, substantial parts of the state were at least in serious drought. And particularly when you got to 2015 and 16, we had really, really, really serious drought. And that produced the potential for the wildfires that we got in 2017 and 18, okay? There's other things that have created the fuel, namely the suppression policies that the Forest Service and the era when I was growing up, Smoky the Bear era, if you like, that has also contributed, but we'll get into that if somebody wants to know. The last and distinctive feature that those of you from the East Coast would not be familiar with is this one about the Diablo winds as we call them up here in Northern California or the Santa Ana winds. And they're basically winds that come in off the desert and then through the compression of going up and down through the sort of hills and valleys and mountains, coastal ranges of California, they pick up speed and they pick up warming and a drying. And that's what we really had trouble with in 2017-18, not so much in this most recent, in fact, not at all, we didn't have Diablo winds, the winds for the most part were more North-South. And I don't think we really had focused as much on the dry lightning problem. We certainly have always had dry lightning, but it was more out into the remote areas. What was exceptional about this storm was the number of the dry lightning hits and also the fact that they occurred in these urban areas where we had not had many fires before. And so the dry lightning is particularly problematic. If you follow this chart, this shows you the number of strikes, just an incredible 9,000 strikes practically. And you can see the yellow ones are the continuous lightning strikes, something I had never really thought about, but I looked it up and apparently there's a lot of different types of lightning, but the continuous current ones are the ones that really start the fires because they had the longer contact with the ground, between the ground and the clouds and the electrical current. And that tends to generate more heat and cause more of the fires. So it was not just the number of these strikes, but it was the nature of the strikes as well. And as I said, the accumulation of the fuel that had happened both over the years and as a result of recent variation in between precipitation and dry years. So, but I'm a social scientist, I wanna focus more on the people problem because while it's true that this is getting worse because of climate change, the reality is that wildfires have a long, deep history with humans in the American West. This is not a new problem. It was when I moved, took a job at Caltech in 1976, we dealt with wildfires all the time down in California. I remember the hills burning quite frequently. So I wanna say that this is a people problem, not just a nature problem. And a lot of it has to do with the increasing population in areas that we know to be prone to wildfires which are called wooey areas, wildland urban interface areas. And basically the presence of people causes wildfires, okay? I'll do these two points fairly quickly, but then I wanna spend a lot more time on what the hell's going on with people's decision-making and how does that make it worse? So you can study this map, I'm gonna leave the slides have been posted and you can take a look at it, but the dark colors indicate where people are living very close to areas that or types of vegetation that are likely to catch fire. And so there are basically two different types of interfacing. One is to be close to a national forest or an area that potentially could burn. And so when I was living in Southern California, Altadena is a good example of that. That's where JPL is. Altadena is literally a but the forest that goes up to the mountains. And people that live there have to experience the terror of the fires on the hills. But the kinds of fires that the other kind of fires that we see a lot in Northern California are communities like, if you wanna just see it, just go up into the Santa Cruz Hills here, just go up and see Portola Valley, places like that, Woodside. And you can see communities that are very much embedded in the vegetation.  And it just, there happens to be a retirement home up in that area behind us.  They've been told that if indeed a fire did start up in those hills, very few of the residents of that retirement home, many of whom are former Stanford professors would basically not be able to escape and would be incinerated, much as what happened in Paradise. Now, that's not the only area. I mean, you can go around the West and you can see the same kind of interspersing. Again, the dark red of the areas which is most likely to be affected by these fires because they're intermingling between the vegetation and the development, the urban and suburban development. And these areas have continued to grow in population. This is just an example of all these Western states are still growing. In California, we have a housing shortage. The governor wants to build more housing. It's not easy for people your age and maybe 10 to 15 years older to buy a home. It's extremely expensive. So there's a lot of pressure to build no homes. But where are you going to build those new homes? Well, people want to build them in areas that are pretty and beautiful, which means they want to be closer to nature, which means they want to be closer to danger. And so the coincidence of what people want in terms of demand of housing, combined with the fiscal incentives that urban areas have to build more housing because that generates more revenue that pays for services, means that people are making riskier and riskier decisions every year, right? So you can study these statistics a little bit more closely on your own, but basically a lot, 43% of the new homes are built in these wooey areas, okay? So we're continuing to make the same mistake. Just as the wooey areas continue to expand because of the drying and the climate change, the reality is that we're continuing to build in these areas, okay? And when we do the polling on this, we find that this is what people want. People say, hey, if somebody wants to go live in a risky area, that's on them. It's not on us. As Michael will point out, actually it is on us, all of us, because of the fiscal implications, but I will leave him to explain that to you, okay? So let me talk about their flawed decisions and then finish, okay? So people make a lot of very flawed decision-making. First of all, and foremost, and this is the one that I think we really need to attack, people are rebuilding in areas that have previously had fires. And I think the idea is that lightning can't strike twice in the same place or something, and that's simply not true. So this is up in the Napa area and all these colored areas, and you can see the various decades that are in the box there. These are all fires that occurred in previous decades and yet now we've continued to rebuild in those areas, okay?  Now, why do we do it? Well, we do it because a lot of people underestimate the damage. They think to themselves, well, what are the odds that that fire is going to affect me? Or maybe they think, oh, it doesn't matter. I'll get out of there and I will pay my insurance and I'll just rebuild. Or they think, well, with new materials and vegetation management, I'll decrease the risk of fire damage to myself and that's true to some extent. But all of it still underestimates both the social and individual dangers that people face. So one of the reasons people decide to do it is they think, well, I can, with new building materials, I can harden my home with various vegetation management strategies. I can create a defensible space. And if we do more prescriptive burns, we can take care of the outlying areas. And all of those things are true, but it doesn't eliminate the risk that you will still lose your property, that you will still put your life at risk. So you can make all these wonderful choices about what kind of shingles to have. You can replace the wood with fiber cement board. You can treat the windows differently. You can make sure that the eaves and the deck are taken care of in various ways to be more resistant to flames. But you got to remember that there are all kinds of little things that you can do that will, particularly if we get one of these Santa Ana winds or Diablo winds that bring in lots of sparks that are traveling at 50 or 60 miles an hour and beating up against your home, you leave your garage door open or you leave gaps in various places. This is, I believe, why there was a warehouse, as I recall, in Santa Rosa that burned, largely because somebody left the door open, okay? So number one, when you're fleeing, you may not remember to do everything you're supposed to do. But the reality is that no matter what you do, there are some fires that are gonna burn so hot, no matter what, you're going to be incinerated. So we try, and again, I'm not saying that it isn't worth doing these things because this is our second best strategy because no matter what we do, people are gonna continue to wanna live in these woowy areas. So a lot of us are spending time thinking about, well, what do we do not only to figure out what in terms of evidence base is the best way to do fuel management and vegetation management, but also how do you enforce any regulations? How do you make sure that people are doing what they're supposed to be doing? The good example, which I'm gonna let Michael explain if he wants to, is that Mill Valley put a fairly modest proposal forward on hardscaping of property. And people, and he has many more stories about the insanity of it all, but this is one of the ads that Rebecca Miller, one of my graduate students focuses on in her lecture on this topic. And basically people were saying, well, which place would you rather live? And you'd rather live in the place that has that vegetation all around. It's much more appealing, much more inviting, but the safer one is the one that doesn't have all that. And so people make these decisions about their lives and their property on, I don't wanna call them irrational, but they refuse to see the importance of safety and tend to make choices which should we say are problematic. Not to mention the fact that really when you put in new regulations, most of it applies to new construction. So if you go into Santa Rosa as we did in our class and you look at the houses that are being rebuilt, you realize that there's gonna be a scattering back in the community that burned between some houses that survived no matter what, they're gonna have the same old vegetation, the same old construction materials next to homes that will be built up to code. So not to mention the fact, none of them will have a hundred feet of defensible space between them. So one thing that I wanna say is that we do believe that quite possibly the smoke may be a wake up call for us. And the parallels are to secondhand smoke with respect to tobacco. And here is a poll that we did. And you can see the level of concern with wildfires after 2017-18 was very high in California. And that basically people had read about fires, over 50% had read about fires throughout the state. And close over 50% had experienced some degree of smoke or new themselves or new family members. That's the personal one who had actually been involved in the fire. So the reason, and you can see by the way that the smoke travels throughout the West at one of our sophomore colleges we were out in Utah around the time that the fire smoke from California was polluting the air in Utah. So it has the political advantage that brings people in to the discussion who might not otherwise be afraid of the fire itself. The other way that risk is underestimated is with respect to insurance. I won't spend a lot of time on it, but people systematically in Santa Rosa did not buy the right level of insurance, partly because of the surge prices, but partly just they just didn't have the right kind of coverage. And that creates a real problem for the community trying to rebuild. So let me finish lastly with partisanship which is a big topic in political science. You'd like to think the Democrats and Republicans see the reality, but the partisanship in this country is so great that in a variety of these extreme weather circumstances, whether we're talking about drought, heat, and in this case, wildfires, Democrats are much more likely to acknowledge the same extreme weather that they experience with the Republicans, okay? So there is a partisan screen that seems to be working even on the experience of extreme weather. And one of the things we've done is we've tried to test, well, what if we were actually taking to account the real experiences of people? Does that close the gap in terms of these subjective biases? And this is an example where we took in the actual measurement of how close you were to the smoke or how many fires you actually saw over a period of time. You can see the error bars as you get out to the extreme are very, very wide. So you don't really only wanna look at the first part of the graph, but you can see that while these biases exist, there is no question that experience even erodes the unwillingness of Republicans to recognize that there are real serious extreme weather and climate change problems. It does so, it takes more experience to change their views, but they are witnessing it. So let me just finish by saying we have as much a people problem as we have a nature problem. That is to say, when we deal with this problem, we have to, if we're gonna live resiliently in the West, we have to be aware of the wildfire risk and we have to adapt ourselves so that we can live there in a way which is safer and does not impose the serious social costs that these recent wildfires have. We can't sustain that for very long. Okay, I'll hand it over to Michael. Bruce, we have two questions, one from Pratik and one from Benjamin. Do we wanna do questions now or do you wanna wait until then? I think to be fair to Michael, we should just go ahead with Michael and just save those questions. I was just saying, actually Bruce, it's fine with me if we take these questions now because they're pretty focused on what you just spoke about and it'd be better to just talk about it. It's okay. I don't mind. Okay, it's your call, Mike. Okay, Pratik, you wanna go first? Sure. Yeah, thanks for the presentation. And I was just wondering, I heard about this wildfire buffer zone that Paradise is thinking of building and I'm sure you've maybe heard about it as well. I'm just wondering what you thought about the effectiveness of that solution. I think they're calling it like a green belt to essentially surround the whole town with defensible space. So I didn't know if you thought that was an effective way to go about adaptation or if that was just people still insisting on staying put. Well, I'll give an answer and Michael, feel free to jump in as well. Mike was on the governor's commission and is quite knowledgeable about all these things. I would say it depends upon what the word effective means. If you mean by effective, which I think you do, it lowers the risk, then yes. Just as all the defensible space measures and the home hardening measures lower the risk, yes. Does it mean it eliminates the risk that you will burn? The answer is no, particularly when you get these fires that have the winds behind them and they're blowing a torrent, a virtual river 60 miles an hour of sparks that are hitting these homes. And when we were up in Santa Rosa and talking to the Stanford people that live up there, they were questioning what the insurance companies and the regulatory authorities were telling them to do because they were saying, well, hey, if we have indoor sprinklers and we maintain the hundred feet or in this case a zone, what happens if you get these fires that are coming with sparks at 60 miles an hour and are beating onto our property or is the indoor sprinkler system really gonna work? And the answer is, first of all, I doubt that anybody, there are people doing the science on this. I don't know whether the insurance companies are listening to those people or they're doing industry science, but I can tell you for sure that everything I've read is that there's no guarantee when you get certain conditions and that's the kind of heat and wildfire storms that we have seen recently that any of these measures will give you 100% guarantee. There's a great example of this from another part of the country after Hurricane Andrew, Florida instituted new building codes for roofing to keep the roofs on the houses. A big source of damage in hurricanes is that the wind will blow the top off the house and then the water gets in. And so these so-called super roofs protect structures in hurricanes from up to like a cat three hurricane. But if it's a cat four or a cat five, all bets are off. And I think that's a great way to think about, we don't have categories of wildfire, but things like what Paradise is proposing will protect paradise from a cat one or a cat two or a cat three wildfire, but not what happened in 2018. And that's good point. And we should have a category system for wildfires. All right, what's the second question? Benjamin, you wanna ask your question? Hi, yes. My name is Ben. I'm an MBA one at the GSB. I was curious, you mentioned, and we'll sort of discuss at length the issue of developing residences or just developing generally in WI's. And I was just curious, what, I know this is probably a question of folks have been trying to solve for a long time and there might not be a simple answer, but how do you solve for a housing crisis? How do you solve for population go throughout the West while doing so safely with regard to wildfires? Like, are there any existing urban areas that are wildfire safe or maybe you could push past the NIMBYism and try to encourage densification? Well, it's a constant battle to push against the NIMBYism. And in fact, we're gonna do an environmental governance class in the spring on precisely that. And I do think housing and land use are absolutely critical issues. And the state doesn't really have control over that because of the nature of local government having sovereignty over a lot of these issues and having fiscal incentives as a result of things that have to do with property tax and the need to sort of fiscalize land use in order to generate revenues, that accounts for why developers are extremely powerful in the cities. So, we have made progress in getting communities to be more fire aware and to take certain steps, but it is a struggle because it's not unlike the COVID thing where there's what you can do to maximize your chances of not getting sick and then there's the economic consequences of those measures. And there's a similar kind of trade-off with respect to the cities as they think about housing because if they can get affluent people to buy beautiful homes in the middle of Woodside, by the way, I will point out that Stanford is planning to put a housing project in Woodside and the people in Woodside are concerned about that for a variety of reasons. So, we're continuing to do it too. People want to live in these beautiful areas surrounded by vegetation, but okay. I point to Mill Valley as an example of a community didn't want to do it. Michael, you want to tell them a couple horror stories? Yeah, well, I just say, I think you're pointing at an issue that gets to deep politics in California and it also gets to the economics of housing development because of that politics. So, it costs a lot to build a structure in California. A lot of that is regulatory costs associated with all kinds of different things, most of which are well-intentioned. But what that means is that if you can amortize your regulatory costs over larger numbers of structures of units, it's a much more profitable enterprise. And that's also part of what drives the big developers out of the city because they can get big plots of land, build 2,000 houses instead of building one building and they can amortize their regulatory costs and risk over a larger number of units. And so there's really like, it's a very complicated problem and one that California is struggling to resolve effectively. And the status quo is we build in the wooey or more of our units get built in the wooey. So, you know, look outside Palo Alto. Mill Valley is a great example. Palo Alto is notorious for limiting development. There are all kinds of communities in the inner area that really need to change in order to solve this problem of avoiding risk, avoiding putting new homes in Hotshawak. Yeah, what Michael means is Palo Alto resists having high density homes and Mill Valley. And so a lot of communities just allow the expansion on them in the zones, right? That's what you meant, right? Exactly, exactly. Yeah, thank you both. Shall we shift gears and talk about utilities? Yep. Are there more questions about bristles? Okay. So, you know, when I started out working on this issue of wildfire and electric utilities, I had a particular idea of what a utility was. But I think, you know, it's sort of like, I was trying to think of examples of things that just sort of work and you don't really know what they are until they break. And, you know, one example that comes to mind for me is like a toilet, right? You just push the button, the bad stuff goes away. You know that like gravity is involved, maybe some water pressure, but you don't have to like know anything about it until it breaks. And then you lift up the lid and there's this like Rube Goldberg contraption in the tank that somehow is doing the things and there's a lot more there. There's a lot of complexity. There are many things in life we don't understand that are essential kind of until they don't work anymore. And I think the crisis that's occurred in California over the last several years, particularly with Pacific Gas and Electric, but definitely not limited to PG&E, the utility in Northern California is a great way to learn about like, what is it that makes utilities tick? It's like, what is a utility?  Whether it's an electric utility or a water utility or a gas utility, what's it doing that makes it special? It's the reason we call it a utility. Does anyone have a thought about that? Feel free to unmute and just jump in. What's a company that's providing goods that society has decided that people need over a service territory, because if you had two companies doing the same thing, it's not economically viable for competition, so. Okay, so I just wanna get your name. Fletcher, is that correct? Fletcher sort of has two answers. One is it's something people need, right? It's an essential good. And I think it's fair to say that things like electricity and water and sewer and gas, are essential needs. Maybe gas is, we're trying to shuffle that to the shadows in California, but it's still something that people need to live in a modern way. He also makes the point that it's a monopoly. We don't want two companies providing a lot of these goods. And that's another important clue, right? A lot of the utilities that we think about are network industries where we decided that we don't wanna build two networks. And so, unlike for many other types of companies, we actually give them an exception, a pass on being a monopoly, right? You're not allowed to be a monopoly in most areas of business. But certainly like for utilities, that's kind of, that's an important defining element of what a utility is. They get an exemption from antitrust law in the United States. So they're allowed to, in fact, they're supposed to monopolize. And if anyone tries to compete with them, they'll go straight to their regulator and start trying to shut that person down. Does anyone have other thoughts about what a utility is? I wanna share them. What makes a utility a utility? Looks like we have a response in the chat here. Oh, great. Okay, sure. A service that allows us to do things. It's useful. Yeah, I think that's another important way to think about this. A utility enables all these other activities, right? It's so fundamental to kind of how we live our lives, that it enables all the other activities that we wanna pursue, whether they're economic or purely for pleasure. Netflix doesn't work without electricity and it doesn't work without some sort of broadband internet service. That's kind of another sort of utility. So a utility is a business that provides us essential good and it's generally a monopoly, although not always, there can be some competition. I'm gonna provide another, I'm gonna inject another element into this. We wanna have these businesses providing us these essential goods at minimum cost, right? We wanna try to see, and usually how do we get to lower costs? What's the way to lower costs in an industry? Competition. Yeah, competition, exactly. You get a bunch of companies fighting for market share and they reduce costs by competing with each other, by lowering prices to attract customers. But since we've decided that we don't wanna have multiple sets of poles and wires and pipes and all that across the landscape, we have to find another way. And so this gets to utilities who are gonna have to operate differently. And I would just say, I'm gonna add one final dimension to this. And I think this is fundamental, but this is the one that's not obvious. This is the one that kind of requires lifting up the hood to look underneath. Another element of what a utility does is create a mechanism for very cheap finance for long-lived infrastructure, right? So the thing about the poles and wires and the pipes and all of that is that it lasts a long time, right? It's stuff that is supposed to last 30 years, maybe 40. Sometimes if you're talking about gas infrastructure, natural gas infrastructure, 60 years. And so it is financed over very long time scales, right? So the loans and the investments that are being made are paid off essentially by customers over long time frames. And in order to do that inexpensively, right? You have to have a mechanism that creates a lot of certainty for the bank, right? For the person that's gonna loan the money. They need to believe that this company is gonna be around in 40 years, that it won't default essentially on its mortgage, right? On its bonds for those of you who are more financially oriented. But we need to have a lot of confidence that bankruptcy won't occur, that the customers won't go away, right? The customers won't find a better way to do whatever the company does. And a lot of that desire for certainty is actually what drove and certainty in order to reduce finance and costs in order to make costs lower for customers is what drove the creation of regulated utility industries in the United States. The utilities themselves actually sought regulation. They went to the States and said, pass laws to regulate us so that they could lower, they could increase their credit worthiness, the confidence of banks to loan the money. And so lower their financing costs, the interest rate they pay on their loans, which the interest rate matters a lot. You guys probably are not in a position to buy homes right now. But if you can go from a 6% rate to a 3% rate, that may not sound like a lot, but over 30 years, it makes a tremendous difference in terms of the overall cost of building something. So all of these answers are right. None of them are wrong, right? Utilities provide essential services. They create infrastructure that we need without competition, right? And that's mostly intended. And they also are structured in a way to reduce risk, right? To be sure that the monopolization, the kind of lock-in that's created, the regulatory oversight that occurs lowers risk or it's supposed to. So I wanna ask a related question is, what's a utilities role in the energy transition? So what I mean by the energy transition is building in California, what it has meant over the last decade or so is a radical transformation of generation, right? So we used to depend on coal and natural gas and hydro and now we depend on a lot less natural gas, still the hydro, but a lot of wind and solar. How have we, what is the utilities role in building that wind and solar? Does anybody know? Does anybody, it's okay if you don't. This is more subtle, but it relates to that credit-worthiness as well. Create a market, right? You have a buyer for a new form of energy. Yeah, they're the buyer, exactly. And what renewable portfolio standards, like the one that California has, which is really where a lot of the reductions and emissions for California have come from, what they do is force the utilities to buy power from these new clean energy providers. Picture a big 700 megawatt solar, utility-scale solar power plant, say in the Central Valley of California or in the Mojave Desert. The utilities role is as a buyer. The form that that purchase takes is like a 20-year contract called a power purchase agreement, right? Now, why sign, the value of that contract is that it turns a solar farm into a solar power plant into like a cash-generating machine and a cash-generating machine with very low risk. Once the plant is built, then the person that owns it knows, they have a guarantee that they can sell power to a utility for 20 years. And long enough to pay off what? What is the solar power provider have to do in order to construct a solar farm? Do you think they have money in the bank or are they doing something else to actually purchase all of those panels and pay all of the workers to do the grading of the land and all of that and install the panels and interconnect it to the grid? You've got, Alessandra said loans in the... Loads, yeah, that's right. You know, they need to convince a bank to loan them the money. And you know what's a great way to convince a bank to loan you money? If you show up at the bank with a signed power purchase agreement for 20 years for all of the output from your solar power plant with someone who is not gonna go bankrupt that the bank can say, oh, you've contracted with a company that we think under no circumstances will default on its obligations which is what bankruptcy is. Bankruptcy means you can't fulfill all of your financial obligations including contracts to purchase things. And so in California and actually really around the country utilities are incredibly important because they're the low risk buyer that allows for the finance of these power plants. You can't build large power plants like this. You can't get loans anyway to build large power plants without an offtake agreement like this under most circumstances because they're incredibly capital intensive and the economics of them don't make sense without the ability to borrow at low costs. So why is this all important? Well, the strategy for California decarbonization is basically clean up the electricity sector which means sign a lot of power purchase agreements between utilities and new renewable generators, electrify buildings and electrify transport. That is the roadmap to deep decarbonization in California. So when wildfires in 2017 and 2018 were caused by utilities and the fires did enough damage to put utilities into risk of bankruptcy, it basically broke this chain because the first link in the chain is an agreement between a company that banks can be sure will purchase the good being produced electric energy for a very long period of time at an agreed upon cost. And anyone else in that chain that might help to create that reality of a giant solar power plant, say in the Westlands Water District in the Central Valley. And so many of us got interested in this problem. We had never really lifted up the hood on a utility. Never really thought about the relationship between the utility balance sheet and the ability to build new solar under the requirements of the RPS until all of a sudden, the balance sheets of the utilities had to take on this new liability, the obligation to pay for the damages caused by wildfire. So, and I should just say, I just talked about this. The utilities in California have a very special role and I think it's worth noting also that the California targets for decarbonization put utilities like PG&E kind of in a leadership role in the United States. Just to cite one number that's rather striking, 25% of the rooftop solar in the country is in PG&E service territory. 50% of the rooftop solar is in California. Half of that is PG&E. So, California utilities are way out ahead of pretty much anyone else in the country and they have traditionally worked hand in hand with the state to try to achieve California's targets which essentially pushed the ball forward on climate policy at the national level. So, what happened? What went wrong? Bruce mentioned and showed some pictures of Santa Rosa and of the Napa fire that occurred in 2017. But in October 2017, we had historically unusual conditions that led to utility power lines starting enormous numbers of fires in Napa and Sonoma counties to the north of Stanford. And this was combined with a legal regime in California that's longstanding that basically says if utility power lines start fires, the utility has to pay for the damages. There's no question of negligence. It's a strict liability regime. It's called inverse condemnation. But the basic idea is that utilities are able to socialize the costs of accidents that occur due to utility infrastructure over their entire rate base. And so, it's a sort of efficient way to get those damages paid for. The problem that occurred in 2017 was that there was real uncertainty about whether the utilities could actually charge customers for the damages costs. They were so large, you know, the 2017 fires caused about $10 billion in insured losses. And that's such a big number in terms of how much money, how much the revenues of PG&E in particular that it seemed implausible that the utility commission would allow those charges to be passed through in rates. Then, in just a year later, the campfire was caused by age, by ancient, I would say, transmission infrastructure up in Butte County that, and Bruce mentioned the consequences, you know, and I'm sure you're all familiar with what happened in Paradise. Shortly after that fire, PG&E was essentially cut off from all access to credit, not just these long-term loans, but even short-term lending. And shortly thereafter, they had to declare bankruptcy because utilities are really dependent, even just for doing the kind of routine maintenance, like if a pole falls over or there are woodpeckers in it and they need to replace it, right? That pole is the capital expense of replacing that pole is financed with bonds and issuance of new shares, new shares in utility company. And so once bond investors said, no more, we're not buying any more PG&E debt because we don't know if we're gonna wake up in fall of 2019 and PG&E will have burned down more of California, right? And once equity investors, shareholders decided that they didn't wanna own PG&E and that happened very rapidly. The share price in October of 2017, just prior to the fires was $70. By December of 2018, the shares were trading at less than $10. And basically all of the kind of normal investors in utility shares had gotten out. But once that happens, the utility basically has cash on hand and they have to try to maintain this very expensive and distributed system of poles and wires with the cash that's coming in and that's just not possible. They need to be able to issue long-term debt and shares. So in January, the utility declared bankruptcy. And the key question was how do we reassure these folks that are the kind of crucial source of finance to utilities that provide the source of demand for utility bonds and utility shares that they should invest in California. When California utilities are causing tens of billions of dollars of damage that they are strictly liable for. It's seemingly every fall or at least two falls in a row. And the solution was to create an insurance fund. Well, the solution was multifaceted. But the financial solution was to create an insurance fund that provides much greater certainty for the utilities in terms of their ability to recover costs on fire. So, and the fund was created as part of this legislation 1054, AB 1054 that did a whole bunch of other things that basically tried to reduce are aimed at reducing the risks that the insurance will actually be needed. So there's sort of a combination, reduce the physical risk and then create a mechanism so that the utility, it becomes low risk again and can be that source of certainty at the heart of this very long-lived infrastructure that we want to build as inexpensively as possible. And that would really, without the creation of the wildfire fund, not only would PG&E not have been able to exit bankruptcy but Edison, the Southern California, two major Southern California utilities, Southern California Edison and San Diego Gas and Electric would now not have investment rate credit ratings. So it was a very important kind of risk reduction measure. So the fundamental steps were better management of the physical risk, right? Reducing the chance that the utilities catch fire but also better management of the financial risk so that there was greater certainty that the utility investors could have in investing in California essentially, taking a long-term bet on California. The challenge moving forward is going to be better management of the political risks because you may have heard and you'll probably experience this fall or you may experience this fall, the situation where the best, one of the best physical risk management strategies, right? The best way to reduce fire risks during dangerous time is actually just to turn off the power right now. And that creates enormous political blowback. And obviously the governor and the commission really do not want to see power turned off. And so there's this tension between minimizing physical risks, minimizing the chance that PG&E will burn down another community and the political risk, minimizing the chance that PG&E has an acceptable relationship with the state that regulates it. And both of those are strongly connected to the financial risk. So I think there are hopeful paths forward but it's a very narrow path and it's gonna be very interesting to see whether PG&E can actually navigate this as we move through this year and the next several years.


There's no way I'm dealing with this right now dude Just no way. This is what I'm doing. I dude. I I just like you never want to play madden again, dude All right, who who should we who should we see should we see Jamar good? We got Gibbs. We got Musgrave and we got Alignment, what do you think we should do boys? Give me comms We've seen enough we get to look at one development player.  All right, what is he? I don't even know if he's super star watching me star superstar Jameer Gibbs Playmaker short in energizer But I really think energizer. Yeah, I don't know why I'm feeling like a bot right now But like short in like he's gonna get open regardless dude. He's running back I'm really not like it's not like he I don't like I need help with him getting open He don't need help getting open. This is going three.  I slept good. I probably slept from two to like the second quarter again I probably slept like two to eight. I got good six today tomorrow. We have a Chuck E cheese birthday party So all you single guys Just enjoy it. Okay boys. Oh here go Earl with basketball takes run. Oh one trap Just no way I'm dealing with this right now, dude Just no way. This is what I'm doing. I dude. I I Just like never want to play madden again, dude I just never want to play this game again. There is no way Friday night. I watched basketball game I spent some time with the family as a 36 year old man I boot up and play this like it's just no way I do this Just no way 36 years old dude that place sucks, bro But when you're not paying attention and you're talking about Chuck E cheese and he runs gun monster I honestly I forget how I used to defend this. I really forget. There was definitely like a there was definitely like a way that I did it though Let me control my man. He's just like just no way we're playing cover one backed up like this Is it like a thing dog?  I feel like number 40 swag No, there was because I used to play Saiya a lot who ran gun monster and then I found out like the bag No, that's a Luke Musgrave. What team is he from? What team drafted musgrave in real life? He's a rookie.  They just like pancake me though. I don't really like this that much Dog, I'm lost right now.  Give me off this guy game right now.    I have never seen a place this work this well. I have never seen this actually work as well dog This is an OP man. I this isn't it. Whatever. I'm doing is simply not it dog He has a brother says nasty streaks And he's like trying to throw sort of nasty streets, bro This is the most oh my god. I can't get over it and he wasn't even close I'm glad gun monsters in the game for people like this just glad gun monsters in the game for people like this But he's got abilities, right? He's got give him that man.     Good play, dude.  This is sad I'm a moodest guy out here. He's gonna do it. That's one. No huddle it again. Come on I mean it's a competitive league. I mean you're not gonna have all you're not gonna have 32 goons Nichols, I don't know how do you not click on that guy?  I honestly hate people that wouldn't click on right there I don't know how kids just chuck this and it'd be wide open. I have no idea how the box just that should be wide open Number 24, I don't know who number 24 is but just bagged AJ Brown on that route the box That dude be wide open all the time wide open all the time against the bots If you run screen right here with eight seconds left, dude, I want you to like just know Good, we made your field go a little closer that you would have got anyway.   I am under the rest though like these do this thing got past Left bum. I can't even do left bumper dog.    It's why abilities are you guys see this guy?  It's why the game is and we're remotely competitive.    I'm looking at muskray again or what's his name? We got open that time and then that dude just shed in my life. Good job, man. I'm not having any fun Central Yeah, it was too fast Don't that's the bet like this is the best play. I like dog.   Yeah, let me switch ball hands Bum. I don't know something was glitchy there though Maybe that blitz like made the tackle go to the middle instead of block his man in a flat. I don't know I'm gonna try it again boys. I mean if I had to play gun monster a lot It would be I mean dude like look at the score like what are we doing here?  Oh, he's so bad right now. Oh my god Reverse the other way. I'm like gonna just lose my mind.  What's the go-to? That's a good question big play I really haven't been to Chucky cheese long enough to tell you like what game I play I was just like playing the Ninja Turtles game the arcade Ninja Turtles That was one of my first video game memories that first level Right boys running through the the apartment or whatever was the hotel wait to play them with my son tomorrow He loves driving the car and standing in the car That dude number 30 was just so slow. I didn't think he was that slow All in all man sit sit on the motorcycles kind of sauce. How do that guy? I this is one thing I hate about Madden dude I hate the I guys I don't do you guys understand how bad this is so at that time I slid and double teamed right and because I did that they decided okay We're gonna let the defensive add on right come free did I ask for that?  This is just I just like There wasn't enough room for that Just hate this I really do I block my running back and just assume that now I want my running back to block Max Crosby That's exactly what I want. Yeah, so you might as well run the screen every like I'm not even mad at him That's his best play is running the screen. I don't know why he's doing anything other than run the screen You should run the screen every play dude There is no other world where you get anything positive other than the screen play Like as much as we want to laugh at this guy about like how ass he is, but what are we looking at man? He cannot do anything other than run that screen. He can't get a person open. He can't know. Oh, oh, they did the stack This is miserable.   I really just cannot get over this throw it up. That's a great play, man I'm too mean, bro. I really am too mean. I Told him bro. You gotta find something else to do at your time. I just like how do you play like that, dude?  I can see if he was like actually like it was calm