Welcome, or welcome to this session. At this session, we're going to talk about breaking through cluster boundaries to autoscale workloads across them on a large scale. And this is Xin Yan Jiang, software engineer at DotCloud. And my name's Ying Zhang, and I'm a software engineer at Zendesk. We're going to cover the HPA pod autoscaling and the benefit of autoscaling across clusters, multi-clusters, and introduced to Kamada. And the Kamada features about federated HPA and crown federated HPA. Then we're going to do a live demo, then five to three to five minutes on the Q&A. Let's, like, I'm assuming that everyone is familiar with HPA, but let me refresh. So HPA, the horizontal pod autoscaling, is a mechanism for automatically scaling the number of pod replicas in Kubernetes cluster so that it dynamically scale an application based on the current metrics. So there's no. OK, so during each time period, the HPA controller manager carries resource utilization based on the metrics specified in HPA definition. The controller manager identifies the target resources defined by the scale target reference and selects the pads based on spec selector labels from the target resource and retrieve the metrics from either the resource metrics API or the custom metrics API. Here is an example of HPA. So you can see that the number of the replicas of those HPA Apache deployments, the number of replicas will be increased or decreased to maintain the average of 50% of CPU utilization, which means the number of replicas will be scaled up or scaled down if the average CPU utilization is above or below 50%. HPA works perfect in a single cluster. When we're thinking about multi-clusters, how can we handle autoscaling across clusters?  And we can have a unified management of autoscaling operations across clusters that can reduce the operation redundant and also break through the resource limitation of a single cluster, for example, there are massive requests coming through in a short period of time. And we run out of, for example, the instance. And also, to meet different scenarios, we would like to have a variety of various strategies that can scale the workload across multiple clusters, not only by the cluster name. And the scenarios can be like the disaster recovery, like the cluster level, autoscaling full disaster recovery also, you can thinking about other scenarios like cost efficiency, something like that. But how can we achieve that? Carmada can help you with that. Let me take a moment to introduce Carmada. So it is a Kubernetes management system that enables the cloud native application across multiple Kubernetes clusters and clouds. It has a bunch of capabilities listed here. So I'm gonna start from the first one. It is a Kubernetes native API compatible. So it's spiked directly to the Kubernetes native API. So which means you don't have to make any change to our current applications. And it's open, and naturally, open is open source tool. Recently, a couple of months ago, it has been moved from a sandbox project into incubation. It avoids the vendor lock-in. So it has integration with majority of the cloud providers. It also provides the building policies to assess for the scenarios like active-active, remote disaster recovery, and geo-redundant. Oh, sorry. It has a variety of workflow scheduling policies, like a cluster affinity, multi-cluster splitting, rebalancing, multi-dimension, HA, multi-dimension, highly availability across the regions, easy clusters, and cloud providers. It also offers centralized management. By saying that, we're gonna move to the next slide. It shows how Kamada looks like. On your right, it shows that the Kamada control plan is more look like Kubernetes native, Kubernetes control plan. So it has a Kamada API server.  Kamada scheduler also has a SED, sees as a database, as a data store, so it can store all the objects. And the members, like each individual Kubernetes clusters, can be joined to this control plan. And either we push mode or pull mode. On the right, it shows like the Kamada workflow. As I mentioned before, Kubernetes native API capabilities. So it used the resource template, exactly the same with the Kubernetes APIs. So, and let's move down a bit. The propagation policies, the user design default, how the workload be propagated to the member clusters. With that, the policy controller gonna bind those propagation with the resources accordingly. And after those resource binding generated, the bind controller gonna take the override policy to do a JSON patch to the resource binding and generate the work objects into the specific namespace. And the propagation controller will take the actual propagation to the member cluster. This is more like a before and after, so before we using Kamada. It's more like an user or something that manually deploy each deployment to each member cluster. So a lot of like an engineer toil or duplicate operations. But after we use Kamada, it offers like a unified or centralized control plan. So all those like workloads can be propagated to the right member clusters based on the propagation policy of that. So it's more like a one step instead of the redundant operations. But you may ask like every have a perfect like CICD pipelines to handle all those work for me. Like I'm not like manually deploy all those stuff, but Kamada also offers the workload propagation policy. So advanced policies. So not only based on the cluster name, it can like based on the labels, the field, tent and tolerant and also topology and available resources. So let's take an example on the topology and available resources. On the left, it shows like in most of the workload are scheduled to like zone X and maybe less workload schedule on zone Y and you can define all those in your propagation policy. On the left, it shows that the workload is scheduled on the cluster member two in the middle because it has more CPU. So it kind of makes sense, right? We have instance reserve instance, like in AWS, we have reserved instance and we want to take fully usage of those instance. So we schedule more workload there and for those like less CPUs available in the cluster member one, in cluster member three, we just schedule less workload. Small customer efficiency. And I've been talking a lot about propagation policy. How does it look like? This is an example. It shows like how we propagate the NGINX deployment. You can see there is a cluster affinity and it's defined like this deployment gonna be propagated to cluster member one and cluster member two with the replication reference as weighted, the last two lines over there. So it can be like static divided, like we just have static members like one or two. It also offers like dynamic weighted. So you can define it like based on the available resources. There is, under that, the last line is replica scheduling type. There are currently two types available. One is duplicated, the other one is divided. We're gonna, in the demo, we're gonna use the divided but duplicated you can think about when you're doing a blue, green migration and things like that, during the migration, you wanna have the workloads existing on both clusters. And I just, I don't think I covered all those policies. There's a URL at the bottom that you can visit to the Commodal website. There's more detailed information there. The other scenario I would like to mention is the cross cluster application failover. On the right, so there are three member clusters registered to the control plan and one of them is in trouble. Maybe the API connection, maybe the API connection cut off or something you can imagine, like all of a sudden the member one is unavailable, it's not ready and it will be detected by the cluster controller with discovery failure and the workload on the member one will be automatically scheduled to the other available clusters. So it's gracefully and migrate ensure on interoperative service. All right, I've already covered the first topics and then I'm gonna hand over to Xin Yan and he will walk you through the Commodal features, federated HPA and crown federated HPA. Okay, okay, thank you. And that's been, as we will know, when we want to use resource in Kubernetes, we should first look at the API definition of natural resource. So now I will introduce the federated HPA. As you can see, the federated HPA is consistent with the Kubernetes native HPA. And for example, the spec field is very similar to the Kubernetes negative HPA. They both have fields such as mean webcast, max webcast, metrics behavior and so on. So you can also see not the metrics field here reference the fields of negative HPA. Okay, so how can we use the federated HPA? And let's take a close look at how to migrate. As I described earlier, the federated HPA is very similar to the native HPA. So if you have any experience using negative HPA, you can easily work with the federated HPA as well. For example, on the left is the negative HPA, on the right is the federated HPA. You can see not, the two examples only differ in API version and can't. Therefore, you don't need to make any change to your existing HPA as spec field. Okay, this is very important. So how can, so let's see how it works. The KMANA, the federated HPA controller under control plan obtain metrics of the deployment through the KMANA metrics adapter and dynamically scales the number of replicas of deployments and then KMANA scheduler will schedule this newly added replicas to different member clusters. For example, member one and member two based on the scheduling policies specified by the user in the propagation policies. Ying has already introduced the propagation policy. So this enables course caster auto scanning, okay. The federated HPA controller is located in our KMANA controller management component. It queries metrics from KMANA API server and the scales replicas, of deployments accordingly. And then KMANA API server queries KMANA metrics adapter and then the KMANA metrics adapter queries the member, metrics queries the metrics server of member clusters to obtain metrics or customer metrics. And then aggregates them and finally returns them to KMANA API server. So you can see that the federated HPA emulates the mechanism of negative HPA, okay. With the latest version of KMANA releasing there is a new API, Chrome federated HPA. It is permanently used for scenario neck. If there is a traffic spec every day at the 19 a.m. I would like to proactively scale up the related servers or hand of time, for example, 13 minutes before to handle your people loads and ensure under gravity availability, okay. And another scenario neck to schedule database migration jobs on weekends, okay. In general, Chrome federated HPA is used for regular auto scanning actions. It can scale workloads, not to have a scale separate source or federated HPA. And let's take a look at the two examples of Chrome federated HPA. On the left, we have an example of schedule scanning for federated HPA, either it includes a schedule field for writing Chrome table expressions and target the mean webcast field. On the right, we have an example of schedule scanning for deployment. It also includes a schedule field for writing Chrome table expressions and a target webcast field, okay. So when the specified time arrivals, the Chrome federated HPA controller will trigger a scale up or a scale down actions allowing you are to proactively scale up or down your deployment, okay. Kemana implements the Chrome fed HPA as a control loop, not proactively checks the Chrome schedule time. If the schedule time is reached, it scales workloads webcasts or federated HPA mean webcasts or max webcasts. So you can see that the Chrome federated HPA working mechanism is very simple, okay. As I described early, a Chrome fed HPA can scan workloads with a scale sub resource and the scanning federated HPA. And so let's take a look at the consideration for usage. In general, it is important to ensure that the scanning operations performed by Chrome federated HPA and do not conflict with any other ongoing scanning operations. And so it is recommended to first use Chrome federated HPA to scale federated HPA and then federated HPA can scale workloads based on their metrics. And then let's take a look at the advantage and the notice of Chrome fed HPA federated HPA advantages. The API and Kubernetes native HPA are almost identical with the same user experience and low migration cost. And notice federated HPA is the type of the center. Oh, okay, okay. And, okay, okay, federated HPA is the type of the centralized and multi-cluster HPA when the scanning concurrently on a larger scale or larger by the way is required. And the storing and then computing Chrome responding data requires match CPU and memory. Okay. In the face of some sort of comments of federated HPA, the commander community is exploring a new API, distributed HPA, which he introduced an agent in member clusters. It will list and watch distributed HPA and calculate the intermediate states. And then the intermediate states will be updated to the status fields of distributed HPA. And then Kimada calculates the final results and adjusts the number of webcasts based on intermediate data reported by the agents in the member cluster. Okay, currently this design under discussing. So you are welcome to join us. All right, at least let's take a quick demo before we move to the takeaways. So in the demo, we're gonna show you how we use Kimada to propagate the workload to the member clusters. We're gonna have three Kubernetes clusters in local.  So this on the right terminal, oh, sorry, right terminals. And those are two separate individual Kubernetes cluster will be joined as member cluster. But currently, in those member clusters, they're not joining yet, but as a prior request, the metric server has been deployed to those member clusters. We have set up the Kimada on the Kubernetes cluster. All right, let's do the join using Kimada CTL. You can see both member cluster have been joined to Kimada successfully and we are gonna get the cluster. It can show that the status of those clusters, they are ready, which means they're ready to be propagated.  So you can see the replica set to two. And then we apply this deployment to Kimada, created. But I would like to, if you wanted to, yeah. But there's no workload on the control plan or other of the member cluster because we don't have the propagation policy set up. And after we, yeah. Let's take a look how the propagation policy looks like. It has, it defined how we schedule the replicas is weighted and one in each member clusters and it's divided. So we are expecting to see one replica in each member cluster. Let's apply this and you can see on the right, there are replicas. And you may see that then we apply the federated HBA. Yeah, let's take a look what it's look like. It defined the minimum replicas as one and the maximum replicas as 10.  With the CPU utilization set us like 10%. So we are expecting to see maybe one of the, oh, it disappeared. Maybe one of the, one of the replicas disappeared because there's no like load, actual load or traffic into those NGINX application. We've, we've logged in to one of the member cluster and we do a log generator using AB, the Apache bench. So is the command is shows like in within 20 seconds, we're gonna make generate 10,000 of the concurrence. And we are expecting to see more replicas spin up and the member clusters and evenly, which means like maybe one, one, one or two, two. Okay, three, three. Is it three, two? All right, you can see the difference is because it's automatically calculated based on the available resources. Maybe in the member one, we have more available resources but all those three clusters are generated by kind. It's all local. And the load generator is completed and we may see that the replicas will be scheduled down within a couple of seconds. Well, a couple of seconds means like 10, 20 seconds. Oh, there we go. All right, that is the end of demo. Let's move back to our slides. So I wanna take a quick moment to recap for the key takeaways. Here is the regular HPA can now break through the cluster boundaries to out of scale the workloads but the federated HPA and the crown federate HPA can help you with that. And Kamada is an open source tool that enable you to run your cognitive application across multiple Kubernetes clusters and clouds. So I would encourage everyone to give Kamada a try and see how it fits in your USKs or scenarios. Here are the links. Kamada doc, the GitHub repo and we have a Slack channel named Kamada and you are more than welcome to join us. Thank you so much. All right, thank you so much for listening and I just wanna give a shout out to the Kamada contributors and the maintainers. You're awesome. And we are appreciate your feedback. Please scan this QR code above to leave feedbacks for us. And yeah, we're, we still have a couple of minutes. I'm happy, we're happy to answer any questions you may have. Yes. Oh, the question was, so the Dow Cloud is currently using Kamada. I'm gonna let Xinyan answer those questions. Okay, we can discuss offline. Yeah, a short answer is yes. Yeah, I got this correct answer. Yeah, but we can discuss the more details like offline. Yeah, cool, awesome. The gentleman over there, it's your question. Yes, so we saw how you were able to scale your workloads across multiple clusters as traffic came in. But my question is, how are you handling Ingress? Because once workload scales out to another cluster, your Ingress endpoint's still on cluster A. But now you've got a workload running on cluster B, but no traffic's necessarily going to that. So how are you handling that problem with Kamada? So if I understand it correctly, it's the question is when the ingress of a single cluster is coming in, the traffic in, but we've scheduled the workload on the other member cluster, is that correct? Right, right, and without something like GSOB, I mean, I'm wondering how that effectively scales across multiple clusters. Can you answer those questions? It's more like underlying networking, setting up. Right, right, there has to be a networking part to it. Okay, so as far as I know, the underlying network, the best practice is to using Submariner. So all those clusters, all the underlying, all those member clusters are using Submariner, so they can be like talk to each other and handling the traffic across the cluster. That's more like an assumption of that. But if you have more clusters, like we can discuss offline and see your use case or you can open the issue for us in the repo. Yeah, I'll hit y'all up in the chat, slack room, thanks. Okay, thank you. The next gentleman. So in your examples, you're using the default namespace? Yes. If we have custom namespaces with custom resource quotas and limits and policies, do we have to create that namespace with all of our resource quotas and everything across all the clusters? Or do we do that at your top level, Carmada, and it just creates that namespace across all the clusters? I got you. If I understand correctly your answer, your question is when we're using Carmada, how it generates the namespace in the control plan? Is that in the control plan? Yes, when we deploy the, when we do apply the deployment, it will generate a default namespace in the Carmada control plan. And after we apply the propagation policy, it will generate the specified namespace in the control plan, like starting with the Carmada-es and it insured all those work objects. And the format is Carmada-es-clustername-namespace that you specify in your ML file. Gotcha. We can show you the real thing. We can show you everything else or carried across all the next cluster that it propagates to? Yes. And it will create all the namespace in all those member clusters. I think you can see from there. Gotcha, gotcha. We should show that in our demo, right? Okay, thank you. Okay, cool. I hope you answer your questions. Thank you. Thank you for your question. Yes. Sorry, I cannot hear you. I have one question related to federated HPA. Does it support external metrics like Datadog or any other PrimotheSys metrics? Sorry, I still didn't hear you. The federated HPA, does it support external metrics? External metrics? Yeah, right there. I think we offer external metrics. It'll be a Prometheus. Like, have you used it with, for me, three of us? Yeah, we can offer that metrics for monitoring. Yes. There is a session in the website and show that how we're monitoring, alerting based on those metrics. Not only for the HPA or federated HPA, but in general, there are metrics available. Yeah, thanks for your question. Yeah. I have the same question, but what happens in a network partition where your control cluster doesn't have access to the other clusters anymore? So the question was, what happened if the network is in trouble? There's an outage in the network between your control cluster that's managing the federated HPAs and the remote clusters. What happens to the HPAs running in the federated clusters? Oh, the question was, what if one of the member cluster allows a connection? Oh, what if the control cluster can't talk to the remote clusters? Oh, I got you. So the question was, sorry about that, the question was about how we handle if the commodity control plan is unavailable. So if the control plane can't connect to the remote clusters, they're happily doing their thing, the remote clusters are working. Yes. But the control plane can no longer talk to them. We cannot answer your question at the moment because we, I'm assuming that all those workloads are running happily on each member cluster. It would just stop scaling? Yes, self-scaling. So it would no longer scale in or scale out. It would just run at a steady state? Yes, I'm assuming that you have your HPA set up in the each member clusters. Yes, the HPA are going to handle all of those products of scaling in each member clusters. I'm hoping to answer your question, sorry if I didn't. Feel free to open. I was specifically asking about the distributed ones. Distributed? Where, because it looked like the example you just ran, if cluster two, if the control cluster couldn't connect to cluster two anymore, would it just not schedule any more workloads on there? Oh, to be clear, so the member cluster still registers as a member cluster to the commodity control plan, right? Right. Because if we un-join, it's going to delete all the workload from member two. But not an un-join, just like the network goes on. Network issue. Undersea cable gets cut or something. The workload is going to be still running it inside a single member cluster. But in the central, the federated HPA cannot work, cannot scheduling those workload across the clusters. It lost control. Right. But it doesn't affect the workload. The workload is continuing running, but they would not scale in or out? No, yes, without scheduling across the cluster. But it can be running in a single cluster. Yes, yes. OK. You had a slide that showed when your control cluster lost connection to one of the member clusters that would migrate the workload to one of the working clusters. How would you be able to migrate the workload if you've lost, if the control cluster has lost the connection to the member cluster? OK, the question is, there is a slide showing that we lost the connectivity between the... But to be clear, it's not a connection between the control plan to each of the member clusters, just only one of the member clusters. One of them, and it showed you would migrate that workload to a new cluster. Yes. But I'm curious, how would you do the migration because you don't have access to that cluster anymore? So how would you shut down that workload on the cluster you can't reach to create that on the new cluster, or on the working cluster? OK, I got your question. So the question is more like the whole cluster, the workloads, you're running on the bad clusters. And we just autoscale the workloads with the other member clusters. How are we going to handle this situation? But what we're trying to explain is that cluster is in trouble. So the workload disappeared and is based on the feature gate, which is the enable failover. So it's automatically scheduling to the other member clusters. OK, so it may schedule some new workloads while the workloads on the cluster you can't reach anymore might still be used. We're just assuming that cluster is down and the workloads can out handle any traffic. I see. OK, thank you very much. Thank you for your question.


When the narcissist has trapped you in their web, their complex system of interconnected elements, their group of interconnected people or things, you feel as though you are enmeshed with them, you are involved in a difficult situation from which it is hard to escape, you want to break free from their confinement or control, but it's not that easy. So instead, you create these temporary distractions from your reality or routine. It is an act of avoiding something dangerous or unpleasant. It's like you can't stay, but then you can't leave, so you try to escape from reality. You may do things that you're not proud of as a means to escape. This may involve alcohol, drugs or gambling, and while you wish that you were not engaged in this behaviour, it helps to distract you. For a moment, it allows you to escape your problems or difficulties. Because while you're around the narcissist, you're always walking on eggshells. You have to be extremely cautious about your words or actions, because anything could offend them. Anything could trigger them to lash out at you, so you begin to withdraw from life. You begin to isolate yourself, because it is the only thing that you can do now that the narcissist has trapped you in their web, and while you may feel like you are missing out on life, while you may feel like you are no longer the person you used to be, I can promise you that it will all get better if you listen to what I have to say. Don't resist the isolation. Don't resist going within yourself. This doesn't have to be something negative. Going within yourself can be very beneficial for your growth and development. It may feel like you're all alone. It may feel like no one is there for you, but you can be there for yourself. Everything you need is within yourself.  You are now going within and learning more about yourself. After this process of self-reflection, you will soon emerge as a beautiful butterfly, and no one can take that away from you. No one can stop this from happening.  Because your inner world needed your attention, and now that you are aware of this, you can now attend to it. You can now become your best self. You can now become a beautiful butterfly. Thank you for watching. I hope this video resonated with you. Please like, comment, share and subscribe. Click the bell icon to receive notifications for my future videos. If you'd like to donate, my PayPal link is in the video description. Coaching inquiries you can email me at nicewithcoaching at gmail.com. Thank you for watching and I'll talk to you soon.


Energy 808, the cutting edge. We're here with Marco Mangelsdorf, our co-host, and Eric Gleason. Formally of Next Era. Is that the right way to put it, Eric? Formally of Next Era? Not to my knowledge, Jay. I'm still employed by Next Era. Okay, alright. Formally of Next Era, as it was in Hawaii. I remember the day I met you, as a matter of fact, at the Plaza Club. I don't know if you remember that. I do. So, we're going to have an introduction, if you don't mind, Eric. This may take, you know, maybe 40, 50 minutes. But Marco has been planning this for some time. So, Marco, why don't you introduce Eric Gleason. Well, thank you so much, Jay. And thank you, Eric, very much for being on with us today. I really appreciate it. Something I've looked forward to these past months. Eric and I go back a ways. We first met sometime mid-2015, when the attempt of Next Era to buy, when I like to get in the streets, was in full play. I met him at my favorite restaurant in Hilo, Hilo Bay Cafe, along with Jen Zelko, my friend from Helico. And I thought, who is this guy who's so much more handsome, so much better dressed than I? So, so just on top of things, I thought to myself, I got to get to know him. And we stayed in touch through the course of the attempts of Next Era to buy Helico and Electric. And we stayed in touch afterwards. And I'm great pleased that I've developed a friendship with Eric. I think we're highly up him. He's a really solid, good guy. And I'm glad that despite his company's, so we say, lack of success at buying more electric industries, that we've had a nice friendship. And I really appreciate Eric and what he does, what he's done, what he will do, past, present, and future. So thank you so much, Eric, for being with us. And I think my first question to you will be, what in the heck have you been doing with yourself since that fateful day, mid-July 2016, when the news came down and that our public utilities commission decided not to go with the acquisition? Well, thanks, Mark and Jay, first of all, for having me on the show. Marko, for the kind words. Well, the first thing we did, so my wife and my three kids were living with me in Hawaii at the time. And the first thing we did is I took two months off and the first month of it, we spent more time in Hawaii. We love Hawaii and we wanted to actually even spend a little more time in the island. So we did that, then packed the kids off and my wife and I spent a month in Japan. And with two months of having decompressed, I was ready to go back to work and went back to the next era in Florida and continued with a job really I've been doing for the last nine years, which is building a utility business around the country. It's focused on high voltage transmission. And you know, that's what I've been doing. Well, you know, I wanted to ask you to, I mean, you know, this was traumatic for a lot of people who wanted to see that deal done, including me, for example. I told you that before and I was traumatized by the way it went south. Not only it went south, but the way it went south, you know, the kind of things that people said it did and the way the community, if you will, handled it. And I've always wondered, we haven't talked in a couple of years, I've always wondered how you felt about that. How about a walk down memory lane? How do you feel about what happened? Is this my therapy? You know, it was tough. It was, to quote Dickens, it was the best of times. It was the worst of times. It was a year and a half that in some ways I'm so grateful that we had. Hawaii is an amazing place, as both of you know, as I have come to appreciate. And, you know, that part of it was fantastic just getting to know people from all walks of life and the different islands. And the support we had from people like yourself, less so Marco, but, you know, I've forgiven him. But, you know, then there was the other side of it, of dealing with politics and then frankly, for me personally, you know, I bore a lot of guilt. I felt really bad that I led my company in here. You know, some things were said about the company that weren't true and weren't nice. And, you know, I let down a lot of people. A lot of people, not everybody in the state, certainly, but a lot of people in the state supported us. And I was really disappointed that we couldn't get it done. So it was tough. And at the same time, we have, you know, just phenomenal memories of Hawaii. We still go back there. So, you know, with time, the sort of the pain has dissipated and I'm able to just appreciate what's great about Hawaii. That's great. That's great. I really enjoyed meeting you, Eric. And I felt this, that if somebody were going to come in as an executive to acquire a company, a large company, maybe the largest company in the state, it should be somebody like you. That's the way I felt about it. I still feel that way. And I wonder if there's any possibility that at some time in the future, this might all mount up again, either in, you know, for a next era or some utility company like next era or for some company that you lead. Well, thank you, Jay. And, you know, it's, it's a consolidating industry. There are probably half as many utilities today as there were 20 or 30 years ago. So, you know, I guess anything's possible. I think, you know, for ourselves in Hawaii Electric at this point, we're very much focused elsewhere. And I, you know, I don't see that in the carts.   And so when you're coming from the mainland, you're starting in a whole and, and the utility is super important to the state. And people know that, right? It's the most essential of essential infrastructure. You know, I wouldn't say, I'm not talking about for myself or next era, I think we've all moved on, but, but I wouldn't say nothing could ever happen with Hawaii Electric. But I think there's a very good chance that they're going to be an independent company for a very long time.  I mean, as I see it, and I'm interested in your comment on it is, you know, we're in a national consolidation, as you said, of utilities, a transformation in some ways. And we're also, you know, in a time when the technology is changing, I guess that's part of the transformation. And in a time, in a time when COVID is transforming our society in general. So there must be all kinds of changes going on nationally that bear on this, on this kind of issue that your business is in, in change, it's, it's changing right now as we speak. Can you talk about that? I mean, that's certainly true, you know, utilities, as essential infrastructure, we've all continued to operate throughout. And so like for myself, I, while I work from home, a lot of people from our company work at home most of the time, we have a lot of employees who need to be out in the industry or need to be at their jobs and the safety, you know, keeping those employees safety, safest paramount and more difficult. And so, like any business, or like any family right now, we're all very focused on safety. But that said, I think utilities are very robust to different economic environments. And I think the utility sector overall, and this is, you know, is true of point electric or next era as it, as it is of any of them, you know, have held up very well. So, so I don't think, you know, that alone is not, I don't think changing the dynamics of, you know, strategic outlook for the utility sector so much. It's just a challenge we need to work through now. Well, you know, the economy is, I mean, it's still teetototter, we don't know which way it's going to go. Powell the other day said, oh, it'll bounce back, but you know, there are those economists that say we're in line for a serious depression. And depressions are not good for utility companies, nor are they good for the price of oil, for that matter. And I wonder, you know, how a utility company, how the industry plans to deal with right now an unknown level of recession or depression, because if, if, you know, if we go into a depression, people are not going to turn the light switch on. And so you don't need, you don't need to have that much generating power, and you will have less revenue. And that sounds like it's a challenge. How does the industry deal with that? Well, I'm glad you're not talking to our stockholders because they don't do it quite that bearishly. But, you know, I think, first of all, I think it's probably true that electricity consumption was growing during the Great Depression. But that's just a quirk of the time that the electricity business was in its infancy and, you know, electrical penetration was happening in our society. But I think more broadly, you know, I think the last thing, or near the last thing people are going to not buy in an economic downturn is electricity. They may have trouble paying their bills, and a lot of customers are having trouble paying their bills right now, that's a concern. But ultimately, I think what you typically find is that the regulators ensure that that does not jeopardize the financial health of the utility because that would be, you know, against the best interest of everyone. And so I think it's quite a recession or even depression resistant business. And I think, you know, really, if you look at how investors look at it, it supports that. Utility valuations have come off a bit, but they've held up better than other sectors.  And also the debt markets are open to utilities. So one of the ways that utilities have responded to this environment and thinking about some of the worst case scenarios that you're describing is actually to tap into the bond markets. These are typically investment grade rated companies. And they have, you know, preserved that access to the bond markets, even when, you know, those bond markets have been quite volatile for others. So, you know, it's, it's I won't say it's this is nothing from a financial point of view or an economic point of view. Of course, it's not this is a big deal. Not to mention the health aspects of it, but I think the utility business is holding up pretty well. And Peter Rossick of Align Electric, you must have dealt with him while you were there. Good guy on the show last week. And one of the subjects of discussion was, you know, how, how people, how lifestyles have changed. And what are people doing? Well, you know, they're on the internet more at home, and they're watching television a lot. And to do that, you really have to have electricity in your house. And if you didn't have electricity in your house, your life would really be, you know, troublesome. You wouldn't be able to do the things that the, the remaining things you can still do. So I think, I think that must be part of this whole formula going forward. But I wanted to ask you, you know, there was talk, I'm afraid I don't know the answer that was talk about the federal government as part of the CARES Act giving, giving support aid to the utility industry. Has that happened? Should it happen? What's, what's the need? What's the, what's the fact? I think you asked Commissioner Potter that last week and I have to say I have not heard that. I don't think there's anything that's applicable to our company. I don't want to say that shouldn't happen because there may be some good ideas people have, but I would say overall the, you know, the utility business is a pretty robust business.  So, you know, we're not immune to what's going on, but I think, you know, economically we're fine. Good. Marco, you have questions? Always. I knew that. Eric, if someone, let's say that our folks at D-Bed, Department of Business, Economic Development and Tourism were to find a not previously unknown stash of money to commission a manual, a primer, and they were to go to, to you Eric Leeson and say, you've had quite the experience in your efforts with Next Era to buy one of the oldest, most vulnerable companies in the state. And commissioned you to write a manual in terms of dos and don'ts. Now that you have had the years and the trenches in Hawaii, now you've had several years to recuperate, so to speak. What are some of the top-brow, high priority bullet points that would make up your executive summary for that manual for those who foolishly or bravely, something in between, want to try again at some point, which they will, sooner or later, something from outside will want to go after one of the jewels in the Hawaiian business crown, so what comes to mind? Yeah, so this is a manual on buying a utility in Hawaii? Well, not necessarily just the utility, but... I'm not sure anyone would buy that book from me. I would. Oh, you'd be surprised Eric. And if they would, I'd probably have some other things I want to sell them. Well, you know, I actually thought about writing a book. It wasn't serious. It was playful. The fun part was to think about what the title would be. You know, how not to buy a utility. So, look, clearly I am not someone to give any lessons or dispense any wisdom on that. But you have wounds, Eric. I do have wounds. You've learned.  Come on. I'm trying to decide what to say and what not to say. Listen, I do have a lot of thoughts on this subject. You know, honestly, I come back to, I think you have to start by why didn't the merger happen? And, you know, there were a lot of things that were talked about and a lot of issues people raised. I think in the end, you know, 80 20, I think for most people, it was about it was about two issues. It was a trade off two issues. One is I think people really do and I know this I've seen the polling majority of people in Hawaii, don't trust manly corporations and they would prefer for the, you know, the utility to be as much as possible locally owned control. I think that's just an inherent feeling in Hawaii. And by the way, I, you know, I have a lot of sympathy for that because, you know, I have some familiarity with the history here in Hawaii, so in Hawaii. So that's one side of it. And that was clearly there was something that was going to be given up there. And then, you know, the other side of it is, at least in this particular case, there was the prospect of, you know, of saving some money off your bill. And I think, while, you know, people in Hawaii when they're polled about what's important for them about about energy they cite cost is the number one thing or at least they used to now. You know, when it came down to it, the cost thing and this is me rationalizing because I don't really know we didn't really poll like how do you view this trade off between cost and local but I think in the end, people knew they would be giving up some local ownership and control and they couldn't really trust some mainland corporation to give many cost savings. And so they were happy for it not to happen. I think that's the reality for the, you know, for the man in the street. And so I think that's a very difficult thing to overcome. Unless you have strong political leadership that sees past that and of course in this case the political leadership actually was more concerned about the local ownership and control and so it's very very difficult to overcome that so probably my advice would be unless you have very strong political leadership behind a transaction that fundamentally, you know can deliver some good things for Hawaii. Don't try it, not in the utility business it's too sensitive. The leadership is going the other way. You know, for example, want to correct me if I'm wrong or one of the elements that you saw that next era saw was the possibility of LNG at least as a bridge bridge fuel, at least, and indeed that's happening all around the country. It's happening all around the world. But the governor, the governor said no on both counts and that that that's not the kind of leadership that can that can allow for a deal. Yeah, I mean the LNG wasn't our idea we we you know was, you know, Hawaiian Electric actually originated that but we could have helped them do it. Well, obviously the governor wasn't interested and so that wasn't going to happen but you know I think the merger was more was about more than that. But I think at the end of the day that my sense is and it's hard to know for sure exactly, you know, Governor Igay's issues because, you know, he said different things at different times I'm not sure we really know exactly what his thinking was but you know the fact that we weren't the fact that we were a mainland corporation did did seem to factor into it based on some things he said. Marco. Over the past, she is I don't know, 10 or 20 years I've been what you know full time by 20 years. There's been discussion from time to time about interlinking the islands with a submarine cable power cable, so that you can conceivably have a big wind farm on Lenai mobile Chi geothermal on the big island and essentially interconnect all the islands and there's been a million spent on studies and so forth. First glance it would seem that it would make sense. Practically speaking, over time to have an interconnected state, electrically, as you have an interconnected US mainland east, east, west, north, south transmission distribution. What's your take Eric in terms of the feasibility do ability possibility of said submarine power cable. Well, first of all, I have to Jay, I think doesn't remember but actually the first time we ever spoke was in 2013. And he called me and wanted to get me on the show and I think it was Skype at that time and I was a technophobe so you know I'm like I don't know about that. But that was a while ago so actually the idea goes, you know where the idea really is Marco. Well, you're close in time, it was actually King David Calakawa and Thomas Edison at a meeting in New York in 1870 something I think. Maybe the 80s. So, is it is it a is it a technically viable thing. Yes. Could it make economic sense. Yes. Do I think it will ever happen. No. The reason for that is, you know, it would be controversial. And I think it's very difficult to get multi year capital projects done in Hawaii if they're controversial. There's an element on that. Yeah, we talked and I supported that also out of Lenai, but that, you know, it's a subculture point in Hawaii, each island has its own subculture. It has its own identity and there's not there's not really a lot of collaboration between the islands. Yes, it's a state, but it's a state of islands rather than an island state. And so, for example, and this is so close in concept that those forces stopped the super ferry from coming here. Those forces have have prevented the islands from collaborating on so many things, which would happen if they were, you know, all one one piece of land. And I think I totally agree I don't know if that's what you're saying Marco but I, I totally agree that it's doable, and it should be done, but then it won't be done, because of those political cultural considerations. In the mainland, there are cables everywhere, isn't it? Well, I wouldn't say that we so actually my business, we own one which is a 53 mile undersea cable that that supplies 40% of the power in San Francisco. So I'm very familiar with with you know how they work. There's not only one of its type on the West Coast, the type of cable that if you were going to do one in Hawaii you would have done is very similar to this cable we call Trans Bay in California. And then there are several on the East Coast that are comparable and then many in Europe, and as well as Asia, I have some as well. But many in Europe to, you know, to interconnect across the various seas, different countries. I've seen or heard of Michael Moore's new and less than wonderful movie about the environment and and energy it's called the planet of the humans. And it runs, it runs renewable energy down in every way you can think of it's very interesting. Well, like I don't know if I can watch that but maybe maybe watch the trailer. Marco, have you seen it? I have and once we've all seen it maybe we can discuss it then but in a minute we have. I wanted to ask you, Eric. What do you think are the biggest juiciest challenges that we face in Hawaii regarding our energy present in our energy future what do you think distill it down to to less than a handful or even two or three points. What do you think the biggest challenges are that we face and do we have much hope should we have much hope of doing a better job with these challenges. Yeah, I don't know if I'm probably not the right person to speak to challenges today. You know, I think over the long term over the next couple of decades I think there are to me there are probably three big challenges. One is high renewables penetration. It's easy to contract to buy or build renewables. But when you get to a higher and higher percentage once you get over about 50%, it gets increasingly difficult to manage and more expensive. And so, you know in the years to come not now but you know, 10 years or so from now it'll be interesting to see how Hawaii how Hawaiian Electric deals with that. And another one is hurricanes. You know, I live in Florida, our company Florida Power Light deals with hurricanes I, you know, how to catfire hurricane coming my way. Last year, it was, you know, get you know supposed to impact us and then, you know, fortunately turn north. This is scary time it seems like storms are becoming bigger and more common. And, you know, I've kind of from from what I watch in Hawaii it's kind of a similar story and so, you know, think about what Aniki did to Hawaii it's very scary what what could happen from a hurricane. So that's one I just think that problem is not is not getting better. And I think the other thing is costs, you know, integrating those renewables, preparing for hurricanes hardening your system for hurricanes, and then keeping the costs down or three are going to be three big challenges for Hawaii and Hawaiian Electric and, you know, I'm a big fan of that company and the team they have there and, you know, I know they're doing a lot of the right things and, you know, I expect the best, you know, from them and for Hawaii but I think those are the challenges. And I think there's probably, you know, if you compare the level of the degree of difficulty that Hawaiian Electric has on those three issues. Really, I think there's no one else in the industry that has the challenge that they have so they've got a lot on their shoulders. What about national national policy national trends and renewables seems to me there was a time when when people and I mean everyone were more excited about renewables than they are now maybe it's the administration, maybe it's Congress. And I just wonder how you feel about whether the federal government could do more should do more to encourage to incentivize to move us further faster on renewables. You know, this might surprise you.  I mean, we're the largest one in the world. It's a big business for us. But even with that, we are fine with, you know, tax credits being phased out as scheduled. Our view is renewable energy is has become extremely cost competitive with, you know, conventional fossil and nuclear fuel plants, and we're ready for for renewables to compete without without the tax credits, and which are all going to roll off by the middle of this decade. And we were there so, you know, they're they're getting so low cost and, you know, in more competitive markets I mean Hawaii is a vertically integrated market it doesn't quite work this way but, you know, in other markets in North America we're able to actually compete head to head with with these older plants and, you know, they're having trouble staying viable so so we feel really good about the future just based on economics. You know, then obviously there's individual states that have policies that support like like Hawaii have policies that support even more renewables and you know economics might might dictate. We're fine with us with, you know, if there if there's a Biden administration might we have that at the national level, you know, we may or at some point we may I think. I don't know if I don't know if I would agree that, you know, aside from the Trump administration clearly clearly the President has it is not a huge fan of renewables but I think aside from him. The national politic politics have really gone against renewables but I feel like one thing that has happened is there's more consensus around climate change and that we need to address it, even amongst many Republicans. And so that leads me to be optimistic that you know maybe over time there'll be some helpful federal policies as well. Thank you, Eric. Okay, we're almost out of time. Marco so it's it's for you to ask a last and very important profound question and then summarize. Can you handle it Eric. Probably not but let's go for it. Thank you to note that I remember one of the criticisms against Florida power and light and next year back in 2015 2016 was somehow they hadn't gone for far enough for fast enough with renewables. And I want to do a shout out to Eric and Jim Robo and everybody there to your companies that from what I can tell over the past handful of years that it's significantly accelerated I mean it's not that you weren't doing nothing so to speak, prior to to the dance with Patrick, but from what I can tell me and one of the things you guys were saying hey look at what look at everything that we're going to do look at everything on our board so to speak. And you in fact, you collectively followed through with that which I think is, you know, kudos to you all for doing that in terms of pioneering lower and lower costs with both solar battery storage and so forth so well done routine there Eric and Yeah, I mean, lots of other questions to ask but I think I'll just close by expressing my my thanks and gratitude for you being on the show with us today and and whenever and Eric and I talk whether it's on the phone, the text email we meet sometimes in San Francisco we meet in Wackaloa on the Big Island and it's always a great pleasure for me to spend time with you Eric in whatever medium, whether remote electronic or in person so I hope to see you and and before too long on your favorite of the Hawaii Island chain. It's true. Thanks Mark. Thanks Jay. Thank you Eric. Eric Gleason next era energy thank you so much for joining us today. Aloha, stay well. And you know when you're not eating and drinking in the restaurant. Yeah, you can wear your mask. But I'd like to be there. I'd like to be there. Aloha Eric.


What's interesting is when I talk to some of the other tradesmen they'll tell me they kind of prefer working for women now because they're much more detail oriented than men like a male GC or a male PM will go figure it out but a woman will go all right let's walk it what are we doing here are the plans let's talk about it and way more thorough you know and I think that's a female superpower to be honest I think women are just caring in general and so whenever it relates to their craft it's no surprise they care a little bit more which obviously means better results