Welcome to the seminar this week. Our speaker today is Professor Sangyong Wang-mian from California Institute for Energy and Environment at UC Berkeley. Just quickly, I would like to remind everyone that our next seminar is in two weeks, on the fourth. Okay, we'll be on demand response and review in charge. And let's quickly give a brief introduction of our speaker. Professor Wang-mian is an adjourned professor in the DTS department at UC Berkeley. And she also directs the electric grid research program at California Institute for Energy and Environment. She's also a faculty scientist at the Berkeley International Lab. And her research focuses on power distribution systems including applications of mobile sensing, analytics and control strategies to facilitate the integration of ER. She has a book called Electric Power Systems a Contactual Introduction. I have read it before. So if you are looking for a book on Power Systems Introduction, I would recommend this one. Such a great receiver. That's a receiver, PhD in Energy and Resources from UC Berkeley. So without further delay, I would let us the slides. Thank you so much. It's a pleasure to be here with you. I'm going to talk about a labor of love that's called the Oakland EcoBlock. It's a very interdisciplinary project, a little bit less technical than some of our work in electric grid research, but very practical and applied. And I think ultimately it will be very rewarding because we're working with a real community within the city of Oakland to build a very novel type of microgrid. So I will tell you about that.  Our motivation in the big picture is a set of three challenges that I think all of us here understand. Number one, climate mitigation. Number two, resilience in the face of climate impacts. And number three, equity amongst different communities that may be impacted very unequally. So in terms of climate mitigation, we understand that we have to decarbonize the energy sector. And there are essentially three pillars to that. One is energy efficiency. Everything starts with cutting the waste, right? Stop the bleeding first. Number two, renewables and carbon neutral energy, which at the local distribution level generally means solar PV, as well as small-scale energy storage for the most part batteries. Hopefully other technologies like flywheels may become practical and cost-effective as well very soon. But then the third pillar is electrification that is shifting end uses over from natural gas and gasoline or diesel to electricity because the electricity is easier to decarbonize than the liquid and gaseous fuels, generally speaking. On the resilience point, really we're talking about survivability and recovery from extreme events, right? This has always been an issue, but particularly with climate-related challenges to the electric grid and to other infrastructures, we need to give more thought to that. And finally, making new technology and clean, convenient, healthy solutions affordable for everyone. So that relates to the equity part. I noticed that in your, I think previous quarter, you had Ross Baldic talking about the Texas power outages and that obviously had a lot of lessons learned from that event. And also here in California, we had some rotating outages. We had some close calls with our California grid. In our case when the temperatures were hot and Texas when it was extremely cold. And from those close calls, but actually really deadly impact, certainly in Texas, the death toll that has been reported depending on whom you ask is in the hundreds from the power outages last February. The takeaways were that extreme weather, extreme climate scenarios are real. Also that there is an important interdependence between and among infrastructures. In Texas, we saw the interdependence between natural gas production and delivery and the electricity sector. Also the intersection of electricity and water being out of power as one thing, being out of drinking water, quite another. And the recognition that all of our efforts to make the macro electric grid reliable, we must not lose sight of the fact that sooner or later there will be a power outage that may be beyond the control of a utility or a system operator to prevent. It could be a malicious physical or cyber attack or it could be weather scenarios that are simply beyond anyone's capability to prepare for. And so that means we have to save lives, not if but when the power goes out. I thought that the main lesson learned from Texas was that we need to focus on how to make blackouts survivable for people. Yes, it would have been great to have all kinds of measures in place to prevent the power from going out. But the real tragedy was that people died from entirely preventable causes such as not being able to power their medical equipment or simply from hypothermia, that wasn't necessary. And as you probably also heard in the case of Texas, what happened is that the distribution utilities simply did not have the tools in place to do intelligent load shedding. Certainly at the system operator level, at the ERCOT level, all they could do was tell the distribution utilities, hey, we need you to shed this many more megawatts. But the only way that the utilities could accomplish that was by de-energizing entire distribution circuits, which is when you think about it, a really sort of gross and imprecise way to manage who gets electricity and who doesn't, because what you want to do is you want to discriminate between the important and the unimportant uses of electricity, right? Anyone probably would have been happy to volunteer even their refrigerator or their air conditioning for the sake of saving someone's life who depended on an oxygen machine. But there was not that ability to prioritize. And one really important strategic change, I believe in smarter kids is to be able to prioritize loads more thoughtfully on smaller scales, both temporally and spatially. So that means sectionalizing distribution circuits. It means being able to turn individual customer meters off so that you can serve some essential loads on a circuit without having to then serve everybody else on that circuit, even within the home. There are now technologies like smart circuit breaker panels that allow you to essentially limit the electric consumption within a house at the level of the breaker panel so that you can power a few essential circuits and dial it down to some specific level of the number of watts and be sure that you don't go over that. So this is technology that we understand how to do. We've been loath to implement it in the past, I believe not just for technical but also for cultural reasons because so much of our narrative about the electric grid has been that there is supposed to be ample supply for everyone whenever and wherever you need it. And what we're facing today is the reality that providing for those last few nines of reliability costs a lot more than most of us are willing to spend on it. And yet it doesn't have to impact people in a particularly painful way if they are able to prioritize. And it can even go down to the level of individual smart appliances. There is a huge opportunity of course with technologies that allow you to prioritize loads also to shape your demand at different levels of aggregation that can then be remunerated in markets whether that is at the distribution level where I think there's a lot of room for emerging new mechanisms run by for instance, distribution system operators. Right now it's mostly at the wholesale or transmission level but there are also now third party aggregators that will aggregate smaller amounts of kilowatt contributions to load shaping into a product that's actually sort of noticeable at the wholesale level. So I think this is an area where hopefully we will see a lot of growth. It's really, there is no reason not to take advantage of communication and information technology that we have today to just be a little bit more proactive in shaping the loads. I just wanted to include a visual here. This is in your neighborhood across the bay. This is from PG&E's interconnection maps of city of Palo Alto is not on there but just to visualize sort of the geography of distribution circuits where the pink line that is transmission the little red triangles are substations from which distribution circuits come and those are the dark blue ones are older four kilovolt, four KV circuits and then the light blue ones are 12 KV circuits. So the level of granularity that is available today for managing rotating outages for instance is sort of on the scale of these large clusters. And if there is a hospital or some other critical load somewhere on that circuit that means all of these customers are exempt from participating in the rotating blackouts. So what that meant for Texas was that that the rolling blackouts were limited to a very small number of customers who then had their power out for days on end. There wasn't the granularity in sharing that but having had this visual I wanna talk about what I think technically really is sort of the necessary next step and we have the technology to do this and that is to enable parts of the grid or sections of the grid to operate as power islands when that is apropos when the main grid is out or when it is not appropriate to try and send power for example during a PSPS event the public safety power shut off if for example a transmission line is impacted by wildfire hazards that it's still possible to energize certain pockets of distribution systems if you have enough local generation and load that you can balance and obviously the local storage is extremely helpful with that. So you could call it micro grids I like calling it balanced clusters without sort of being agnostic to the regulatory or institutional label like who owns this under what rules are they managing this just from a physical standpoint what you need is enough kilowatts or megawatts to be balanced in real time to manage frequency and voltage on the island and that can be operated independently or it can be reconnected to the grid when that is apropos. Now conventionally we know of micro grids today those are owned by essentially one party it would be one customer, one piece of property it may be a corporate campus, a university campus a large building, I'm thinking of hospitals there are places that have built micro grids that can be grid connected at some times and islanded at other times that may span multiple buildings but this has really been confined to a single parcel of property or a single owner and the innovation that we're trying to push with the Oakland ECO block is that the cluster of electrical demand and supply can extend across property lines it can be a community it can go across multiple residential properties and when you start to think about what that means to aggregate people together in a community like that you realize really this is a great opportunity to do a number of things so beyond just being an electric power island preparing to meet the criteria for that really is an opportunity to start with energy efficiency in particular if you're whether you're doing new construction or retrofitting existing buildings you want to start by reducing any unnecessary energy use it's the opportunity to electrify to switch over from natural gas to anticipate that there will be I mean in California with our EV mandate there will be a lot of new EVs that will require in many cases upgrading the distribution system so now is a really good time to think about future-proofing our distribution systems as we are thinking about adding these new capabilities accounting for the fact that we will have to have EV charging in addition to the existing electrical loads but this is also a great opportunity to improve indoor air quality efficiency, better insulation means better health and comfort coming back to the survivability I think many deaths would have been preventable in the case of the Texas blackout simply if homes had been better insulated and people wouldn't have to be shivering in the cold when there was no electric heating available because and conversely in the summer people during power outages die from heat stroke and having better thermally performing buildings is really a health and safety issue as much as it's a comfort issue then there's an opportunity here for community ownership of assets being able to make investments that an individual household might not be able to make it's easy for us to think about well, I can get solar on the roof and I can get a Tesla power wall and then my house can operate as its own power island not everyone can afford that not everyone can qualify for a loan to buy that but what the EcoBlock project is looking at is ways to facilitate that financing and providing access to those technologies and then to make the financial bottom line pencil out really what we would like to do is also for these balanced clusters or controllable micro grids to be able during blue sky conditions to support the grid and perhaps help the utility with problems related to integrating uncontrolled EV chargers uncontrolled solar generation to essentially be a good neighbor because with the micro grid control capability comes the ability to be of service but the vision here really is to think about the electric power and energy as one piece of a larger picture of greening our cities or making communities more resilient and sustainable and that may cut across many areas it may include water management, transportation as I mentioned, it could involve community gardens food growing, managing micro climates and a lot of it is about people working together. So the hypothesis we're testing in the EcoBlock project is that there is money to be saved by doing many of these things together. The other part of the hypothesis is really about scale and it's that doing retrofits in particular may be easiest to do not at the individual house level but at the level of tens of homes where a contractor can swoop in and basically do all the houses on the block where a section of a distribution circuit can be modified so that it can operate as an island where you have a number of people that still recognize and know each other and can work together in a community and you get some of that aggregation of the load diversity from among these neighbors to sort of make a meaningful contribution to shaping the net load on the grid something on the order of 100 kilowatts or hundreds of kilowatts. And the argument here is that at the individual person or home level, it's sort of too small to make a difference. There's some economies of scale but then when you scale up to the level of a city that gets to be more complicated and there's more people involved than can really work with each other individually. And I wanna anticipate sort of the question about how do you retrofit a block in a way that can actually pay for itself? And I'll say a little bit more about that but I wanna preempt the question of how can this be profitable to say we know that there are some public funds that have to be spent on climate resilience and equity. Who knows what will come out of the spending plans in Washington DC right now that are being negotiated? Who knows what's in and what's out of the package? But ultimately there will have to be some amount of public spending on reaching these goals. And really the question is not can these improvements in decarbonization and resilience and equity can these improvements be profitable or be immediately paying for themselves in strictly financial terms? The question really is where do you get the most bang for the buck? If you're going to be spending public funds where do you get the best return on that investment? How can you leverage investments to make communities healthier and safer and decarbonize them at the same time? So the hypothesis is that if you are going to be innovating infrastructure anyway and I believe that's not going to be a choice that will be an absolute necessity then maybe we can do all these things together most cost effectively that is to introduce islanding capability at a local level which then means being able to shed that load on any given day if we need to and have that corner of the distribution circuit be on their own. Innovating, beefing up the distribution system so that it can accommodate the solar PV and the EVs that will have to be installed to meet our targets to basically go beyond what now are the feeder hosting capacity limits for those new loads and that new generation and then ultimately retiring the natural gas infrastructure because to get to zero net carbon, we will have to do that. So again, we can't think of that as something that's going to be profitable. It is an investment that will need to be made in our collective future to really address the climate problem that we're confronted with inevitably but asking how can we make these adjustments together in a way that we get the most per dollar spent? So finally here I'm coming around to the title slide for the EcoBlock project. It's funded by the California Energy Commission as part of the electric program investment charge or EPIC program. We are in the middle of phase two which is to actually build it. Phase one was just to talk about how cool it would be and whether it might be feasible. So there were a number of projects in this advanced energy communities program with an EPIC of which four ended up getting funded to actually build their proposed microgrid solution. And the Oakland EcoBlock again is focused on doing retrofits using older housing stock with regular folks not particularly high income on a city block. And I can talk about how we identified the block that we ended up working with. They self-nominated and we looked for certain qualities in choosing the block that's in the Fruitvale neighborhood of Oakland. So we are building a 100% solar microgrid for roughly 30 some electric customers or individual units including some single family homes, some apartments. We're developing new legal and financial structures so that this community association which now actually is incorporated and they have a board of directors and they will own the assets which in this case the California Energy Commission funds are paying for the solar PV in the battery and the microgrid controller and all that. But ultimately the replication model is for that association to get a loan and hopefully a low interest loan that they can together finance and build this. And that is what we hope will bring these options and resilient power into the range where lower income electric customers in neighborhoods can afford it. And our goal eventually is to have a recipe for how the eco block can be replicated elsewhere. I wanna call out our two most important partners on this. One is the city of Oakland that has been amazing to work with and is actually putting up a lot of their own staff time and effort to work with us on how does a project like this get permitted? We have to do some work on the street, on the electrical infrastructure. How does one go about doing these things efficiently? PG&E has been working with us on the electrical infrastructure and our model approach for how to execute this microgrid in phase two is actually different from what was studied as an option in phase one, which I'll say a little bit more about, but PG&E has been working with us both on the technical side and on the tariff side because conventionally there was no such thing as a multi-customer microgrid. So there's a lot of different disciplines that go into this just, you know, the quick overview here, energy, community, urban planning, design and construction, business and finance, legal, water, mobility. And just a few pictures, one of the things we're doing on the block and I will say we're not doing as much on the water side as we had hoped to do, but it's a start, we're doing energy efficiency and some stormwater management on the block, mitigating runoff and essentially beautifying the street a little bit. This particular street is up against the creek and so there's sort of a nice integration that we hope to do. We have done energy and water audits in the participating homes and are planning to go in and do the actual physical construction work starting early next year. All of this was delayed a bit by COVID but were persevering and trucking along. So for the microgrid design, the original vision was to have DC coupled solar PV and batteries all interconnected with a DC infrastructure that was going to be separately owned by the association and separate from PG&E. And our regulatory argument, which we'll come back to is that this should qualify under the own use provision of California public utilities code section 218 that says, well, you can't sell power across a property line or if you do, you have to become a regulated utility and you'd be competing with PG&E here. So now our interpretation is that a homeowners association for example, should be able to do that because it's a nonprofit and it's for the shared use of the participants. However, that is a longer legal regulatory argument and which we haven't won yet. But meanwhile PG&E is actually very interested in exploring microgrid options because their thinking has evolved a lot since we've had these horrendous wildfire seasons. They've had to do their PSPS and they're also looking at areas where they're serving customers that they might prefer to serve in a standalone microgrid way than to have to keep lines energized that run through high hazard fire areas through the forest for many miles. So PG&E is actually very keen to experiment with different types of microgrids. And in this case, we are under their community microgrid enablement tariff that will be sort of the regulatory umbrella under which the eco block becomes legal to function as an entity to basically allow the eco block to operate as an island although PG&E distribution operators will have control over the connect disconnect switch. But the assets including generation and storage and microgrid controller will be operated and controlled by the association or that is a contractor. HALA technologies will be doing the actual physical operation of management of it but PG&E continues to own the distribution wires and transformers which means that PG&E still has the liability and the responsibility to come fix something if it gets damaged and so forth. And the insurance is actually a sticky point. Carrying liability insurance for power distribution equipment is a very expensive thing that could easily be prohibitive for a community like this. So here's a view of the block and we don't want to advertise the exact location really to protect the privacy of the participants for the time being. So I'm not advertising the name of the street it is in the Fruitvale neighborhood. And as you can sort of get the feel for it it's the older homes you don't see on this picture an apartment building and a small commercial corner store. So the microgrid physically will combine rooftop solar essentially as many kilowatts as will fit on every roof and those assets are shared and every solar array just contributes as much as it can into that distribution circuit there will be some smaller batteries at the home level and one larger community scale battery on the 100 kilowatt scale with a grid forming inverter that's sort of the main resource that will have the ability to black start that microgrid. I already mentioned we start with efficiency retrofits and the intelligent control is a huge part because in order to save some money on the sizing of these assets and to be confident in knowing that this microgrid could really operate indefinitely even in the winter we need the ability to prioritize loads and to shut stuff off that's not essential depending on how long we think we're gonna be out of power what's the state of charge of the batteries how much sun do we have right now? There needs to be an intelligent process for prioritizing that at the individual house level and at the block level. So the microgrid controller will be able to do that. Now electrical engineering is easy compared to the challenge of governance, right? Seven different languages are spoken on the block. There are a lot of different expectations and views of how people should work together how the community should be relating to the utility company how the participants should relate to each other their financial stakes and ultimately it's the property owners who are becoming part of the association who are putting a piece of their property value on the line with their commitment. And that's a big ask. There's a lot of trust that we're asking for and really the participants have to trust each other. There's obviously protections built into the bylaws of the association and into the contract that or the agreement that each participant ultimately signs that nobody can really get screwed over but there has to be really the bottom line of trust that people are doing this in good faith because it hasn't been done before like this. We're hoping that with the fifth or 10th or 100th eco block, it will be much easier to point to the previous examples and say, well, we're organizing ourselves like this other block has done and here's the blueprint for the governance process. But really a key goal in our research project has been to really cultivate transparency and make sure that everybody understands what's happening including the things that we as researchers don't know yet. I mentioned the PUC section 218, right? We believe that and Dan Kamen and I authored a white paper that was shared with the Public Utilities Commission. We believe that the own use exemption should apply to cases like this. It should be permissible for a neighbor to share power with their neighbor, for example, to cite a battery on one property that's being charged with PV from across the fence. There is a working group that is looking at these issues. Also, the tariff question, how do you design the rules and the rates to ensure that there isn't any unfair sort of cross-subsidization among different utility customer types or customer classes but to make this type of resilience accessible to all? So I think that another huge challenge here is really working with existing homes and an existing community of people that just happen to live on the same street as opposed to starting from scratch with some intentional community or new development where, yeah, we can build a new green sustainable neighborhood and attract the kinds of people who want to live there and work together as a community that would be an entirely different thing to do than to just grab these 33 households and say, okay, we're now going to work together and you'll get to know each other and we'll translate where we have to translate where people don't speak each other's language but we're going to make this effort because ultimately, and I do think this is ultimately in human nature to want to support each other, especially when there is a kind of physical emergency that we're all facing in the same way. I have this personal memory of really getting to know my neighbors in San Francisco after the Loma Prieta earthquake in 1989 where it suddenly became very clear that, hey, we're here to help each other and make sure everyone is okay. I have also a personal experience with a group of neighbors in the high Sierra where we have shared water resources and lo and behold, people from different political parties and people with very different cultural backgrounds can come together and work together with the understanding that we all need drinking water and we all want each other to have that and to be safe. So I'm actually very optimistic that we can challenge everything that's good in human nature, that we can channel that to work cooperatively, to deal with extreme weather events and emergency kinds of scenarios but we have to facilitate that process and create an infrastructure that's set up for that that allows people to help each other. So right now we are, as I said, in the middle of phase two which is to build the project. We're designing everything to be specific to the site but also keeping notes, keeping close track of what decisions we made and why and how for a different block, for example, in a different climate zone with different building stock, with different solar resource on a different distribution circuit under different circumstances might have opted for different solutions at different points along the way. So that will be our eco block handbook and then we hope to scale this up to have a recipe book for how to adapt this to other blocks and be able ultimately to make a business case that's good enough for at least in a non-profit way and perhaps with some public funding to go into this hopefully ultimately to have a positive cash flow for projects like this to find a way to make this pencil out and we think there is absolutely the value proposition is absolutely there. There's some work that has to happen in the regulatory framework to facilitate this and things like the cost of insurance which reminds me of the analogy I think about is trying to get liability insurance for an indoor rock climbing gym which for the first climbing gym was really hard to do because the insurance companies thought, oh my goodness, indoor rock climbing that sounds terribly dangerous, people are gonna die, people are gonna get hurt, we're gonna charge you a huge premium and now of course there's an entire industry for this and it's a standard product, right? And there's standard safety procedures that are agreed upon in the field and I believe that multi-customer microgrids will evolve in that same way that clearly there are hazards, there's intrinsic hazards and risks that have to be managed but there will be standardized procedures and it will become a standard sort of insurance product like sure will ensure your community microgrid and there will be a standard financial product as a standard type of loan to take out for that. And the reason I've chosen this graphic here which illustrates blocks of buildings is because the block is really a unit that's replicated all across the country and all across the world. So if we have a cookie cutter solution that can work for a block it can be replicated very quickly in many places. Still, this is hard as I said, it requires trust, it requires cooperation amongst many parties including the technology developers, the electric utility, trust between the property owners and the renters and things as simple as, you know who saves money on their electric bill when the solar is producing and how does that get reflected in the dues payment for the community association? Will the landlord raise the rent for the tenant or should they be allowed to do that? How are the savings allocated amongst the participants? It requires really operating in good faith and with some ground rules in place. Requires cooperation with local government, with the city permitting and the regulatory agencies. So as I said in the beginning, it's a labor of love, it's a huge team working on this. And I think all of us are inspired by what this might be able to do really to accelerate our whole, our states, our countries and the world's movement in the direction toward climate mitigation, also unfortunately climate adaptation and resilience as well as equity. So I will stop here for Q and A. Any questions? Thank you first to share this whole project. And our question is, have we done projections on whether this pilot program would result in overall savings to PGE so bring benefits to other non-participating customers or is it gonna incur like more expenses? So there would be some cost shifting to other customers. A great question. So PG&E is very sensitive to the cost shifting issue. And I think I can say that there will absolutely not be any cost shifting to other customers to pay for all of this fund in particular because for this project, and we haven't penciled out the details of how much some of the upgrades, for instance, we need to do a transformer upgrade and a new transformer and how much that's gonna cost and some changes to what is now a single phase distribution lateral that for some stretch of it will have to be upgraded to three phase that will cost some money. And in this case, the project will pay for the bulk of that. In the long run and for scalability, I think those types of upgrades have to be rate-based in the sense that they're inevitable anyway, right? Because when there are upgrades for load growth those are generally speaking rate-based that is all the customers pay a share to allow for these upgrades. And what we're looking at in the big picture is that electricity may need to be more expensive than it already is, right? In order to reach our goals. Now, to reach our climate goals and our solar and EV integration goals there will have to be some upgrades done that cost money. At the societal level, I believe that investment is absolutely worth it because the alternative is much more expensive, right? We can't afford not to make those investments in our electric infrastructure, but how you allocate it sort of between micro grids and other customers, it stands to reason that those customers who benefit from increased reliability and resilience should have some burden to pay more for that privilege, right? And yet, on the other side, you can argue that, well, wait a minute, the utility has an obligation to serve. So if this is what it takes to serve reliably, some group of customers, for example, if because of the wildfire hazard conditions, if a micro grid is the most realistic way to serve that load, then the utility has to take that route and pay for it. So I think that's very fertile ground for detailed conversations. And I'm looking at what's on the shared screen. You've got two questions in the Q&A. Say again. Do you see the questions? I... Yes, so, you didn't tell the economics to have block-based rate making for electricity and gas to encourage the local solution. We currently have peanut buttered rate making, I love that. I don't know. And whenever I have the opportunity, focus on electrons rather than dollars, because I find the electrons a lot easier to understand. And I think that there are people much more qualified than me to talk about rate design and tariff design to find the best solution. I come at it really from the perspective of the societal level. And what is collectively the least cost way of making sure people are safe, period. And I think that... So my hunch is that there are actually some savings that can be had at the utility or system operator level, which I didn't mention previously. And that is that if we can guarantee that people will be safe and healthy when the macro grid goes out, that means potentially saving billions and billions of dollars that we now spend on the last lines of reliability. This is politically a very difficult point to discuss because nobody wants to be the person who sticks their neck out and goes on camera and says, yeah, the grid should become less reliable, right? Even though it makes sense, it might make sense from the societal point of view to say, as long as we make sure that the most vulnerable people are taken care of and that nobody will get hurt or will die when the macro grid goes out, wow, we can save a whole lot that we're now spending on generation reserves, for example. And or certain other system upgrades that we wouldn't need if we can actually depend on the local more granular management of load in both time and space. So I think there are savings there to be realized and people who are better steeped in the economics should take this up and discuss it. How will this affect existing utility companies facilities? How will the existing power plants be paid for? So the electric supply costs are really, I think a separate, quite separate bucket from the distribution charges, right?  And but it's a great question because it actually reminds me to say something that I neglected to say earlier. That is, I think there are potential savings that is distribution system benefits that can be realized from controllable loads and controllable clusters or microgrids for which there is not presently a tariff. And we've talked about this with PG&E and they're very collegial in talking these things through with us, but the reality is there isn't really a mechanism to capture and both validate and remunerate things like voltage services at the distribution level or being able to alleviate local overloads, basically helping defer or avoid other distribution system investments. I'm convinced that that value is there somewhere. There's not a mechanism to capture it. Now, as far as on the generation side, a microgrid that can be intermittently connected clearly is taking advantage of the rest of the grid as a battery and as a reliability resource should pay for that, right? It shouldn't just pay zero if it has the benefit of connecting to the grid. And I see this is coming back to the other generation, to the electric supply costs, which of course are mostly owned not by the utilities but by other generating resources. You know, there will be a demand for those resources because by the kilowatt hour or by the megawatt hour, those will still be less expensive. That's where I really see a complimentary approach between having the local distributed renewables and having the larger scale centralized solar and wind farms that can really produce megawatt hours less expensively because you can cite them in the best, you know, the best regions, but which happens to be not where people live because it's either too hot and sunny or too damn windy to live there. So I think that this sort of approach to do local generation to meet essential needs and then do the less expensive bulk energy production in a centralized or more remote way, basically thinking of that as the, you know, the benefit of accessing cheap bulk resources but also using that in the sense as a luxury to supply the convenience loads, that's what we're sizing the system for but we may not want to have to count on the network to be there, you know, without fail. For us. I don't know if that answered the question but I'm gonna go on to the next one. How are drought problems such as inadequate water supply for water utilities farming and water reservoir maintenance? Water and electricity are resources should be included in the rates. How can they be separated? I'm not quite sure. I understand the question. I mean, there's a couple of different intersections between water and electricity, you know, on the one hand, we use hydropower to produce electricity. So, and that might be referred to here. That is, I mean, last year has been a terrible year for hydro resource, right? A lot of the reservoirs are at historic lows. And thank goodness it's raining right now. Fingers crossed for a better season. But so that impacts electricity scarcity. And on the other hand, we also need electricity to pump drinking water, also to pump sewage, by the way. So water inflow and outflow is, you know, absolutely dependent on having some amount of electricity. But the pricing, you know, in my mind, those are quite separate. And I'm not sure maybe somebody else wants to help discuss this question, just to hit the last point on this screen desalination. Is necessary to discuss. Certainly desalination is one type of electricity use that I think is easy to shape around the availability of solar wind power. So it's a kind of load that, you know, doesn't have to run uninterrupted at a certain power level, but could be helped to basically flatten the overall demand profile. How did we present this complicated info to the community? Ha, multiple outreach sessions. Yes, indeed. So we started with hiring a professional community outreach coordinator who is an Oakland native. And basically her job is to know everyone and talk to everyone, both in City Hall and on the blocks in the neighborhood. And then we've had a number of sessions, some, you know, outdoors masks during COVID, out on the street with posters and, yeah, little pieces at a time. I think this would have been easier if we as researchers could have gone in from the beginning and known exactly what we're doing. But we've had to go in and say to people, okay, we're gonna do this project together. That is, we hope you'll do this project with us. We can't force you to, even after you've signed a letter that says, you know, you're interested. We don't know yet exactly what we're doing, but we're figuring it out with you. And that gives the participants the opportunity to shape the process. Certainly the, you know, the Board of Directors for the Community Association, just a group of folks on the block has taken the lead in self-organizing. And as researchers, we now sort of sit and listen in, but we're not driving that process, right? And they're actually taking it to the next level of communicating with their neighbors. But we also, we have a monthly newsletter. So we have a website that's public facing ecoblock.berkeley.edu. And we have a part of that website that's accessible to the participants only. And so that involves a lot of technical explanation. And, you know, sometimes talking about stuff more than once. And sometimes going back and saying, oops, we're sorry, we said something wrong, here it is corrected because we just figured this out. So thank you. Can I elaborate further on the EV charging? Right. So the details of the pricing are not yet worked out. The basic approach is that there will be one or two vehicles that will be accessible to the participants to borrow and then come back and charge from the microgrid. The goal is that this will allow people to, some of the households to get rid of their second car. What we've learned, and these are things that you learn when you actually do a project on the ground is that the super hard currency on the block is parking spaces. And we, you will like incite a mini war amongst neighbors if you start to take away parking spaces. So actually the toughest issue on the EV front isn't the pricing, isn't the low shaping or the duck curve. It's the question of, well, do you dedicate a parking space to the EV? And how do you tell everybody else that there's now one less spot for parking any other car? We have looked at vehicle to grid two way charging, probably too complicated for what we're doing here. It's a possible resource. I think that would be the way to go if there were a larger number of EVs. And that's, I think, one of those topics that we're like, well, let's do it next time. Let's scroll down. I don't know if that was all. I think that's the last question. Great. Thank you so much. That was great. Thank you.


We are happy to see you here. And we're going to talk about simplified, inner and outer cloud-native developer loops. We're going to talk about developer experience. We're going to talk about some tools. And we're going to show you the live demos of how we think the inner loop and outer loop could work for some environments. I hope you're going to enjoy that. Why are we talking about this in general? It's because we want to talk about productivity. Developer time is a very, very expensive resource. And the more productive you are, the better it is for the business. So you save money, which is good. We all would like to save money. Yes, we all would like to get more money and improve the margins and everything. So how do we waste time? One of the most in an abstract sense biggest bottlenecks is when we do things again and again, but they're slightly different. And we need to resolve the problem again and again and again, but we cannot reuse the existing solutions. And we need to change small things. And we get the tooling that doesn't fit us, that we maybe are not fully embracing for working with it. And that is not great. I think that's called reinventing the wheel. Yes. When reinventing the wheel is one of the biggest time things. So today, we would like to look at the workflows for your typical DevOps pipeline for your software development cycle in a loop from code going to production and see how some tools interact and how you can enable maybe better application developer experiences without sacrificing the needs and your interest in production. Let's do it. My name is Aliok Shalayev. I work at Docker as a developer relations person. And I work on the test containers project. And this is what we're going to use later for the demos as well. You can find me by at Shalayev almost universally online. And if you are into that sort of thing, check out my YouTube channel. I do short videos with people. Is there any more fun facts on those videos? Maybe I'll check them out. Hi, everyone. Thanks so much for coming. My name is Alice Gibbons. I am head of customer success at Diagrid. If you haven't heard of us, we enable tools and APIs for developers, building distributed systems. And yeah, we are big supporters of the Dapper project, which I'm going to talk about today and how it enables productivity. So as Oleg mentioned, how do we enable developer productivity? Developer time equals money. You all saw the slide. You all believe it. Maybe you don't. Essentially, we did a survey and we surveyed over 150 developers and figured out if they were Dapper users or not. And 95% of them told us that Dapper saves developer time.  Essentially, this was a huge number of people that we surveyed in terms of large enterprises, all the way down to small and medium, all different industry verticals as well. Fintechs, we had a few other mid-sized retail companies. And the amount of time that it saves you is debatable, as you can see on this slide here. But we had stakeholders saying that around 30% of developer time was saved by enabling Dapper within your code. Yeah, so that's something I'm going to talk about today. And today we talk about Dapper, not just to introduce you to Dapper, but as an example of a production cloud native tool that you would use in your development loop. And we're going to see how it impacts our application developer experiences as well. Exactly. And during this talk, I'm going to be more the person that's on the production side. I'm going to be the person on the production side, and Oleg over here is going to be our developer side. Perfect. And we are going to work together in harmony. Hopefully. OK, so what is Dapper? I'm going to do a quick introduction just to get us all on the same page here. If you haven't heard of the project, it stands for the distributed application runtime. Essentially what it is is a set of APIs that you can access from your code to write distributed systems. And you're like, OK, what's a distributed system? We have a huge number of API services, or APIs like specifications to create patterns and applications for things like workflow, for things like message, publish and subscribe, for things like synchronous communication via service invocation. And there are a huge number of users that are using Dapper in production today. OK, how does this look? And you're like, that's cool. What does it actually look like? You can see on the side up here, we have a number of these APIs that you can access directly from your code using your language of choice. We have a number of SDKs within the project so you can build the Dapper client in natively to your code. Or you can literally use HTTP and GRPC native clients in your language of choice to reach out to these APIs. There also are a number of observability, security, and resiliency features kind of built in at that application layer. So if you want to do a little bit of shifting left and move some of your security features onto the application side, one of the things you can do is take advantage of some of the Dapper security features there. This is hosted really on any cloud or edge infrastructure. We see over 95% of people hosting this on Kubernetes. And hey, this is KubeCon. But there are ways that you can run this on virtual machines as well. And yeah, this is running as a side car model. I didn't mention that. So this is enabled through what are called Dapper components. The Dapper component model enables the access of infrastructure from your code directly using the side car. And it actually abstracts the code and the languages or the libraries away from your applications, making them super duper modular, super duper composable. And you can also shift the platform that you're running on. So for instance, you can see a huge number of infrastructure services on here, things like for publish and subscribe, maybe you're using Azure Service Bus, maybe you're using Redis, maybe you're using Kafka. Dapper doesn't care. You can just swap out what's called the Dapper component and keep your code exactly the same, running from development into production. As I did mention, Dapper typically does run as a side car pattern. There is your application code reaching out over local hosts to these APIs, again, your choice, whether this is HTTP or GRPC. And there are a huge number of, again, these APIs that you can reach out to. So you can kind of see on the side here, we have this local host endpoint, which is exposing our Dapper sidecar. And this is effectively my service invocation endpoint where I am invoking another service via the Dapper API. Maybe I'm publishing a message on my message broker. All things you might want to do while building a typical kind of distributed system. This is also really important for moving from development to production because your actual URLs do not change, right? You're running the Dapper client locally, you're reaching out over local host and then you're moving to production, running that as a sidecar, still calling over local host. So to illustrate our kind of end-to-end inner and outer developer loops, we're gonna use this service called the Pizza Service. Everyone loves pizza and essentially what we're gonna do here is we have a pizza store and this is a store that I'm gonna place orders in and I'm gonna order a couple pizzas and then it has multiple composable services, one for the kitchen service because how are pizzas made, they're made in the kitchen and then one for the delivery service to actually deliver that pizza to our end users. This is kind of a pretty standard, simple distributed system, right? We have a couple things going on. We have our Redis key value store where we're saving those orders in. We also have a message broker. We are publishing messages essentially from our kitchen and delivery services to let our customers know, hey, what's going on with my pizza delivery? If anyone's ever seen like the Domino's Pizza Tracker, think like that, but a super low quality version of it. What's cool about this, if I rename the boxes here, I can have any application. It could be your bank service. It could be your movie streaming catalog. It could be anything. This is your template for architecture. Exactly. And if you have a, one of these services, you are typically taking dependencies on your infrastructure directly within your code, right? So you can kind of see in my pizza store service here, I have both the Kafka and the Redis client because I am not only saving my orders in my state store, but I'm also receiving my message broker events to tell my customer, hey, what's going on with my pizza order? And so this is kind of the pizza service, how you would typically develop it. With Dapper, a lot of these dependencies are removed. So you actually can remove the SDK of Kafka and of Redis from your code. And you can actually just put in the Dapper client or the Dapper SDK, and then reach out directly to the Dapper APIs via their sidecars. We also have a number of features kind of built in here around observability, security, and resiliency. And from a operational perspective, since I'm the operational team, all we really have to do is install Dapper into the cluster as the platform team and then wire that available infrastructure. So platform teams really like Dapper because what they can do is actually create those Dapper component files and then hand those over to the developers with all the necessary specifications for them to connect to their underlying infrastructure. All right. Sounds great, don't need to reinvent the wheel there. Okay, yeah, no reinventing the wheel. So should we check this out? Yeah, yeah, so we have the pizza application and of course it's an actual application running in the Kubernetes cluster somewhere in a cloud or locally where Alice knows where it runs. There are three components that we have. The main one is the pizza store application. It's a Java application. How many of you speak Java? Very good, very good, because we were not fully sure but it's good that all of you, almost all of you read Java naturally. Amazing, thank you so much.  We're importing our Dapper client. This is how we're gonna reach out to our Dapper APIs via code on the top here. And then where am I using this Dapper client? We're gonna check this out. We're using this method in the state store method. And essentially what we're doing is we are saving our orders, our orders that we're getting in for our pizzas. We're gonna save those within the state store as a key value pair. And essentially you can see I have this. I'm instantiating the instance of the Dapper client and then I'm actually reaching out to my state store. All I really have in code is this state store name. And this is an environment variable and this is relating to my Dapper component file under the hood, which in this case I'm using a Redis key value store. But again, the nice thing about Dapper is that it's entirely abstracted from my code. So if I do like a control F and search for Redis, you can see I'm not getting any results. So no Redis within my code. On the right. Yes. So in production if I want to switch my database from Redis to say Postgres, I can do that without. Sure can. Very good. I love that. Amazing. And you can kind of see on the right here I have my component file. And this is the key things is I have this metadata name which is KB store. And I actually have a couple metadata details that are specifically reaching out to wherever that Redis cluster is stored. In this case I've deployed it as a Helm chart on my cluster. This could be like your hosted Redis in a cloud or a container on your local machine running locally. Again, the only thing that has to change going from development to production is these Dapper component files. So that's kind of how this is wired up from a Dapper perspective. If I hop over now into Kubernetes, I can see I'm just gonna check out what I have running. So this is a pretty standard Kubernetes cluster. I have a couple namespaces here. You can see I have my Dapper system namespace. That is gonna be my Dapper control plane that I am, that I'm gonna be keeping up and running for my Dapper sidecars. I also have Kafka and Redis which are my infrastructure providers which I care about because I'm on the platform operations team. If I look at my pods, one of the things I'm gonna notice is those three services that I was talking about earlier. So my pizza, kitchen and delivery services. And you'll notice a couple of these have multiple containers running. So I have, if I check out, let's look at the pizza kitchen deployment. So on the pizza kitchen deployment, I have a couple of these containers running. I have one of them is my Dapper sidecar as well as my pizza kitchen container image as well. And these two are super important because again that's gonna allow that connection to those Dapper APIs from within my code. Okay, and then I also have, last but not least, to prove everything is running. I also have my Dapper components which are again those components, CRDs, those custom resource definitions to reach out to my infrastructure. Okay. Does this work? I don't know. That's a great question. Okay, so what does this look like? So here is our pizza store. We have, thank you, we have the our KubeCon pizza store and it was just lunch but I'm still hungry and I'm a vegetarian, sorry Oleg. So I'm gonna get this vegan pepperoni and ice tea. Very good. I support it.  Yay, it's working in production, fantastic. So you can kinda see these events coming through to our UI. Again, this is wired up also through Dapper via publish and subscribe messaging and you can kinda see where these events are coming from. They are coming from the kitchen service. They're coming from the pizza store service as well as that delivery service and hey, you know, should be coming in a minute now. Yeah. So this demo application is of course like public, it sits somewhere on GitHub so if you want to tinker with it yourself you can absolutely use this very, very easily. So we do this and it works in Kubernetes. It works in my production cause why not? Yeah, I have my Kubernetes cluster, I have my things. So what about then the inner loop of our development process? As an application developer, do I like this setup? Production always wins. If something has to be done in production as a developer I kinda just say no, right? If my production system uses Dapper, I need to accommodate. So that means I need to leverage and learn and use the same tools that I would use for production to deal with my setup. So that as an application developer that very often doesn't bring me a lot of joy and a lot of productivity cause I need to learn new tools that I might not be familiar with. I need to work with my complexity of like running a Kubernetes cluster. I need to understand how to install things. I need to understand the sidecar model for the particular things. I need to understand how to debug that when stuff will go wrong and it will. And I will hit my head on the wood. I like a woodpecker. And then I also need to have some sort of resemblance of how I run this locally on my machine when I develop my applications. And then I also will need to have the same setup in the continuous integration environment because I'm going to run tests there. I need that setup also work with my application. So I need some Kubernetes clusters in my CI. And that is the complexity that the production choices bring to me as a developer. And this is not ideal cause it complicates your development process. It complicates your onboarding. Everyone on your team needs to figure out how to do this. New people joining your projects need to figure out how to do this. And in general, there are very many concerns that are not application development specific in this scenario. Can we do better? That's the question. And we pondered that and we tried that. And we figured out maybe a different setup. And the main premise of that setup is that my production systems are environment specific. And the tools to work with production systems are all sort of environment specific. In production, I work with clusters. I work with deployments. I roll out deployments. The application is a mere detail there because I work with infrastructure and so on. So if I, as a developer, need to understand this whole picture, there is a lot of moving building blocks there. As an application developer, do I actually work on all of that? Absolutely not. Most often than not, the actual things that I'm as an application developer working on are my applications. So my local development setup and my processes are very application-centric because I want to work with a single application. I want to run the test for that application. I want to implement changes in a particular application and then pass it into the CI and then make CI build an inversion of that. And then it will flow into the outer loop of the development naturally. But I don't want to deal with Kubernetes clusters. I mean, I can, because I'm a professional, but if I could avoid that, maybe this is a little bit easier for me. So, and this is where we can try and build it better. And developers love API. We know how to work with API. We know how to request things. If you are building any sort of developer-oriented tools, if you give developers API, they can build things for themselves. So if you give building blocks of the solution, developers will be like, we are very, very good at building things, right? We are doing this for a living. So, both Dapper has the API, which is very, very great. So it's easy to use in your development. And also, we would like to have an API-first solution for development. This is where Test Containers enters the picture. How many of you know Test Containers? Very good. How many of you knew Dapper? Oh, Test Containers wins. Very good. Application developers in the whole. Right, so, for people who are not familiar, Test Containers are open source libraries and they give you a programmatic API to do things with containers. It gives you API to configure containers, manage the lifecycle, configure services in containers, perform operations with containers. And it's all in your favorite programming language that there are multiple implementations of those. We target pre-production use cases and we try to build in things that are excellent for integration tests on local development setups to kind of nudge you on the actual good paths so you would do things easier and those things would be more correct and less correct things you still can do because this is just a generic API, but it's a little bit harder and maybe the golden pass doesn't lead there. Test Containers gives you like a large ecosystem of modules which are predefined abstractions, which is very easy to do because this is just libraries in your favorite programming language. So, as an end user, you can run particular technologies in containers like databases or message brokers or dapper without actually knowing how to run them like yourself using the low level API or container configuration. We give you API to configure things easily, lifecycle and also the convenience API to wire your application to use those services in containers very, very naturally. In a nutshell, there are different implementation languages and it allows your tests or your application lifecycle create environments in which your application wants to run, which is exactly what we want here, right? We want to create a complex environment with dapper sidecars and potentially Redis, potentially Kafka service running there, potentially other tools in our local environment or in our CI without dealing with third party provided Kubernetes clusters or anything. So, this is what test containers does really, really well. Here's an example how I would run the Postgres database in my Java code. I would specify that I want a Postgres SQL container abstraction. I will specify that I want to initialize it with a particular schema for the given Docker image that I would use. I will start it and then I can get the JDBC URL which is the connection URL which I will pass to my application and my application will know where to find the database. Test containers will take care of the resource cleanup after the fact so you don't need to clean up manually even if something goes wrong and your tests or your application process will crash and it burn and your Docker environment where we run containers will be abandoned. We will still do the cleanup so to ensure that you have this repeatable environment that you can run your tests or run your application again and again and again and you always will get an ephemeral environment and you will never connect to this stale instance of a cluster that is running on a cloud somewhere with 319 days of the Dapper system component installed. I didn't think you noticed that. Nobody notices that. And then you get this fresh environment. So, as an application developer it takes away a lot of concerns for how to operationalize. Yes, how to make operations for your integration tests. You can do the same thing in for example, GoLang which is the base primitives of the abstractions are the same across all test container implementations. I can say run the container that is a Postgres initializes in an appropriate way. Wait until a certain log line will appear in my container and then get the connection string that they can pass to my tests on my application to connect to that particular Postgres. It's a very flexible approach. Can we see it? Yeah, we can see it. We can see it. We will not see the Postgres because this session is not about Postgreses. This session is about Dapper. So, test containers team offers the whole experience and the multiple tools for you. So there are open source libraries, your test and there's desktop free application and then the cloud backend if you want to run containers in the cloud and you don't want to run them locally in your Docker environment. So, this is how we do it. Yeah, let's look at the pizza application. What am I doing? Pizza application, come here. Pizza, right? This is the same pizza application. It's a pizza store application and if we look at the store method or store, store, store. This is the code that Alice showed you as well, right? This is exactly the same application. It comes from the same repository. This is just I have it locally, right? So, now to run this application because it depends on Dapper being available in my environment, I need to provide it with a Dapper system somehow. How do I do this? I would run Dapper in a container and I would do that with test containers. So, for that, this is a Java application. There are very good integration between test containers and frameworks in the Java ecosystem but you can do the same in any language is just for a particular project with just a couple of lines of code where you can say like, if development is enabled, do the containers in production? Don't do containers, right? So, I have my pizza store application entry point under my test and it will run in my actual pizza store application but augment that with the configuration for containers. And in my containers configuration, what I see, I see a Dapper container. The Dapper container implementation comes from a library. It comes from the jar file that I add to my project and it encodes how to run Dapper in the container. As an end user, as an end developer, I don't need to know all these details because the community has already provided us with the implementation for this. As an end user, I just need to say, give me a Dapper container, make this my application name and application is gonna run on port 8080. I will not just have the application called the Dapper Sidecar, which is not the Sidecar, but the container now, but also I will expose my application to the Dapper Demon, Dapper service. So if I need to receive some messages, it will also flow back into my application. So when I'm receiving those pizza events, my pizza store is gonna be receiving it that way. When kitchen is done with the pizza, the event can flow back. I expose my host on the host machine for test containers, for the port where I will run my machine. And I also configure my application to run where my Dapper service is running. It will run in the Docker container. It can have dynamic configuration. It might not run on the local host. It might run on a remote Docker Demon. So I need to programmatically tell my application. That is where your Dapper is running. The same way I would do that for a database saying like, oh, look, application, here's your database. So, right, perfect. I do that. I also create the mock of my kitchen service because I don't want to run more than a single service in my application because I want to ensure that if something breaks in my tests, it's only because of my actual application is at fault, so I can fix it. So I can run my application here. Can I run this? I think it runs. It runs. Test containers will find my Docker environment. It will, if necessary, pull the containers. It will start the containers. So here, we can see that we are pulling the Dapper latest. I'm using Test Containers Cloud. So it got, I closed the machine, and it got a new isolated Docker VM in the cloud. So it will pull the images for a second here. But it will start them, and then my application will run normally. And as an application developer, I don't need to know the details of Dapper. And the more of that, anyone on my project, I will check, it's all in the code. The whole configuration is in the code. So it will go in my version control, and then everyone who will pull it will have the same thing. So to run the application in the environment, they just need to run it. They don't need to mess with configuration or understand any details. Anyone can do this. Anyone can be an application developer nowadays, which is a good thing, right? It's a learning curve is getting flattened, right? So this is how I did it. And I can do this the same for the tests. I can have my tests run, use the same configuration. So it has the same containers configuration. And it can have high level integration tests that will exercise my application in a way that I would use it in production, and I would expect things. And the test here is rather simplistic, but well, we are not testing the pizza application logic here, right? We are trying to figure out how the environment is working. You can see that I have my three containers there, my mocking for the kitchen service, my dapper and my SSHD, which is exposing my host there. And it just works just like that. Amazing. I think I can run this and we'll see the blue checkbox. Lovely checkboxes. So now we have this local experience that I, as an application developer, love because this is what I would otherwise do manually as well, right? That I would do for other applications that are not working with cloud native technologies that don't depend on dapper or Kubernetes services. I'm just, I'm a happy bunny with this. So what next? Amazing. I mean, that was a great, you know, a great example. Oh, that was your demo. That was a great example of, you know, our local developer experience, again, with dapper and test containers, kind of working together and being able to install all those dependencies as a developer. That's amazing for me. It actually works, right? Like the application is actually running. Local inner dev loop, check. Okay, but you know, there's an eight in our dev ops even though it just got moved slightly. There's still is an eight and we have to make these work harmoniously together, right? So I've kind of showed this application running in production. Oleg has showed it running locally. We want to do a full end-to-end, you know, push deploy test and see how this all kind of works together. Yeah. Yeah, so how, how do we do that? How do we, you know, enable kind of this, this working together? We need to kind of figure out this plan in this, in this release step. I think of this, of this dev ops pipeline. We need to introduce and change and we're going to be, we're going to be responsible and when changing our things, we're going to, we're going to, we're not going to just throw it in production and just let other people deal with that, but we're going to release it properly behind a feature flag. And this, and for this we're going to use open feature. An open feature is a open source project that does standardization of feature flags. It allows you to use a standard API and then the, any sort of implementation of a flag provider. So in this case, we're actually using flag D, but it allows you to take advantage of really any different flag evaluation providers in this case, in, you know, in that I want to take advantage of. Yeah. So for now, we're going to try open feature and flag D and we're going to release the production. Right. So in my local experience, I'm going to use it the same. I'm going to run flag D in my test containers environment in my, in my Docker environment using test containers. And we're going to try to introduce a change and see how it works. This is the most dynamic part of the demo. So please continue breathing. Don't be super stressed, but we're going to do some coding here. I added the open feature already libraries to the thing and I will add the configuration for my open feature beans for my spring application. They get some configuration that I get from my properties and I enable the open feature bean that I connect to my flag D flag service provider and then they create the open feature API client. Now in my application, in my application at the very top, what they can do, I can say like give me the client for the open feature and then I would have a very simplistic sort of code that says, that says I want to look at the flag called V2 enabled and then pass it to the, to the property. Right. And here we're going to check a value and essentially change a color. Cause you know, I want to test out, I'm the production lady and I want to test out a couple of different things in production here, but I don't want to put extra onus on to my developers for having to push them through an entirely new cycle of development. I want to test out maybe I want to do AB testing, maybe I want to show this application to different folks and see how they respond. Right. So I get this feature flag, I'm going to inject my open API client into my pizza application, right? And so this, everything will compile. Will it work if I run it locally? Who thinks it will work? Who thinks it will crash and not work? It will not work of course, cause there is no flag D provider. It will be like, where's my flag D? I need my flag. I can't work. We need our feature of value or flag evaluation provider. So the same thing I do with all containers. I can just, just get this. This is the best part of development. Development by uncommenting code is the best. It almost never fails. However, in my container configuration, I say I would like a new generic container for the flag D, flag D Docker image. And then I expose the protocol ports, I override the commands and they pass my configuration into the container as a string and that will be my flag definition. So here is V2 enabled, options true and false. And currently this is true. So when I run this, let me run this my application again. I think if I run this, it will pull and run my flag D as well on top of everything. If we didn't forget any changes and then I can see my application. There it's pulling that flag D latest container. I see it right in there as the dependency. Yeah, oh, yeah, yeah, yeah. It pulled the container. It started the container. The container is being configured. My spring application picks the configuration. So when I go to my pizza store, you can see it's gloriously green. Wow. So I will finish my loop here with just building say a Docker image of my container with changes that I will just push into the registry and then the ops part of the team can pick it up and deploy that into production. Awesome. And then since we only have two minutes, I'm gonna show this really quickly in production. And so essentially what we have is, as you can see what Oleg did is we have, you know, our flag D provider running locally. We also have this deployed on Kubernetes with the open feature operator system. So this allows us to, you know, provide our different flags via CRDs. And we have, if I pop open my open feature configuration, what I have is, you'll notice it's the very same configuration that Oleg ran locally, but we have it within this feature flag CRD that is specifically for open feature. And we have this flag D source. This is just telling us, hey, we're providing these flags from Kubernetes in this case, and these are our pizza flags. This is, you know, built into our pizza store with a couple annotations. So, you know, our pizza store service, which is the one that we wanna change is pulling from these two CRDs. And essentially it'll allow us to effectively make these changes live and see how they get updated. So, you know, this was false. Let's update this guy to true and see if we can actually make this change color. So I'm gonna apply that guy, all right, configured, and then I'm gonna head back over to that UI. What do we think? Yellow or green? Yellow or green? Green. Yay. Woo-hoo. Okay, awesome. And that is kind of what we wanted to show. The, yeah, so kind of just wrapping up here, what we did show kind of an end-to-end developer loop. We talked about some consistent APIs, both from the developer perspective with Dapper, as well as the APIs that test containers gives you to develop with with your local dev experience. You know, simple inner and outer development loops, and then, you know, we're going to begin these changes with open feature and feature flagging, using best practices in production. Yeah, I think we showed a simple but nice iteration of the whole development loop into production. We made sure that my application developer experience is what I usually do and like and know how to work with. And we saved developer time. Yeah, hopefully. And then it's all was enabled by the combination of the projects. So you can build similar things yourself. And then there are a number of things that you can read online about this and try the application yourself. Yes, try it out yourself. We have this published online. And then, you know, if you had any other questions about this session, we're just out of time right now, but please come up, ask us at the end, or, you know, come to our booths at the Docker and Diagon booths this week. Thanks so much for coming. Thank you. If you... KubeCon asks us to ask for feedback. If you have great feedback, please fill the form by the QR code. If you think we can improve something, just come tell us personally. Thank you.


They said they were having problems with the furnace and of course they told me this after I turned the power off of it. We are on a no-heat call here and it's for the lobby area. So they said they were having problems with the furnace and of course they told me this after I turned the power off of it. I was here to do a filter change and do the PM so there was probably a code there didn't know about it but because the fans was running it's kind of hard to pull these filters out of there and over here on the ductwork. It just turned it back on and of course the draft motor is kicking on.  Pressure switch. A lot of surprise. Alright let's verify that that truly is the issue and see if we got a trap issue or what's going on. I went ahead and just checked across the switch and it is open so we did verify that.  Let's see if Aaron takes got an issue so let's pull this cover off and see if it changes its tune. Okay I took the cover off and it immediately started. What a surprise. The pipes go up through the ceiling there. Amazing it left that open.  You can, oh wow, that thing's really pulling on that hard. Yeah we've got a restriction of some sort in there. Could be, this does not flex a whole lot. Old three inch pipe don't move a whole lot. Go outside, we might be able to blow through this thing. Vacuum it out hard to say. Now if this was after hours and or snows on the roof or whatever you could always pop off the side there of the other side of the box there because you can pull it in from the air from either side and have it get them going or you know I wouldn't want to leave the cover off like I'm doing but something's in that pipe and that stuff that we got there's just remnants of probably what is probably in there so let's go and see if we can see something outside. From what I'm seeing here that looks like the old flue pipe for the old furnace and that black thing is either a vent for the plumbing or it's the vent for the furnace. Well lucky here. There it is. There it is. So at least it's three inches all the way up and out. I would speculate we should be able to blow anything in there that's in there out of there. Might even be able to run a fish tape through it and lucky there there's evidence all over the place like I always say you got to build the case. See that right there? That's the bird's nest.  Let's see if we can get that blown out. I've got my little Milwaukee tool. Let's see if it'll work. It's three inch diameter is pretty big so it may not do much. We just tried blowing through there stuck it in there wrap your hand around it. Let's see if anything came out. Chances are probably didn't do anything. I don't know if that was there or not. Some fluffy stuff so I don't know if that's it or not. It's got to go inside and check and see if it works now. I'm gonna run some fish tape through there. I've got just about as much as I can get in there but they've been such a big diameter pipe. It tends to want to bend and stuff and not get all the way in there but I'm hitting against something so I've gotten at least all the way up. I would suspect into the attic at least. I don't see the end of it up there. Let's go ahead and put it back together and see if the thing runs. I don't want to start chopping pipes apart because I don't have three inch couplings on my truck or out of town again. So there's hardly anything up here to get parts.  It's starting back over again. It's still feeling out. We could put the vacuum cleaner on the other side. That's a funny feeling. I can't remember if that plate there was removed last year. See there? It just came on. But you can't trust that because the unit is warm now. It's warm. It tends to fire off a lot easier. Alright, so the pipe's clear all the way down there on the exhaust to the first elbow and you can feel we got really good air on that. Now remember we took that cap off the other side. They glued this on which really makes it hard to do much with it unless that paint's just holding. I'm going to smack it a few times. I thought about even just drilling a hole in there to see if we can look down through. I got my borescope. Actually you know what? Good golly. I forgot all about it. I got a borescope now that's like way longer than what I ever thought it would need to be. That right dear locks will make it a lot easier. We can look in there and see what's going on. So let's get that thing out. Let's go take a look. Now I gotta admit this cable is a floppy noodle trying to get that through that three inch pipe. So we've got a straight and a completely out first which kind of sucks because it takes a lot to do it after it's been wound up this long. And that's why I really don't like having, I think you're 16 foot here, something like that. So we'll go ahead and get that shoved through there. This has got like a set of camera on the side, camera on the front which comes in great. If you want to look inside the engine looking different multiple different spots and it's got a light on that one and light on this side and multiple on the front. Alright you can't hardly see it but we're on the edge of a coupling there. I'm looking at the front and the side view at the same time. So we're cleaning up to there. Let's go from the inside out. Alright so the end unscrews. We put a magnet on it or they get this hook.  I went ahead and drilled a hole in there and put a piece of tape over it because it's an air intake. It's not like there's gonna be any water in it. I'm trying to go up this. If I can get through it great if not we're just gonna leave off the cap off the end. This place is not going to care whether or not they're pulling the air out of the inside of the building or not. There's plenty of combustion air in here. You've got holes through the wall. You've got a chimney going past up into the outside. Say the least we're gonna see if this will work. We were able to get through. If you look there on the bottom screen that says F for front you can see daylight out there. Turn the lights off. See the daylight. Flip it to front only view. Just hold the camera button and there's the forward-facing one so there's full-size picture. Go a little bit further on down along just about all the ends. We know it's visually open. We can check the pressure switch with the dual monometer. Generally I don't have pressure switch to go back that often at least not the carrier anyhow. And a little hook thing really comes in handy if you need to hook a wire in the wall or whatever. The link is down in the description down below if you want to support the channel. Okay so we went ahead and hooked up the monometer to it. We were tied in on both sides here. We got our pressure switch. Reading on top is what we got on pressure port number one. Pressure number two is 1.5 and our differential is 1.78. I do not believe carrier was really bad about not putting their pressure switch ratings on there on the switch themselves. So it just kicked off and it's acting a little bit rough but it is doing it. Pretty good. Now as you can see once it's got running it's already started jumping up quite a bit. It's because that warm air is lighter and it's freezing through a little bit quicker. But that's the most accurate way to do it is to put it on both sides of the pressure switch. I'm sure you guys are in need of that. Now that at least I know that the pipe is visually open I would feel better bringing some air from inside here. Let's go ahead and pull this off. It's a 2008 so it's just a matter of time until it's good bad. The one I'm using here is the dual port pressure monometer and pressure switch tester which is the SDMN6. I'll have a link down below on that. Pill piece doesn't sponsor me or anything as of yet. This isn't a really good one. This has a vacuum pump actually in it so that you can test the pressure switch and then it has continuity checker across here. So if you wanted to know if that switch is opening and closing or if you have a adjustable one you can actually set it up with this thing. So it does a really good job on that. I went ahead and cleaned that up. It's got some white tape over that. It's like I said none. There ain't nothing coming through it. We're going to go ahead and clean up the flame sensor. Check the igniter's resistance. Cleaner constate trap out. We've already changed our filters. Check fan blades and motor ampurals. Electrical connections. Then we're going to do the walking coolers and stuff like that. So it's going to wrap this one up guys. The only other new toy I've got here is another Phoenix light which this one's pretty cool. It is also a head, head deal. I'll see if I can make a link on it. It's kind of cool. It'll tell you if the battery's good. It's got a total of five different settings. The lumens on it is 1300. My other one here is 1000.  It's got a magnetic charger on the back and it's magnetic and then when you don't need it for that you can put it in your headset. Boom like that. We are on a no-heat call here and it's for the lobby area but you made it pretty easy to find which is kind of nice and we just got checking it over here and I've got a number three code which I'm not real sure what number three is. We're going to find out here in a second. It's an ICP Comfort Maker. Number three, pressure switch did not close or reopen. So that's what we got so far. So see whether or not we got a pressure switch problem or a draft motor problem. All these filters seem to be pretty dirty. Notice this one here. You can even get it out. Look a little shady. So that one don't look so good. This one here is packed nasty. These all get out and then these right here we're in there. Make a fair unit here. Heat recovery. Just pulled those out a second ago and they are just whistling Dixie. So we're gonna wash out those washable layers once we get the system back up and running. I went ahead and unhooked. Gee. Let's see if this thing kicks on here. Just finally came on. Draft motor is running. Hard to say. Looks like a pretty common simple vending system going on here. There we go. She gots the number three. All right let's go ahead and dig in a little deeper here. We got the filters washed out. Luckily they had a place out there. We could wash them out without having to get a hose out. So those are clean. Got her back in there. Yeah they don't have any filters so we're gonna have to get some. Let's go ahead and see what we got going on here. Generally what I'll do is clean my condensate trap out first and make sure all my ports here are clear. Seems to be clear. Got this one coming from there to here. I had some water in that one right there when I blew on it. It made the water noise. So with that being plugged up which probably means that that's not draining. Likely what's going on. I bet you we run it now. It'll probably run. And now when there goes to the draft motor right there. The one on the other side of the switch. Let's see what it does now. Well we're waiting on the other one. Let's see if we can get this one turned off. I don't know why I can't find the switch. You'd think they'd have a switch on there somewhere to make it easy but they don't. Guess that's as good of a switch as we get today. These filter racks these things are garbage. Total garbage. Yeah that's pretty bad. That still hasn't came on yet. Speed this up a touch.  Oh there it goes finally. You can usually hear it girdling in that trap if there's any issue with the trap. Concentrate trap. Look at that. There it goes.  Dirty flame sensor plug condensate trap. Igniter bad blower motor out draft capacitor. Temperature rise ain't right. Not enough to return air. It's just the same stuff over and over. Just does not make for very good videos up when you're trying to get a filter out.  Problem the installers don't crimp down these sharp edges. You've got a sharp edge here that'll get caught on the edge of the filter and you got a sharp edge up on the top. So we're going to squeeze that down work our way across. I'll either use these or I'll use my channel lock slash you know whatever. But that generally will allow you to be able to pull the filter out next time. That way you're not tearing it all apart trying to get it in and out. Especially if you just want to check it. We've got that out of there unfortunately. Had to tear the bottom of it off to get it out. What we'll end up doing is brushing off the filters till he can get the new ones. I hate leaving them out because there's always a good chance they may not get around to it or they forget. Then you have nothing in there. Then it starts looking like it already looks in there which is not good on it either. Let's go ahead and shut down this other one here and get this condensate trap pulled out of there. That's about the only thing that generally keeps it from draining properly. They kind of they make it easy to get to that screw and they put the pipe right through here. It's not the greatest location for it. Looking at this trap it's got a little bit of crud in it but not really that bad. So I wonder we've got some drainage issues inside that collector box possibly. I usually like to see this hose kind of drain towards it more which if you make a trap like say I did it like that that actually make it worse. So it may shorten that hose up a little bit or possibly pull it back. Yeah I mean no matter what we could shorten that up a little bit and get a little bit better.  It'd been a lot easier if we could have went out through the side and up but then you'd hit all that. So it cooled itself down. Let's go ahead and kill it here. Let's get that pulled out of there. What I find to be the best to get those clamps off is usually some wide lines like that with the crisscross pattern. Usually grabs right a hold of that no problem. Now they have a little port that you can see right there. It's on the other side too. See it right there. That'll be on the backside of this one. You can literally blow through the hose to see and you gotta be careful you don't want to blow the diaphragm out but you can hear it there. What that's supposed to do is break you know when it's pulling through this way it's supposed to pull anything that's in that hose out and back into the collector box so it doesn't plug up with crud which is kind of what it sounds like happened here. We can shorten that up a little bit. They leave these longer so that be able to move it around to different positions whether it be upflow, counterflow, whatever it is that it's being used for. But I don't know if that I mean it's been in for a little while. It's not like this just happened. Now you can see when you pull it through and lock them back into those pieces right there and there there's plenty of leftover here and you can see the hose can dangle down and see how that makes a trap. That's a lot of the issue a lot of times so let's kind of do it so they can't trap. So we need to take off about that much right there. I just trimmed that up. Let's push it on there.  But you can see we're not we're not pocketing now it's a nice solid slope coming down to it. I left it fairly tighter that way.  But that's pretty much all I think's wrong with this one. Now while we're here we're gonna go ahead and clean the flame sensor out and double check you know all the normal stuff I always check. Every heating call I clean the flame sensor, clean the constate trap, check the heat exchanger, make sure the flu pipes are properly pitched, you know make sure the fan blades are clean. If you know if I'm feeling frisky I might go ahead and check the capacitor on the motor if it's got one. You know it's just common things I do every call because you'll find a lot of stuff by just doing the normal things over and over. As much as I hate doing service checks and things like that I don't like callbacks. We don't get paid for our callbacks.  Now contrary to rip-off beliefs you can clean that. You don't have to replace it. I'm always been a big believer in the only way that things bad is if it got it so hot that it warped that metal or that porcelains broke. Otherwise you're just selling parts. So let's go ahead and get that cleaned up. I've said it a million times. I use my wire brush which is stainless steel. It's my preferred method and that's what I like to use. Use your dollar bill, rubbing alcohol, whatever else makes you happy but that's what I use. Here's what it looks like when we're done.  You can see the cleaning marks but there's no deep grooves. You're gonna have some when it gets that hot because it's constantly in the path. Lennox always seemed to be the best about not getting damaged. Carrier is not too bad but boy there's some other brands out there just bake them and they just tend to overheat. I don't know if they're not using stainless steel or what they're using but some of those just tend to really get damaged really easy when they get too hot. But there's that. Let's finish cleaning up our trap here. I'm gonna go down and rinse this out in the sink. It really don't look that bad.   There's no way to really truly get it cleaned out very good. I mean you could put a Q-tip through there possibly but it ain't really doing anything. I truly believe most of our problem there is with that hose the way it was. As much as I hate to do it like that I mean it's about all you can do. I brushed them off a couple times and then smack them against the pole right there and the fluffy stuff just goes flying. It's good enough to get them by until they get one. Hopefully they'll get it within a few days to no more than a week but that'd be good until then. We got that all shook out and just looking Jim dandy.   To put the side it actually has the piece on it on in there. So that's back up and gone got that one back in there. Let's go ahead and check this capacitor out make sure it's okay. It looks like it's rated for 35 which is a hog doggie. Let's see if we can get on that while it's in place. 35 so we're looking good there on that. No issues. This thing's only I think a couple years old looks like 17 2017 and looks like it started up 10 1 of 18 so 19 2021 three years old is all this is. Now do what you want but whatever I work on these especially in a counter flow like this I take them all to their maximum run time. Now this fan runs all the time on this but what happens is if there's any residual heat in there it'll come up to here and on these furnaces they do not have a limit but on the Linux's they had limits on the blowers they had you know all different limit switches and they would trip. It's a good thing to kind of remember and to do it can prevent a future call-back but this one here the only limits they've got on it is the primary there in the center and some rollouts there and there so we only got two rollouts. I've had some where they've got two in the front one on the top one on the side one on the side they're just rolling out and they must have been singing the song that day when they were putting them in. This is one of those times where that long extendable head is able to get into those nice tight spots that aren't ideally placed you can see we're at an angle there works out great because you can push her right back the way she was and she's a normal screwdriver again. All my tools that I use are down in the description underneath my kit anything you guys buy helps support the channel it goes to my Amazon account you guys still pay the same money but I get a couple pennies on each one of them so there you go there's that got that back on got to hook this green thing back here on top of the trap itself there we go and that goes up there to the top there we know that's clean and clear let's go ahead and start over which I hooked the G back up so the fan's gonna run let's see if she kicks on just revved up make sure that gas fells back on that's why you always cycle this thing last thing before you leave that way if you got anything turned off you're not to forget and run it so we're good on that we pretty much went through everything you need to check I mean the floor capacitor is not bad we've got the collector box blown out we found the water in there which is great because it gave us the right lead of where we're at we're not just guessing that that's what it was because as soon as we heard that we knew that that was blocking off the pressure switch suddenly do I have a bad pressure switch on the carriers it does happen so we're good on that went ahead and got their fresh air make up there taking care of and cleaned up that wasn't part of the deal but you're here you might as well just give them a little extra for their money same thing with this unit over here got the filter cleaned which I would have preferred to see it replaced but they don't have any here and we don't carry them on the truck which it looks like they've been doing says they're not the brand filters we use that's gonna wrap this one up guys I'll go and contact this on possibly with another video if you enjoyed it you want to see more like it make sure you hit the like button subscribe and check us out on Instagram and Facebook and until next time we will catch you on the next one