My name is Yoriko Kishimoto. I'm former mayor Palo Alto and I'm a member of the organizing committee here and I'd like to welcome everyone to the panel on local generation and control Does it matter and doesn't work? and So this is called the Silicon Valley Energy Summit and one of the I just wanted to start by reminding us that Silicon Valley is famous not just for our our global leadership in in technology But the structure and the culture that that got us here That that we broke the mold of kind of large established companies which had an overwhelming advantage in scale scale and market position because of our Entrepreneurship and because of our our unique culture for innovation So this panel is in a way a panel about a different model for managing and delivering energy And that is local control and local control and local generation Which are two different but related ideas and asking if if as a society We should be pursuing that paradigm shift more energetically We've heard from the lunch speaker that Governor Jerry Brown has been promoting since 1970s this idea of local and distributed energy and the goal of 12,000 megawatts of renewable power generated within local distribution grids So so far a panel we have three distinguished speakers from the front lines and so I'm going to introduce them briefly and to give you an idea of the flow and So I'm going to be first introducing Sean Sean Marshall to speak Sean is the founder and executive director of lean USA and that stands for local energy aggregation Network US which has a mission to expand and and support clean energy CCA's and CCA stands for community choice aggregation programs Sean was the mayor of Mill Valley in 2008 and she was one of the co-founders of the Marin energy Authority the first CCA in California and after her will be Mayor Yahweh Ye who is The the current mayor of Palo Alto and he has been our representative NCPA northern California Power agency and I know in his day job. He has had the job of many years of experience as assistant controller for I'm sorry city auditor for Oakland and before then San Francisco And he's going to be speaking to us as as As a publicly owned utility and I know there's a number of publicly owned utilities here and in the audience as well So I hope you guys speak up and then to balance these views from publicly owned and and the community choice aggregation Debra Wang is Going to be sharing her perspective as NRDC's California energy Program director and she is an expert on statewide policy and policy results Right and so before and she told me that before joining NRDC in 2002. She was Worked briefly at the California Energy Commission During that famous crisis. So that should take us to about halfway through So especially since we're kind of getting towards the end of the afternoon I'm trying to leave plenty of room for questions from from the floor so Please be going ahead and thinking about questions. So let me start by inviting up and please help me welcome Sean Good afternoon everybody I hope you guys are having a good day I first want to answer the questions posed which is does local control and local generation matter and You will not be surprised that I say with a resounding Enthusiasm yes. Yes. Yes both matter. I think we're going to see that they matter more and more as time goes on so what I'm going to talk to you today about if I can see this lovely screen is One of the newer methods in the state of California at least not new across the country to aggregating local energy at the the county and or regional level through which then to take over your portfolio your supply portfolio and have some choice and Control over the kind of supply that you're offering up your customers and that is called Community choice aggregation is the hideous name. So I apologize for that CCA think about it as like a big group on or Living social deal for electricity at the residential and small commercial level because that's what it is Alright now folks. Let's see. So what is CCA? I often find that This picture works very well for those of you. Well, first of all, I should ask how many of you are familiar with the CCA model Okay, all kinds of you. Okay, how many of you here are representative of public utilities? Okay, great. All right. Good. So I'll just run through this quickly So what we talk about up in Marine County is that CCA is the biggest change you'll never notice Because what's happening here is that the local governments in our case the the cities and towns of Marin Got together and decided to form a joint powers agency or through which to run the CCA to take over the energy supply We partner with the incumbent utility in our case. It's PG&E in Southern California. It's SDG&E so Cal Edison The utility is our partner. It was a shotgun marriage. It's getting better And they continue to provide customer billing customer support Pull and wire maintenance and so the end-use customer you doesn't actually experience anything different about your You know receipt receipt of energy or your billing But what you are getting in the case of Marin County is a default product that is a 50% Green and then up to a hundred percent opt up So it's a voluntary opt up to a hundred percent green product at rates that are comparable to PG&E's but that green underlying renewable portfolio that we've got is Twice and now pushing into three times as much as what PG&E is currently offering its customers We also explained CCA as a hybrid approach kind of a third option that lives between two vertically integrated approaches one of which is the Shareholder investor owned utility PG&E's of the world and the other is a full municipal utility Palo Alto for example City of Reading there are a number of them around the state in the end Personally, I think that the home run is having a full municipal utility and the ability to control not only your supply side But your distribution. Okay, what is the reality of that though in the current condition in this state? And frankly, why would a local government or group of governments want to? Bond for multiple millions if not billions of dollars to take over a system that is antiquated so what CCA offers is a way to Control the front end of the destiny if you will and what we're hoping for over time is functional separation So that as we see in other states the utility then really focuses on its core bread and butter And what is painfully needed which is the T&D system smart grid really bringing our grid up to speed? Let communities have some control and some choice around how they source and And procure their power and that's what CCA offers This is the quick policy framework CCA in California was enacted back in 2002 There was a another piece of legislation last year which strengthens the law in this state CCA especially if you're focused on the clean energy aspect absolutely Feeds right into AB 32 requirements when I'm out talking with local governments That really matters to them because ultimately it saves money over time in terms of AB 32 compliance by participating in a CCA My town which is small up in just north of San Francisco We're able to meet our compliance within about four years with nothing more than having joined the Marin Energy Authority Which is buying greater and greater percentage of renewables Yoriko mentioned the governor's renewable energy mandate of 12,000 megawatts by 2020 What's interesting about this and perhaps Nancy touched on this. It's 12,000 megawatts of local Distributed generation 8,000 megawatts of utility scale So I find that interesting from a policy perspective in terms of what the governor is saying His priority is and I'm very pleased to say that within that report and I've given you the link here You can go online and find it CCA is mentioned several times as one of the key methodologies of incentivizing and facilitating new renewable energy development throughout the state I Think I told you the customer experience really isn't that different because What essentially happens is by legislation around the country CCA is an opt-out program and the reason for that is that For those of you who remember Green Mountain Energy and some of those others those were opt-in programs They were never able to achieve scale and it's not because they weren't well run companies or offering a good product It's because the average American Doesn't really think about their energy They just want to make sure that the lights go on and they're getting to work and feeding their kids so in order to Create a level market an open market and create a level playing field CCA has been legislated as opt-out a Customer can opt out anytime Without charge and then after a certain period. It's for a small charge in our case It's five dollars if a customer wants to leave and go back to PG&E at any time they can do that And then as I said the utility continues to provide all the T&D and billing services CCA and it's called municipal energy aggregation government energy aggregation all kinds of different names, but Aggregation is the primary term Currently exists in six states. I'm happy to report that that orange state is New Jersey This slide is only a few months old, but they are now getting ready to launch their first two CCAs We see a huge surge in growth in the Midwest 243 communities went to ballot to approve municipal aggregation Their primary reason is that they're saving 25% on average upwards of 30% on their electricity bill over the default Commonwealth Edison and and Amarind so and there's tremendous growth in Ohio as well and we're seeing some growth in California I'm not going to go through these here, but if you want to take a look at these stats This is how CCA is playing out around the country We think that in California We have an opportunity to create kind of the most mature model Because what happens here is that the communities actually own their aggregation you form a joint powers agency Which means that you are able to Capture the existing revenue that was previously in our case going to PG&E You're taking that percentage of revenue Redirecting it into a local agency to pay for your cost of power to run your agency But then there's also an operating margin and what we're seeing in Marin is even within two years We've got an operating margin through which to do electric vehicle charging stations where we run an energy efficiency Training program with in one of our low-income neighborhoods. We run a green business program. So there are a lot of local Goals that can be met through the residual that comes through the rate payer income So this is not a tax payer funded initiative. There's no federal subsidies. None of that It is a market-based solution that is enabled at the local level Here's how it's growing across, California Two minutes, okay Many cities Why do you do it three reasons local economic impact don't underestimate the value of that redirected rate payer revenue on the previous slide We're looking at anywhere from in Marin. It's going to be a hundred million dollars annually if San Diego should decide to go forward It's going to be one point three billion annually in redirected revenue So it's it's not chump change in terms of what's happening at the local level the environment if you take over your supply portfolio and you have control over what you're buying and the the programs feed-in tariff net energy metering all of that think about what you can really do for the environment in terms of Pushing ahead with more nimbleness than the utilities on what's happening in terms of renewable supply In this state consumer choice matters. We don't have it right now So CCA in those areas that where it's being offered are offering choice And that's true even in restructured states This is an example of The town of San Itzamo's participation in the Marin energy authority that long orange line That's a third-party analysis of the number of metric tons of carbon reduction that they're getting from just their membership in MEA And by the way, they didn't pay to become a member So it's it's been a big win-win for the town this slide shows job creation How can CCA's be competitive we'll skip through this because you can take a look at this and feel free to call me about it There are risks the biggest risk right now in the state of California is price competition It's hard here to you know over promise and say we're gonna beat the incumbent utilities rates. We're at natural Historic lows for natural gas, which many of you heard about in the previous session And so it's tough to be competitive But what we can do is remain competitive and really as I said push the envelope in terms of renewable supply There's political risk just because it comes through a local government your city council and board of supervisors has to board Vote to enact it that can be hard because we're not typically energy experts This is Marin's program. I'll let you take a look at these slides later I think the three things I want to leave with you about Marin's program is that We came through years of fierce utility opposition and tons of money thrown at the opposition to make it to the launch point Nobody's lights went out Everything is working the organization is is meeting its bottom line and making money There is you know 5% margin or so We've just expanded to the city of Richmond just joined the MEA thereby offering choice to those residents and businesses there We have just signed our first feed-and-tariff project, which is a solar installation at a Regional airport. We have net energy metering. We have 400 customers with solar on their rooftop Feeding back into the CCA. So these are examples of what can happen if If you would like to know more about CCA or what we're doing feel free to give me a call No, thanks Sean and and thanks for your work and in putting Bringiness to market. It's a tremendous accomplishment the mayor Yang Good afternoon everyone. Welcome to Stanford and Palo Alto It's just great that we actually have a chance to get together Convenient and have a discussion around all these different topics today Let me take a quick moment as I bring up my presentation and I will be quick So it's interesting actually with Marin because we we hear a lot of the the same values Within CCA as I was you do within municipal utilities So today I really wanted just after seeing the number of hands that were raised around CCA's I'll keep my early sides, which just provide some broad context about publicly owned utilities And comparing them to investor owned utilities to as brief as possible Just to give an overall sense though in the US there are about 46 million people That receive their electricity from publicly owned utilities there are about 2,000 public power entities throughout the the country and It's Something that we are proud to be a part of that tradition in Palo Alto We have had our own electric utility for about a hundred years a little over a hundred years actually I think since 1900 Which is a lot older than me So what we wanted to focus on today was what has that meant for a community like Palo Alto But first I wanted to start off Where there is similarity between publicly owned utilities and investor in utilities? There are state laws that impact both sets of utilities and both models The example that we have here is public benefits programs There will always be a need within whatever model you have for an electric utility to have cost-effective energy efficiency and conservation new investment in technologies specifically around renewable resources R&D and demonstration projects and then finally within whatever utility structure or model you have there will always be low-income Rate payers within those service areas So it's important for this kind of legislation to be put in place so that you have this flexibility and acknowledgement of ability to pay within the different structures a Comparison though really highlights where there's a lot of value for the rate payer within a publicly owned utility There's a lot of transparency two weeks ago the city of Palo Alto just passed its budget as part of that budget process any member of the community that's a rate payer for our electric utility can submit basically their I'll call it in very diplomatic terms concerns about any rate changes and They are given the time of day during a city council meeting to come and speak to their concerns basically to Express that they object to any kind of potential increase Ultimately, if there is a certain percent if there is a majority of rate payers that express objection to a rate increase That rate increase won't move forward and you you don't get that same Model or structure within an investor owned utility. So there there's one benefit in itself For the return on investment for a lot of the investor owned utilities sometimes it's not clear Exactly where some of the profit is going to but you do know that it's going towards shareholders for publicly on utilities Any kind of equity transfer from a publicly on utility will benefit the general fund So the electricity electric fund within Palo Alto Where wherever we have a return on an investment that flows directly into our general fund which then is reinvested directly into public services And that's probably defined our general fund funds public safety to our parks to our streets and sidewalks So there's a lot of I'll call it a virtuous cycle of how we can use our funds from Basically what we're we're receiving in terms of return on investment from rate payers and the last area is Sean did alluded to it the city council functions as the governing board for the electric utility This is my fifth year on city council, and I feel that quite a large proportion of my brain cells are now Dedicated to thinking about our utilities within Palo Alto and just you know that there's a lot of very interesting policy areas that you can wrestle with Programs services that our utilities Local utilities excel at that ultimately the city council has to be savvy about In Palo Alto, we've created something called the utilities advisory commission We're fortunate to have a community where if you think you know it someone from the community is going to step up and say I actually have three PhDs in it and we're going to be a part of this discussion and we want to be We want to contribute basically to the good thinking that the utilities doing So to move on and to just be quick But a very important aspect also what Sean highlighted was the joint action So again because Palo Alto is an an older established public electric utility What we've been able to do through and I know Jim Pope our general manager of the Northern California power agencies in the audience through joint action, we've been able to partner with other municipal utilities and through a whole variety of areas The power of joint action can have a lot of economies of scale through procurement It can have a lot of economies of scale through actual construction of generation projects There's an incredible value through joint action as it relates to the legislative And add legislation and advocacy when you have joint voices that represent a breadth of Basically different types of customer bases in the different public utilities. It's an essential addition to What's going on in Sacramento or in Washington DC? So NCP I really has been a key Convener and action driver for a lot of us within this public space in Northern California One particular value is that we can as a public utility control our own supply the community Has very clear Objectives when you as a council member are voted in or out part of that discussion also necessarily involves well What's your perspective on what you want our electorate portfolio? our resources to be and in Palo Alto there's this the set of Discussion areas that I've just been very much Part of our utility commissions annual review of rates and programs and services and then ultimately the city councils vote on our annual budget So in general some high-level points from the American public power association 14% is the differential between investor owned utility rates versus publicly owned rates with the publicly owned utility is being 14% lower and the Equity transfer that I just had talked about earlier. That's quantified at 15% In Palo Alto in particular there's a huge commitment to sustainability and what we can do through our electric utility This is true through all of our sister agencies in NCP a and as a result of the ability for local communities to define this Their own policy objectives We we all have and are aware of the 33% RPS goal that ultimately will impact IOUs by 2020 in So in Palo Alto we have a 33% by 2015 and that's because of our ability to have that that local plan one One highlight from this year that I really wanted to point out we in a typical hydro year have about 70 to 80% carbon-free our No emission Resources as a result. We set a goal that by January 1st 2015. We wanted to have a hundred percent clean portfolio Will be one of the first utilities, you know, whatever the structure to achieve that and it's led to a lot of late night discussions and sessions and I mean affectionately nerding out with staff and utility commissioners and city council members Let me skip over then to Really? The last area which is about local generation This this particular opportunity Really is it's still in development within Palo Alto. We've recently put in a feed-in tariff We priced it at 14 cents per kilowatt hour. It was focused on solar rooftop We found that that has not quite penciled out for some of the solar developers For us when we saw that there was this opportunity to move forward with it We wanted to be able to partner with our business community and in particular It's iterative The advantage of being and having local control is that at the city council level and in coordination with utility staff We can revisit what we've done in terms of pegging our feed-in tariff rate and The last area that I'll focus on I feel a hook But the last area I'll focus on though is that This is just one of a menu of options that we're trying to provide to our ratepayers We've had a PV partners, which is a rebate program to install solar panels available to our Commercial customers since late the late the late 90s. So it's really about how you provide a menu of options around local generation that I think and truly enables the Publicly own utility to have a lot of value Thank you Great. Thank you very much. I'll marry a and actually could I ask Jim Pope to stand up so we can acknowledge him as well Great. Yes. So I so especially since we're nearing the end of the day I'm trying to leave enough time for questions and and so let's welcome up Deborah Wang Thank you. Well, thank you for the invitation to be here. I Am with the Natural Resources Defense Council, we're a national nonprofit environmental organization and We advocate for strategies to cut pollution and build a clean energy economy And so we're often asked to weigh in on this question of one type of utility structure or another and People are often surprised to learn that we're actually agnostic when it comes to the the basic structure of the utility and that's because we really focus on environmental outcomes and what's going to get us there and that's environmental policies and We've really seen both The best performers and the worst performers at both investor and utilities and publicly owned utilities so when we think about the different types of utility structures, we really look at who is the policymaker or the regulator that's going to set these key environmental policies and Mary a touched on this a bit for the investor and utilities the policymaker is the California Public Utilities Commission It's a five-member commission. The commissioners are appointed by the governor and confirmed by the California Senate For the publicly owned utilities. It's either the city council or a local board of directors and for community choice aggregators Usually a local board of directors So, you know, I think a lot of people have raised Marin as a great example leading the way on CCAs in that case In my mind the question isn't so much the energy provider in that case Marin switched from PG need to shell and I think most people would question that direction And but the the real question I think is who's going to make better policies Is it the Marin board of directors or the California Public Utilities Commission? And who do you want setting these policies about environmental issues rate issues and the many other issues that have been raised? so what I'm gonna Talk a little bit about is the track record of the investor and utilities and the publicly owned utilities in California on environmental Performance and our top priority energy resources. So you've heard earlier today our top three priorities In the state our energy efficiency Renewable energy and distributed generation and then getting off dirty power So I'll I'll touch briefly on on the performance that we've seen in each of those areas So starting with energy efficiency, this is the state's top priority because it's both the cleanest and the cheapest resource What this graph shows is in red as the investor on utilities and blew the publicly owned utilities And this shows their annual energy savings as a percent of retail sales So it's aggregating for both class of utilities Their energy savings and then normalizing it to make it apples to apples comparison across the different size utilities And what you can see I'm sure many of you know that the investor on utilities in California have been leaders for decades on energy efficiency Few new laws were passed in the mid 2000s and the publicly owned utilities have really been increasing their energy savings dramatically Thanks in large part to NCPA's leadership and you just recognized Jim and and Mary a who work with them And we hope that leadership will continue And so we're seeing a lot of dramatic improvement there in that gap closing with renewable energy It's been a similar story that the publicly owned utilities have started to catch up across the board And everyone needs to now reach for 33 percent by 2020 of course There's variations among the different investor and utilities and publicly owned utilities as Mary yay noted some of the publicly owned utilities have done far better than Some of the investor and utilities and vice versa So when it comes to distributed generation the state has numerous policies to promote DG This slide just gives an overview of the investor and utilities policies The publicly owned utilities have many of the same or similar policies But there's somewhat less consistency across The PO use as a whole so there's two main categories of distributed generation There's what's often considered wholesale Distributed generation which is projects that are on the utility side of the meter and then there's projects on the customer side of the meter So the reason that there's so many different types of policies to get at it is that the policies are really aimed at overcoming the obstacles that each type of project faces and The developer and the level of resources or sophistication that the developer has so starting at the large end The public utilities Commission adopted a renewable auction mechanism for any project up to 20 megawatts So these are pretty big sized renewable DG projects It offers a standardized contract, but these projects are expected to compete based on price So it's intended to get the best price for customers They have a feed-in tariff for projects up to one and a half megawatts and there it's for smaller projects So there's both a standardized contract and a standard fixed price And then on the customer side of the meter, you're probably much more familiar with these Net metering which allows customers to get full credit for the retail value of the electricity that they're generating the solar programs that provide rebates for both new and existing homes and Then there's also this self-generation incentive program which is really designed to advance some of the newer more emerging Technologies like fuel cells or or distributed wind generation This graph I just wanted to give a sense of the size and scale of each of these types of policies what they're trying to get at So starting with the smallest projects customer side Programs those are the smaller distributed generation projects But we're expecting to get a fair amount of capacity from those policies the feed-in tariff is is more of a niche But in a very important one and there's been a lot of projects. Thanks to the development of these That's anything up to one and a half megawatts You can see the renewable auction mechanism bigger projects somewhat more capacity and then of course the renewable portfolio standard is expected to get Significantly more capacity and that goes up to much much larger sized projects so the next category is getting ourselves off of dirty power and in 2006 the state passed the greenhouse gas emissions performance standard which essentially requires that all utilities Ensure that any new long-term financial commitments that they make to power plants be with plants that are Low emission basically The public utilities Commission has been implementing that Aggressively by next year the investor on utilities will no longer have any ownership stake or long-term contracts with power plants that do not meet the standard The publicly owned utilities mostly in Southern, California still get about 2,500 megawatts of power from plants that do not meet the standard These are mostly the big coal-fired power plants that are out of the state Importing power here and that's a big problem. We're polluting our neighbors and trying to take advantage of that low-cost power and The CEC is in the process of a rulemaking to try to help the publicly owned utilities Get off that dirty power. I'm sorry. I couldn't hear you, Jim So I want to leave you with just two of the current issues That have been live and you heard various folks earlier in the day talk about these the first is research and development This is of course of great interest folks here in Silicon Valley and all of the clean tech entrepreneurs the public utilities Commission recently approved a hundred and sixty two million dollars a year to go to the energy Commission's public interest energy research Program and it remains an open question whether the the publicly owned utilities will contribute to this statewide research and development effort Proportionally, they should be contributing roughly on the word of fifty million dollars a year in research and development if they were going to be proportional to the IOUs and then of course the the AB 32 cap and trade program is beginning this year and Every utility in the state is being given allowances which are basically permits to pollute On behalf of their customers and those are very valuable and so the public utilities Commission has an ongoing proceeding right now to look at what to do with the value of those allowances and The publicly owned utilities are going to be getting allowances worth somewhere in the range of three hundred million to five hundred million dollars a year And so are we going to use that to cut emissions? We should start having having that discussion So this very brief overview. I wanted to just show that the spread in terms of environmental performance has been narrowing That's the good news Six years ago. I think it was I challenged my friends in the public power community to to take The leadership role and and do better than the investor in utilities as you can see they've Begun to close that gap and I renew that challenge to you today and I look forward as a customer of immunity I would love to be able to get up here and say you've you've taken the lead. So next time. I hope to do that. Thank you Yes, can I make one clarifying comment? Yes, please Just just on behalf of MEA just so you know I think it's important for people to understand that MEA does have Shell Energy North America as its as its supply customer energy service provider But what's critical and what we're teaching other communities is that it's fine to have the third-party supplier as your partner But we also have For direct-to-market power purchase agreements for both solar and biogas and coming up on Thursday We have two more long-term PPAs that have nothing to do with Shell They're going direct to the market for new renewable Infrastructure, so it's a diversified portfolio and and you know just wanted to be clear about that the second thing is that in its first Reporting cycle Marin energy authority beat the California RPS by about 40% So it just shows you that with that nimbleness and you know, it's expensive to do it So, you know, it's we got to build some market scale But you've got the nimbleness to to meet and beat that RPS in pretty short order. Thanks So that's good because actually I was going to just ask if there was any quick responses to each other because I Do want this to be discussion But and I also have some questions, but I'm going to give priority to the audience who has been with you know So if we if you're interested in asking question, please please come on up But as you're coming up, were there any other quick other responses from either Mary a or devra to each other's Okay. Okay. Great. All right, so just please introduce yourself and Well, try to try to limit each kind of question and answer to say two or three minutes Just so we can hear as from as many people as possible, right Greg Ham stratolithics consulting to the most probably the most prominent Local generation option right now prominent in terms of people being aware of it is Residential solar that seems to have two significant problems as it's being implemented so far one is that it Tends to have a subsidy from lower income Electric users to higher income electric users second in California much of that is concentrated in coastal areas that have Economics that are not quite as good as inland areas in California Are you concerned about those worries? And if you are what are you doing to address them? I will just say that I expect that our procurement manager and Marin is thinking about those things But for right now, that's not something that comes up necessarily from a policy perspective with CCA I Within Palo Alto, I'll I'll actually add another issue another potential barrier, which is permitting Which is the process actually to go through local? And that's something that we also are working through to enable residential customers to you know, there's a difference between new construction and whether or not I can be built in and integrated through a new permit versus existing structures. We have some Rebates for example for solar water heaters that we found some Challenges from a residential perspective We don't have the uptake actually that we had hoped for around solar water heaters And this is anecdotal. I'm also going to acknowledge that our utilities doctor from Palo Alto Valfon is here So if I misspeak, I she'll jump all over me But just that there are structural issues that sometimes come up from residential Perspectives that say well if we have you know older rooftops We clearly have a desire to move forward on this But the permitting won't allow us to do this because of your concerns around structural integrity And that's something to where we are addressing it and we don't have an answer just yet So it's on the radar And I would just add I think you raised one of the key issues that is really starting to be debated across the state when it Comes to the net metering policies because it really is a subsidy or an incentive program When you are getting credit for the full retail price of the power That means that you're not paying for all of the transmission and distribution and other fixed costs of the system that you're still taking advantage of so clearly solar technology deserves to have Incentives from the state the question is how much and what forms and I think that We need to have a fuller discussion across the state about this issue Recognizing all of the different policies that are in place and making sure that we're Taking the best advantage that we can't have all of them. Well, and and we can't and say CCA does have an interest in this Right now mainly from the perspective of cost shifting that's going on So because net energy metering is definitely an integrated program into CCA we We encourage folks to actually supersize their Projects in order to sell the power back into the local CCA that way you're not getting out to the the large Transmission at all, but if you've got cost shifting going on over to TND that's creating some competitive problems Thank you next. Hi, I'm Jerry Glaser from Sunnyvale. I'm playing two roles there right now Thank you for answering the third of the questions that I had here about CCA on that metering One of my roles is Chairman of the committee called Verizon 2035 which was taxed with coming up with the climate action plan the general plan changes for Land use and transportation and after 18 months of working and studying and rewriting all of that Now the consultants come back in and say you can only achieve your climate action plan goals if you have a CCA It turns out if you make all the changes in the city You couldn't actually achieve the reductions that you want to achieve In Palo Alto, thank you for the RFP one of the things we had as a committee was we said We want to have 50% of the power from our city Be generated from within the borders of our city So your RFP recently where you're trying to pick up a hundred kilowatts a clip from inside the city and Sean I've gone to your sites and gone to your seminars So the second job I have is Sunnyvale neither what of which I get paid for I'm a sustainability commissioner and the reason that I got reappointed for a second term Was I pointed out that what we needed was a CCA and I'm now on a subcommittee which is investigating CCA's So all your materials been very very useful. Give us a call. Oh, yeah, I'd like to talk to you quite a bit about it The main thing that is a concern By the way, we'd have more accounts than you plan on having ultimately because we're the second largest city in San Clara and the fifth in the Bay Area Is the resistance that people will have For having a CCA and yet I can't see the facts that would stop somebody from wanting to have a CCA But yet I can already tell that most of the discussion is well Why are you gonna force this on us? And I know you've been wrestling with that. Do you have any particular? hints and ways of twisting this even the press you got recently wasn't all that great of Nipping that in the bud Well, I think you have to start with just an acknowledgement that if People think something ain't broke then why why do they want to fix it, right? And as I said earlier in my experience as an elected official Our constituents aren't thinking about this stuff really. I mean, it's it's a big nerd party And so so I think I think what you have to really explain to them is that it really is about Making the shift to a clean power cleaner air You got to bring it back to what matters to them because it's not about the model at all And I think the other thing that really is gonna resonate with them is the value of that redirected revenue It's not going back in CCA's case. It doesn't go back into the general fund So you can avoid any, you know, sort of perception of that, but what it is doing is that it's stimulating local develop local Renewables it enables Solar rooftop. There's there are many integrative factors of CCA which I'm happy to talk to you about The third item is oh Gosh you just asked me I'll think of it and get back to it. Yes, we do have some strategies to deal with that Hello, I'm Richard topping with Green Data Center's worldwide and I have a quick question With a brand new technology that can literally take and feed into the power grid that has zero emission How would I get something like that approved to? Go into the feed-in tariff. Is there I mean, is it an extended process or how quickly can that be accomplished? For our particular program in Palo Alto, we actually have a monthly Process a cycle to actually invite people to submit applications if they're interested So the good thing is our utilities director is two rows behind you But it's it's something where it you know, like I said, we're we're learning lessons through the we started in April 2012 so we've just had a couple months to learn from it and And all from my perspective We're continuing to tweak it just based on the lessons that we've learned through is there currently a limit on how much power We can produce we've put a cap of four megawatts on the entire program. This is seen as a pilot But I think a minimum of one megawatt So there's it's a tight band and it's really something that it's I think you know as we get more Basically data points and intelligence from the people who are even considering and interested in this program within Palo Alto Then we can start tweaking it as necessary. Well, I definitely am Thank you. Sorry Val Okay, thank you, so we have three more Questioners and I think we'll just run the two or three minutes into the break if you if it's okay Well, we have I know there's a coffee break until four so okay, welcome All right, my name is Robert Ferber from San Jose State University and Sustainable Silicon Valley Today we heard about a lot about local control But I don't feel we heard enough about local generation from both the point of view of the CCA and the PUC how do you rank the following in terms of importance for generation the cost per gigawatt hour? the proximity of the generation or environmental sustainability and When you say environmental sustainability, you mean the type of generation. Yes I'm happy to jump in a year for one of the projects that we're considering ex use an aerobic digest digestion project We had a ballot measure measure e that was really related to waste to energy facility We have a regional wastewater treatment plant That also is immediately adjacent to landfill The vote the ballot measure was to undedicate ten acres of parkland to consider a long term You know to create basically for ten years to give us a time frame to look at some local generation options as a related to You know regional wastewater treatment plants have sewage sludge at the back end of the entire process And whether or not we could consider some waste energy options for that in addition to our landfill and it's It just this past just recently Actually Monday night if you are looking for a good time, you can come to the city council meeting We're going to be talking about it until the wee hours of the morning and it's all about local generation it's You know for prioritization of cost proximity and environmental sustainability it all depends I mean all of those are important. It's hard to say that one's going to trump the other if something ultimately is you know going to be this pristine project in terms of environmental sustainability But break the bank as a local utility you have to be very cognizant that you have You basically have a captive audience a captive population that are ratepayers So you can't de prioritize cost in any way at the same time, you know for proximity We want to consider all things that are just within local So you you don't have to think through transmission issues. You can just really identify it That was the basis for feed-in tariff ultimately where we can have distributed generation Rather than you know really have to tackle through I mean Even if you're a nerd transmission issues can really just challenge you with the level of detail That you have to wade through and the bureaucracy that exists around transmission and the dynamics with Your relationship with your investor own utility and for us, it's PG&E. So that's a Current example of a project that we're looking at that we're excited by but it's really going to come down to the details I would just add I think that environmental performance is really a threshold issue when it comes to distributed generation And the policies that I talked about there are both very clean technologies and extremely dirty technologies In fact, some of the dirtiest technologies out there are Fossil fuel fire distributed generation like backup diesel generators and from a public health perspective It's much much worse to have the pollution in the place where everyone's breathing it as opposed to somewhere further away from where people are living So I think that Environmental factors are a threshold for these policies Proximity is also to some extent a threshold because it's really not distributed generation if it's far away from population centers And then I think a lot of it really does come down to cost. We have these policies to try to help these technologies gain acceptance in the market and bring down their costs, but today the costs are still higher than The large central station plants that have economies of scale. So Hopefully with a lot of these policies will start bringing those costs down So Deborah, I just have a quick follow up question there. I remember I was going to ask you about the avoided cost and in distribution Do you do if we are able to produce locally and avoid the the long-distance Distribution, how much do we save in general? I mean it's what's what's a good way to look at that? So it's a great question. Of course the answer varies wildly depending on where you are. I would say very roughly speaking The transmission component is 10 to 20 percent of the retail price Distribution is a significant chunk as well, but with distributed generation You're not avoiding distribution costs usually, right? So the transmission costs are significant in some cases. They can be very significant of especially if you are Avoiding or deferring the need for a whole new transmission line But they're not as big as the entire, you know avoided Costs of the TND and the generation. So it's okay small but meaningful components. Thank you. Thank you And just to quickly answer your question, I would say that For CCAs that are focused on a long-term goal of asset formation and ownership Within the portfolio. It's going to be it's going to be a spectrum of time because I think Until we build market scale and we're able to compete against the 10,000 pound gorilla in the room called the IOU Cost matters to us, but there's no question that the environmental benefits Are part of the reason why we came to be we led with the greenhouse gas goal. So that's why Marin CCA came to be We're really focused on local solar. So we have a solar shares project going in over our ferry building That's a solar structure, but it allows individuals like me who live under trees to invest in a piece of that and Credit that toward my bill as an as a net energy metering customer. It's a lot like Smud's program We've got the program going in at the airport So we're doing everything we can to push not only local in-county generation, but also we're very interested in a regional picture So how do we sync up with Sonoma, for example, who's considering a CCA and tap into their Geothermal stuff. So there are a couple of different ways to look at it Cost matters proximity definitely matters, but I would say in order its cost environmental proximity Thank you very much. Yes My name is Emily Thomas. I'm a graduate student here at Stanford and I guess you've kind of all touched on my question a little bit I'm interested in very local generation and net metering and I wondering if you've talked about kind of the implications on the local infrastructure and What improvements might need to be made to the substation to allow electricity to go both ways And if you're including that in your your thoughts and about costs and moving towards this 33 percent Renewables, which will probably include a lot of local generation Well, you're absolutely right that in order to sort of get to that vision of a very distributed grid We do need to make significant improvements in the underlying grid infrastructure So all of the smart grid issues that were discussed earlier You know having a grid that can communicate with different parts of itself and Have the grid operator understand what's happening in real time at all the different points We expect to see a shifting peak and that's gonna have really big implications for how we do the rest of our Planning in the in the whole system So right now some of the folks talked about earlier solar tends to peak Somewhere in the noon to four time frame if we get to really high Penetrations of distributed generation then we can see the the local peak shifting and that Influences everything from the big central plants that you build to the size of the distribution transformers that you have Locally, so I think that there does need to be a lot of changes I'm not sure that the state has really developed a vision yet though for Exactly what the grid would look like if we got to really high penetration rates with DG But it's something that the CEC is exploring very actively. Yeah, I think I'm thinking even more mechanical Just how you can change what's already in the substation to allow for electricity to go backwards I see that as something that isn't talked about very much If you're saying for instance, you encourage your customers to oversize Then you're going to need to go through and reevaluate your substations So I just haven't heard very much discussion about that. Yeah It would be great if there was an engineer in the room to talk about that There you go I mean, I'm not an engineer, but it's kind of interesting to hear how we're We're demonizing that metering likely have demonized Electric car chargers versus air conditioners and how we've demonized Electricity is a fuel source for electric vehicles versus gasoline and in full disclosure I've had a slow rate my roof for since 1999 drove an EV1 rap where you be now got the leaf So the thing is that wind is an intermittent resource and wind is a real problem because it is kind of the anti-load follower It blows at nights and not every night. It does not blow at 3 p.m. We have the supercritical peak in California solar energy in the distribution grid is a huge win for us and It may be arguable that net metering is in fact the right model for Solar generation because it's taking high-cost power peak or power plants offline during the day It's reducing the need for those It's also lessening the load on the distribution grid because what drives the actual peaks it's air conditioning plus a lot of other things Going on but it's air conditioning. It puts us over the top and kills us, right? And I know I've served I've suffered through two five-day blackouts from low distribution grid That was air conditioning, you know, not EVs and stuff. Okay, and if we had distributed generation there that wouldn't have happened so Get to your just it can you get to your question because I we do need to wind up in a second That's exactly where I'm headed. Okay, so, you know address the fact that net metering Lowers your your transmission and distribution costs. There are benefits there It takes the peak or power plants offline that help, you know Your your emissions portfolio and help lower the cost Well, I know I certainly didn't intend to demonize net meter in any way. We strongly support it I was simply acknowledging that there are legitimate Cost issues that the state needs to discuss more fully and you're absolutely right when we do that cost analysis in the discussion We need to value fully all of the significant benefits that Distributed generation is providing and you've hit on a number of them And again, I agree with you that we shouldn't treat treat EVs any differently than we've treated any other load that enters the system in terms of I think just to get to the kind of combo of the two questions, I mean it's This isn't related directly to your question, but it's it's relevant around our substations as a public utility We do have nine substations that we We have the responsibility to then maintain upgrade A related discussion that we just had a couple weeks ago at council was What we need to do with our substations as it relates to our wireless infrastructure I mean that has some impacts around smart meters in the long term But then ultimately what what how we can think about, you know, the substations that have space to allow co-location for incumbent providers for wireless Technology to then consider use of what's ultimately now a public a public asset and It's that kind of it's very it is detailed. We're bringing on consultants We want to go through a competitive process to be able to bring on some of that that has that kind of specialized background But it's an example of what we do need to think throughout the local level around the use of substations the maintenance of substations And then ultimately the future use of our substations You know that that has impacts on on metering in the long term I mean from I'm not an engineer. You'd have to talk with my brother, but you know, I think it's a great question Okay, well, well, thank you very much, please help please join me in thanking our panelists


Okay, good morning, good afternoon, good evening, depending on where you're joining us from. Welcome to today's CNCF webinar, Routeless Containers in Gitpod. My name is Kristi Chan and I'll be moderating today's webinar. We would like to welcome our presenters today, Christian Weichel, Chief Architect at Gitpod and Albin Query, Director of Kinfolk Labs at Kinfolk. A few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen. Please feel free to drop in your questions and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of all your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinars page at cncf.io slash webinars. With that, I'll hand it over to Christian and Albin to kick off today's presentation. Take it away. Thanks, Kristi, for the intro. So welcome. Today we're going to talk about Routeless Containers in Gitpod and to dive right in we first have to talk about what is Gitpod. Gitpod is an open source project that automates development environments and you can think of it as a CI system that automates regular builds. Gitpod automates the provisioning of development environments for pretty much every developer. So it has ready to code dev environments, meaning all your tools are there, code is downloaded, code is compiled and you can start working with a click of a button. And it does that behind the scenes by provisioning Kubernetes pods. So each workspace that you start within Gitpod is actually a Kubernetes pod. And we want those pods, those workspaces that you can start in Gitpod to feel pretty much like your local machine, except you get a new local machine for every task you want to do. So there's no previous state that can impede what it is you're trying to do. And when we started out for a long time, one of the big differences between your local machine and what Gitpod would give you is what you could do within such a Gitpod workspace. For example, there was no sudo, meaning you couldn't install things after your workspace was running. You could only do that in the Docker image that you would bring to the workspace. And also there was no Docker, which in a cloud native environment is a bit tricky. And so what we really wanted to have is we wanted to enable those two key features where you could sort of have root in your workspace and be able to install things after the fact once it's running, but also where you would have Docker and do Docker build, Docker compose, etc. And this talk is pretty much about the technologies that made this possible and how we enable this in Gitpod. So now this is possible. Now you can run Docker, you can do upget install. You basically have root within your workspace. And the most naive way possible of doing this is by simply giving you all the privileges within the workspace container. We could just run as root, so to speak. But the clear and obvious downside is that that would also mean that everyone inside the workspace would effectively be root on the node. They effectively have all the privileges they need to potentially escape the container and to have really a lot of privileges on a node that is shared with say 25 other users. So clearly this is not an option and we need some good way of isolating those workspace containers from each other, but also to the node. And this is where Linux isolation tech comes in. And I'll hand over to Albon to talk about that. Thank you. So there are different ways to isolate more of the parts from each other and from the host. One way is to think about VM-like containers on time. Those are, for example, nav-like containers, GVizor, cata-continuous, Vcranker. And this technology to provide improved isolation compared to what Linux containers are. They work in different ways. For example, nav-like containers use Unikaners. This means that for every new workload there will be a different Unikaner build specifically for that workload. There is GVizor. What it does is re-implement the system call interface. So it's re-implementing in code. So when your application makes a system call, instead of talking to the Linux kernel, it talks to this interface, this application kernel. There are cata-continuous that build lightweight virtual machines, and it's compatible with several high-level servers, for example, QMU or FAC crackers. Those different VM technologies, they provide more isolation, but they also give more limitation compared to normal Linux containers. There could be compatibility issues, or they could have decreased performance, for example, with network traffic or higher system access. What we want in general is to have higher density. That means to be able to put a lot of ports on the same node without having to mean that too much. So next slide. There is an alternative approach, which is not to use VMs, but use what is called username space. So username space is a feature from the Linux kernel. Among other namespace, for example, network namespace, pin namespace, and so on. Currently, that's a feature that is not provided by Kubernetes. So Kubernetes works like you see on the picture on the left. It has worker nodes, Kubernetes, and so on that don't use username space. What username space does is to isolate users. So that means the user root inside the container is not the same user root on the host. So provide some estimations. There are different ways to use username space in Kubernetes. Here I provide three different explanations of different ways to use it. The second one from the left is called KEP 127. What it does is it adds a new field in the podspec, a bit the same way that you have an host network in the podspec to say whether or not you want to use a new network namespace in your pod. It adds a new field for username space. So I'll present it in this picture where the user's namespace will be located in this architecture. So that's a KEP. It means Kubernetes and Lenspan Proposal. That's not something which is merged in Kubernetes yet. That's something we work on that with others in the community to provide that. Another way to use username space is the next one, KEP 2033. It's so-called the hostless mode because it allows to run the different Kubernetes component without being root. For example, you cannot run Kubernetes without being root. You can run the container without being root. So in this way you have a username space that goes around all the components of Kubernetes. On the last solution is the one written by Gitpod where we don't touch Kubernetes. So we can use a Kubernetes upstream without modification. On the inside the pod, inside the workload, it makes use of username space so it creates the username space at this time. In this way it works on current Kubernetes without patches. So how do you create a username space? And this is an example, sort of walkthrough how to create such a thing. And it starts with the unshared syscall, the other syscalls that can also do that, that create the username space itself. And then once you have that username space, you need to establish the UID and GID mapping that maps a user ID from within that namespace to a user ID outside of that namespace. And this happens by writing to files in the proc file system. And then lastly you need some execve syscall to get a hold of the capabilities inside that username space. And you basically get the full capability set at this point, including Capsis, Upman, CapNet, Roar, etc. You can try this yourself with this command. This lets you observe sort of the things, the steps that happen to make this work. So unshare minus U, minus uppercase U creates the username space minus R, maps your current user, your current executing user to UID 0 inside that namespace. And the s trace in front just traces what's happening. So this is all fine, except in a Kubernetes pod, we would need to give quite far-reaching capabilities to make this work. So to write these two files, you need Capsis, Upman in the outer namespace. And because at this point Kubernetes does not provide username spaces yet, this outer namespace would need to be the node as a whole. And we don't want to provide Capsis, Upman for security reasons on the node inside the workspace. So we need to find a solution to that. And the way we built this within Gitpod is the root process that we start inside a container we call Supervisor. And Supervisor Ring 0 is sort of the thing that gets started that's the command of the workspace container. And then it starts the username space as Supervisor Ring 1. And once we have that, we make a GRPC call to a node demon service that runs on the node that we call workspace demon. And this service then has the capabilities on the machine to actually write those files. And we pass the PID of the process that identifies to use the namespace we want to write this UID and GID mapping for. That's all nice except now we have to do PID translation. And the reason for that is that containers in general are in essence a collection of namespaces and other isolation tag. And one of those namespaces is a PID namespace. This is why any process that you start sort of as the root of the container becomes PID 1. And it's not the actual PID 1 on the node, say system DO in it or something like that. So the PID that we'll receive from Supervisor Ring 1 will not be the PID in the namespace that workspace demon sees in the node namespace. So we need to do some translation here. So outside of the PID namespace of the container, this might be is something completely different. And in order to do this PID translation, there are a few ways how this could be done. There is no syscall yet that can just do this translation for you. There are some tricks using UNIX pipes, but also it's in the in the proc file system. So if we look at the status file of a process, we see that there is an NSPID entry, which lists all PIDs in the children namespaces from the perspective from the process that's looking at this file. Because we know that the PID that we're looking for must be a child process of the container of the workspace container, we can look at the children of that workspace container, look at their status files and this way identify the correct PID. So now we can create a username space and we can establish the PID mapping within this username space. Now we're left with a problem because this is working really well. If we look at the file system, we see that the UIDs now all of a sudden don't make sense anymore, at least at first glance. But thinking about it, this is exactly expected behavior because on the file system, we have some files that belong to actual proper UID 0 and we have some files that belong to a user that has a mapping within this username space. And the ones that actually belong to proper UID 0, they are shown as 65,534 in here because we don't have a mapping that maps the user inside the namespace to UID 0 outside. To illustrate that, what we would like to have is a file system that from within the username space looks like this. You have a whole bunch of files and folders that belong to UID 0 and you have some that belong to say some other user in this case, 33333. And in this example, we have a UID mapping where UID 0 inside the namespace is UID 10,000 outside of the namespace and UID 33333 inside the namespace is 433333. So basically just plus 10,000. So in order to get this view on the left from within the username space, we would need to have a file system on the node that actually looks like this, right, that actually has this UID shift implemented. But in reality, the file system that we need to do the shift for is the root file system of our container. And this root file system was put there by the snapshot of the container runtime. And it doesn't know about this UID shift and it also doesn't care. So in reality, the file system looks exactly like we would want it to look like from within the username space. So we need some process that dynamically does this UID or does this UID shift for us. And there are a few technologies that can do that. For example, there is Fuse Overlay FS, which has the benefit that it can be used without any privileges outside of the username space. So you can use that completely from within the username space because Fuse can be mounted within username spaces and the rest that's needed is a username process. There is very little upfront cost. All you got to do is start a process. But the runtime cost is comparatively high because it has to go through user land. On the upside, it is not very platform specific. There is also Overlay FS meta copy. Meta copy is a mode in Overlay FS where it just copies the metadata to the upper dear. So what we could do is we could basically mount an Overlay FS on top of the file system that we would like to shift and then basically do a change on onto that file system. And this is exactly where the upfront cost comes in. This change on is potentially very expensive if the root file system is large. The runtime cost is comparatively low. In terms of platform specificity, Overlay FS, my knowledge, can only be mounted from within username spaces on Ubuntu because they have a non upstream patch that takes the right box, so to speak, on Overlay FS. And lastly, speaking of Ubuntu, Ubuntu has support for a file system they call Shift FS, which can do this UID shift at mount time, so to speak. It doesn't completely work from within the username space because you need something that they call a mark mount. And this you can only do with privileges in the outer name space. But it has very little upfront cost. All you need is a mount. Runtime cost is very low. It is quite fast and it runs entirely in kernel space, but it is very platform specific. It only works on Ubuntu. For Gitpod.io, which is the SAS offering, the SAS version of Gitpod, we ended up going with Shift FS because we have control over the environment that this runs in, and we deeply care about workspace startup time and performance. So now that we have the PID mapping established, we're using the same trick that we used to write to the PID and UID map to actually create this mark mount, this privileged operation that we need to do that. So we make another GRPC call to the workspace demon who then creates that mark mount for us. Once we have this mark mount, we can use it to mount the shifted file system. And then we do bind mounts to dev, proc, etc., other bits of the file system of the container. And then start Supervisor Ring 2, which basically does a pivot route to this new file system. And this is how Inside Ring 2, your A inside this username space, but also you're looking at a shifted file system. So to you, all file system permissions and ownership looks correct. This is all nice, except we cannot just mount proc for this new file system. But we want to do that because Supervisor Ring 2 also creates a PID namespace to sort of hide this mechanism away and also to prevent users of the workspace from sort of escaping this new root file system. And we cannot mount proc because if we look at proc within that container, we see that there's a bunch of files that has a mask placed on top of it.  For example, proc kcore or scat debug, which might even leak information about other namespaces, hence other containers on the node. And so what Kubernetes and more specifically the run times do, container run times do is that they mount masks on top of the files and folders in proc in order to prevent workloads from accessing those files. And in the kernel, there is a check that checks if such a mask is present. And if so, it prevents users from mounting slash proc because that would essentially render those masks useless. In order to work around that and to never sort of offer an unmasked proc to to the workspace container, we again rely on workspace daemon to make that mount for us. And the way that works is that we call out to workspace daemon with the PID of the, excuse me, passing in the PID of the target PID namespace, we do that proc mount, establish the masks, and then move this entire mask proc mount into the mount namespace of Supervisor ring one of our new file system that we're creating. That's nice. So now we have root inside our workspace and it feels like root and things like upget are working, but Docker isn't working yet. And rootless Docker has a, has seen a lot of work first and foremost by Akhi Hirosuda who has worked relentlessly on things like rootless kit and in general making Docker work in a rootless mode, but also our friends from, from Kinfolk, Alban and his colleagues have done a lot of work in this space. So how do we make this work? And the key issue here is that Docker needs a lot of capabilities with regards to networking and we can provide those capabilities by wrapping Docker or the Docker daemon specifically in a network namespace. And to do that, we need to provide some networking into the outside world, so to speak. And for this we use slurp for net and s, which is a user land mechanism to make, to make network namespace or the connection, their connection to the outside world work without needing privileges in the outer namespace. For proc mounts, because the container that run inside this or run in this Docker daemon, they will also need specific proc mounts because among other things, they're also PID namespaces. We use the same trick that we used to create the proc mount for the workspace container as a whole or for supervisor ring, ring one. We basically call to workspace daemon and ask it to mount proc for us. Now, this isn't quite as easy as it might sound because we need to sort of catch the right moment to do that. And we do this by sort of interjecting into run c. So as part of the OCI runtime spec, the container orchestrator, so to speak, in this case Docker or container d actually will provide, it will create the OCI runtime spec and in there it will have something like mount proc. And we sit in between there, we modify the the OCI runtime spec and add ourselves as hook in the container lifecycle to actually do that proc mount. Okay, so much for how this looks like on paper. Let's have a look like how this looks like in the real world. So this here is a is a git port workspace that runs in my browser in a browser tab. Obviously, there's a full blown container behind it. This is what we've just been talking about. And so in here, I can do things like this. So I can just install install new software, for example, but I can also run sudo docker up. And this will give me will start the Docker demon with the process that I just described. And now I can run, I can just run Docker containers. Right. So I just started Alpine. I can also do that with starting ports. And then git pod will realize that this port is now served. At the moment, there's nothing actually running on it. But if I if I start a web server in here, right, then I can access this service that now runs inside a Docker container in my workspace. So networking also works across this boundary. Right. Back to Evan. Thank you. As you have seen, there are different way to make it work. There are some difficulties that it might be easier in the future to implement such an architecture if we have more things in the Linux kernel. And I will talk about a couple of that now. So one patch set that is currently being reviewed is ID amount. And that's something to do something to do something similar to shift FS. But instead of being open to patch, it's something that is pushed upstream and is currently being reviewed. And once we have that, I'm hoping it will be easier to do this kind of shift FS operation. That will be useful both for the root FS of the container to be able to have this different ownership of file that will improve the performance both in time and in this space. And another use case is for volumes. So when you have in Kubernetes a host volume, you do a bind mod from the host to the container to be able to have this shifted ownership on this file. So that's one thing we like to have on the next slide. There is another thing. So that's another feature that I'm enthusiastic about. It's called second modifier. It's kind of a new second architecture with a second agent. So what is the use case for that? As you have seen before, there is this interface between the workspace and the demand outside that do some methods like prepare user in space or mod proc that run a previous operation like mod. And I'm enthusiastic about second modifier because it will be able to provide a proper interface for this kind of thing. So what you will be able to do is to have the container run the mod system call normally and then second notify will intercept that and send a message to the second agent that will run the mod system call on the other for the container. So on the next slide, I will explain a bit on that. At the top right, you see I have a second policy. That's where you define for each system calls if you want to know or deny their access to that same system call. But with this second notify feature, you have an additional action you can take called notify. And what it does, it will say every time the process in the container use that system call, it will defer the decision to an external agent called the second agent. And then this agent will be able to take decision on the system call on behalf of the container. Here I put a diagram where you see at the top left a run C when you use a run C or it's the same thing in communities when you start a pod. What happens internally, it will fork and exec a couple of time to create this child process and then it will execute the second system call with this notify feature and then you get a file descriptor to be able to get the events in this example, the mod system call. And that file descriptor will be passed to the second page that that will be able to run actions on behalf of the container. So when the container in the end of the month, the second page will do that. What it means is it has the potential to make things simpler for the pod because we could just use Docker inside the pod normally. And when it's on the month system call, it will automatically call the second page on to do that without having to implement this gRPC interface. Okay. And on the next slide, part of summary of the different future technology in the Linux kernel or in general that I think are interesting. So first in Kubernetes, the support for username space, there are two Kubernetes and a small proposal that are for that, that I described before. And in rootless kit, if you go to this GitHub page about rootless containers, you will find a lot of, a lot of history, lots of projects interesting like a rootless kit. Usanities, which is about running Kubernetes without being root, sleep for net-to-nest, which I'm talking about, which is the same thing, but with more performance using second notify. So second notify in Kubernetes doesn't exist yet. That's something that is a work in progress. But here, some different pull request. So there is a work in progress to make it available in RCA on time spec and in RunC. In Sierra and on command, the work is done only. And I think we are working on this second page, which is a generic second page to make it easy for you to use this kind of, this second notify feature. Thank you. That's my last slide. Yeah. So briefly to sum up, Gitpod provides deaf environments that are built for the cloud and automatically provisioned. Username space are the key tech to make, provide root within these workspaces. And then thanks to all the amazing people that actually make this stuff work, first and foremost, Kenfolk, also Agahiro Suda, and the community as a whole. Thank you very much. All right. Thank you both for a great presentation. At this time, we're going to move into our Q&A segment. So if you have a question for our presenters, feel free to submit it either through the chat or through the Q&A box at the bottom of your screen. It doesn't look like we have anything submitted yet, but we'll give folks just a few seconds here to submit their questions. Okay. It looks like we might have a shy group among us today. Albin and Christian, I know at the beginning of your slide deck, you have your Twitter handles. Do you want to go back to that slide just in case folks do think of questions later, place where they could reach out? Perfect. Awesome. So you can see both of their Twitter handles here on the screen. So feel free to reach out with questions. I'm sure they would love to chat with you more about this cool thing called Geppod. Well, that'll do it for us today here at CNCF webinars. Thank you again, Albin and Christian, for this presentation, and thank you all for tuning in. A reminder that the recording and slides will be posted later today to the CNCF webinars page. Stay safe out there. Continue to wear a mask, and we'll talk soon. Bye.


The press conference provided an opportunity for the management of the Millennium Heights Medical Complex to review its operations over the last year and to outline its new plans and vision. The complex comprises of three facilities namely the Owen King European Union Hospital, the National Mental Wellness Centre and the Turning Point Rehabilitation Centre. This hospital admits in excess of 6,000 patients a year. It sees about 15,000 patients in its own patient clinics or accidents in the emergency department sees on average about 15,000 patients a year. It's quite a busy facility. We do about 1,900 surgeries of different types, deliveries at over 1,100 and we see at least 94 patients at our hemodialysis plant here at the hospital. Dr James noted that the facility has been collecting data on patients experience. The process uses a software called Power BI and questionnaires which measures the happiness factor of the services provided to the accident and emergency patients. There's a fair amount of happiness with respect to nursing care attention, interaction between our caregiver and the patient. Where we fell down is with respect to supporting staff going to the bathrooms. Quality of our meal services is not up to par. And to a lesser extent, comfortable with privacy, treatment for pain management, staff position and respect, all these are given fairly good scores. However, the chief executive officer says the hospital is not without its challenges. He cited irregular wait times to financial constraints which impact the maintenance of the facility. Dr James says areas of concern for outpatient care and A&E will access to diagnostics and waiting times. The complex also questioned patients on their likelihood to recommend someone to the hospital using the net promoter score method. This metric, 73% of the patients that were interviewed at this hospital in the month of December said that they were very likely and likely to make a recommendation for their friends or their relatives to use the hospital. So in spite of what you may hear, the hospital still enjoys a fair amount of goodwill with the constituents that we serve. Dr James also gave some insight into the board's commitments for 2023. So our priorities really is built around what is called people first. The first, the F in the first stands for improving our fiscal management and performance. The I speak to building our internal capacity in terms of goodwill, in terms of our finances, in terms of our credibility with the public, in terms of our financial strength, this is what it is about. Revenue generation is finding ways to generate new sources of revenue. We are not increasing the fees contrary to the view. No fees will be increased, but we have to find ways to stop the bleeding where it exists. All we are seeking to do is to collect the funds that are legally lawfully due to us so that we could have the funds to be able to replenish and keep the system going. The Board of the Millennium Heights Medical Complex says they are looking forward to implementing the new plans during the course of 2023 in an effort to enhance the quality of service provided by the facility. It is also the intention of management to keep open communication with the wider public on the performance of the facility. For the Ministry of Health, Wellness and Early Affairs, I'm Sande Justin.


Alhus is the second largest city in Denmark, located right here. Don't let the small size fool you, there's a whole host of things to do in this gem of the city. And this is me and my study group after we graduated from Alhus University. Alhus University is a major international research university that offers a full range of world-class degree programs, with an arts, technology, health, business, and social sciences. Speaking of world-class, take a look at our expansive campus park. This is a favorite spot for students to take time off and find some afternoon huga between classes. The teaching style in Denmark is student-centered with a more informal learning environment. This style promotes creativity, self-expression, analytical, and critical thinking. There's a close collaboration between students and teachers, and you can see this represented by students calling their professors by their first name. That was definitely something I had to get used to, but it's the norm here in Denmark. There are tons of social activities to join in on at Alhus University, such as concerts, international student nights, and trivia get-togethers. And remember that world-class university park? It's also home to the annual Kapsales Boat Race, which is the largest student-led sporting event in all of Denmark. The university's main campus is nestled right in the center of Alhus. From the campus, you can bike or walk to a number of the city's main attractions, such as the historic downtown quarter, the calm, clean beaches, or the extensive natural parks. In Alhus, public transport is super easy to access and navigate, so you don't need a car to get to all the amazing places around town. However, when it comes to transportation, the Danes prefer to bike and walk everywhere, even in winter. And I've joined them. The city offers charming cobblestone streets, impressive modern architecture, daring art, eclectic music, inspiring museums, all with a beautiful natural backdrop.  In Alhus, you'll get that big city feeling with a welcoming and homey atmosphere. As a student, you'll find the city swarming with great study spots from cozy cafes to outdoor workspaces and even the state-of-the-art Harbor Front Library. And did you know that Denmark is one of the safest countries in the world? The country's crime rate is very low and there's a great sense of trust within the community. Haven't been brushing up on your Danish? No sweat. Danes are ranked as second-best non-native English speakers in the world, which makes it very easy to get around the city with English while you slowly pick up some fun Danish slang. So come join us and experience a semester at Alhus University for yourself.