Hello, everyone. I think we are going to get started now. It's two o'clock UK time. So welcome everybody and thank you for joining us today. My name is Juliette Tunstall and I'm the external events officer here at IID. Today's event is looking at how subsidies can accelerate universal energy access. And I'm really looking forward to the discussion we're going to have with some great speakers today. The event today is hosted in partnership with HEVOS and Tier Fund and is part of the IID debates webinar series which aims to create a space for conversation and debate on key and current sustainable development issues. If you're interested in getting regular updates on these events, we have a newsletter and I will send a link to that towards the end of the session. We are expecting a lot of participants today and you're hoping you're going to be joining us from all over the world. So while we wait for a few more people to tune in, I'm just going to quickly run through how we're going to use the Zoom tools and get the most from this session and then we can get started. So that is it from me. I'm now going to hand over to Rita Poppe who is the Global Advocacy Officer for Green and Inclusive Energy at HEVOS and our moderator for today's session. So thank you very much and I hope you enjoy the conversation. So thank you Juliette. Welcome to all of you and thank you for joining. We have a strong set of speakers today and an impressive number of participants and that's even counting. As said, focus of today's session is an exchange on the design and key components of subsidies, energy subsidies to accelerate universal energy access up to now significant significant progress has been made in improving access to electricity. However, some 800 million people still don't have access and for clean cooking the picture is even more grim 2.8 billion people worldwide still go to go without clean and safe cooking solutions. Again, it's 2.8 billion availability of finance in the right format is one of the major tricks to closing the energy access gap. And within finance, subsidies can play a major role. The use of subsidies is currently often neglected, but key players sharing away from using it. This event will cast some light on how subsidies can accelerate energy access and how it should be designed to do so. For this, I would like to present you our speakers who will contribute to this discussion, each from their own perspective. First up, it's Stephen Nash, who is the founder of Kugana advisory and has been working with different on subsidies research. He's followed by the Punika Pereira, who is a researcher in IAD shaping sustainability markets research group. And after the two presentations, we have a panel consisting of Satish Kautam, who is the national program manager of renewable energy for rural livelihood in Nepal.  And we have Daniel Kiedmer, who is the finance lead at Africa Minigrids Development Association based in Nairobi. Finally, we have Dana Riesemkova, who is the global lead for energy access at the World Bank. So this will be our set of speakers and panelists. A quick overview of the event. We will first start after this with a poll, quick poll to have a sense of your subsidy feelings. Then we go over to Stephen and the Punika, who will both present recent research, and then followed by the panel as I described, just described. And then we do another poll to see how successful the panel and the speakers were in explaining you the usefulness of subsidies. And then we close with a Q&A session in which you can participate as explained by Juliet for all your burning questions and you can pose your questions already throughout the whole session in the Q&A box. So we're looking forward to a vivid exchange. And first of all, we go over to the poll. So the question is, do you think energy subsidies are essential for achieving universal energy access, and the answer options are quite straightforward. Very much so, somewhat slightly or not at all. Okay, just sharing the results Rita, can you see those. I know, I don't know why but I can't see them. I can run through the results for you so we had 73% at very much so 26% at somewhat and 1% at slightly 0% not at all. Okay, so there's a clear preference but still somewhat to gain, I would say. So, I would like to continue by inviting Stephen to present the findings of his recent research. Thank you Rita for the introduction. I'm just going to share the slides which are hopefully up now. Okay. So thanks very much for the introduction Rita. As Rita mentioned, I'm going to be presenting some research that recently been carrying out was commissioned by Tier Fund. Tier Fund is a Christian Relief and Development Agency, focused on ending global property. Research is part of their work on climate change and energy access. And this is actually the second paper, following a paper that we published in 2018 that was looking more generally at the role of subsidies in accelerating progress on energy access. So I'm going to start just very quickly with a little bit of context which I'm sure that most people on the webinar are familiar with.  So Sustainable Development Goal 7, the goal of reaching universal access to energy. So on trends in 2030, we would still have several hundred million people without access to electricity. And as Rita was saying, a far bigger gap without access to clean cooking. The current COVID-19 crisis really puts this into focus. On the one hand, the challenge is greater than ever. On the other hand, this week, you sales data for the first half of this year come out of Goggler showing a big drop in in sales, but at the same time we also know of course that energy access can be a great facilitator of giving communities more resilience to cope with some of the challenges that are being posed to them at the moment. So, what are the subsidies then and how they can be used. So, I think when we're talking about subsidies, the first thing we need to be really clear on is what we're trying to achieve with the subsidy. And normally we are talking about one or two things or more frequently I guess a combination of the two. The first one is high cost of service. So in a lot of the areas where there remains a large energy access gap in rural areas, for instance, the cost service is higher. So CAPEX might be higher and the ongoing costs of servicing those areas might be higher. So there might be higher customer service costs higher maintenance costs and so on. So of course it's not unique to delivering distributed energy access solutions. It also is the case in delivering a grid based electricity system. It's also not unique to developing markets. So this is a challenge when you're running an electricity system in a developed market or middle income country as well. And these differences importantly will persist. They'll remain the case as is illustrated by looking at those more developed markets. The other thing that we're often talking about when we're looking at designing subsidies is looking at affordability gaps. So also in these rural areas we tend to have communities that have a lower ability to pay for energy services. And so there's an affordability gap. Now hopefully this affordability gap will decline with time as households increase their earnings and so on. But some of those affordability gaps may persist. We see in more developed countries that energy poverty remains an issue in many cases for instance. Now, because in both of these areas to a greater or lesser extent these issues may persist. Maybe me talking about subsidy design is a little bit of an error at the outset of this discussion. Maybe you should be thinking about this more as a market design challenge. And how do we design a market? What's the target model look like in let's say 2030 or 2035, not just to deliver energy access and to close the gap, but to sustain that performance as well. Before I move on, I just want to very quickly nod to a few other reports and trends over the last couple of years. I think the debate has moved on a long way in subsidies and the poll that tested to this as well. That we've done this presentation a couple of years ago, I think that would have been perhaps a lot more pushback against the idea of subsidies, but the papers that I've got on the screen there is definitely not a complete list at all. There's lots of literature out there and it's growing all the time some great work, but, but there's a lot of work being done and it's obviously a lot of debate around what the subsidies look like, but increasing acceptance that we need to use them or as I was saying maybe it's not quite some sort of intervention or market design intervention that helps us to deliver energy access. Now we talk about subsidies. It's not quite that straightforward. There are lots of different design levers that we can pull and subsidy design itself is complex area. I've listed some of the design characteristics or levers that we can pull on this slide. It's by no means an exhaustive list, but just gives you a sense of some of the decisions that you need to make when designing a subsidy scheme. How targeted is that scheme? Where does the funding come from? To what extent does the subsidy just provide a funding input versus providing some sort of results based signal? Is the subsidy only awarded to one or two service providers or is it open to everybody? How is the subsidy allocated and what is the scope of the subsidy both in geographical terms and in terms of the technologies or mode of delivery that it might cover? So as part of the work for tier funds, we've looked at experience across a wide range of subsidy schemes to try and understand what lessons we can draw from those, what has worked and what has worked less well. We've looked across a range of different geographies and that's shown with the map on the slide, but we've also looked across a number of different delivery modes. So we've looked at on-grid electrification, we've looked at off-grid electrification, but we've also looked at clean cooking to try and draw out some of the key messages. A lot of this is covered in the paper but I want to just highlight a few of the really key lessons that are coming out of that. So one is that particularly when we're looking at distributed technologies, sustainability is a really big issue. So a lot of schemes that have been deployed in the past have achieved good results up front of increased connections, which of course is the thing that they're trying to achieve in the first instance. But those results have then maybe not been sustained over time. So when the valuations have been taken, have taken place several years after the scheme was rolled out, people have perhaps lost access to energy, maybe companies that were delivering the services have dropped out the market, maybe the maintenance provision that's in place was not sufficient, maybe there's not enough customer service presence in place. And then those cost of service differences that we were talking about earlier have persisted and there hasn't been the provision in the market to address that persistent gap. Now, we've got a lot of results-based finance schemes that have been introduced over the last few years that again are achieving good results up front. And I think it's far too early at the moment to really understand the extent to which these issues have been addressed with those, but it'll be interesting to see over the coming years to what extent these issues persist or whether they have been tackled. I just want to have a look quickly at one of the lessons from elsewhere and this is looking at an on-grid electricity system and it's looking specifically at Thailand. Now this Thailand example that I'm going to talk through quickly is repeated really in on-grid electricity systems all over the world in developed and middle income countries. I'm just picking the Thai example just because it's particularly transparent and easy to understand. In Thailand there were two different distribution companies, one for the urban areas around Bangkok and one for more rural areas. And this huge cross-subsidy that takes place from the urban distribution company to the rural distribution company every year. On top of that, there is a funding mechanism for free electricity to the leasewell of consumers. Now without getting into the merits or otherwise of these specific policy measures, the point that is really important to take away from this is really that it's primarily funded by consumers and cross subsidies. So this is moving about $350 million every year around the system, most of it just consumer money to deal with these persistent cost of service differences. Now of course we can't just lift this model and drop it into sub-Saharan Africa today. One of the many reasons why off-grid electricity systems are being used in such a way is that the utility systems in many of these countries are broken and not credit worthy. But we can use the model we've just looked at, we can think about that as a target model and then think about how we might step back from that and evolve towards it over time and how the funding sources might change. So acknowledging that on day one, most of the funding would need to come from the likes of donors. Over time, governments might step in to take a greater share of that before eventually moving towards something that is self-funding and sustainable and funded by customers themselves. So I know that I'm just about out of our time, so I'm not going to go through every single one of the policy recommendations in detail, but I just want to highlight two really key points. The first one is the sustainability point and the idea that we really need to make sure that we're clear on what our subsidy is trying to achieve or market design intervention is trying to achieve and how it's going to sustain results over time. So the idea is that probably means it needs to be systemic, it needs to be a standardised as possible and we really need to think about how we're going to scale results to bring in two forms of funding as well. And then the second point is really thinking about where we can draw lessons from elsewhere and not reinventing the wheel where we don't need to. There are also rules that we can draw on from elsewhere in the energy sector, but also of course out from other sectors as well. So you can get a lot more detail in the papers that have been published on Tear Funds website and do of course get in touch with us if you have any questions on that and I look forward to questions during the Q&A shortly as well. Thank you. Thank you for the very clear picture and nice overview of designs and means of delivery of energy subsidies. I would like to hand over to Nipunika, who will give another view on subsidies. Yeah, thanks Stephen and hi everyone. So what I'm about to present next really builds on some of the discussion points that Stephen has already made and looks a bit deeper into some examples that that we've looked in this particular research. So just to start with how we started this work. So in order to unpack subsidies in line with the discourse and how subsidies are currently being looked at. So with support from he was developed this discussion paper, particularly looking at demand side subsidies that aim to reduce cost to the end users and supply side subsidies that aim to reduce cost to the energy supplies who want to enter remote geographies on new markets. It is well known that subsidies on its own is not the silver bullet. It is an instrument that is heavily dependent on already scarce public finance. Therefore, commercial finance has a significant role to play. So this paper also aims to highlight some insights into how commercial can finance need is needed and how it can play a role. And this diagram really, really links to what Stephen mentioned earlier on how we need subsidies and why we need subsidies. And it also builds on several recent publication that aims to present how markets can be segmented based on affordability gap which in this diagram increases as you go from top to bottom and commercial geographic reach which reduces as you go from left to right of the diagram. The SS and the DS here indicate where supply side and demand side subsidies are more suitable depending on these two conditions. However, in reality, this is not always neatly segmented like this. There are overlaps between different affordability groups, living in a specific job in one geographic location, and the economic conditions shift constantly. So to, we will discuss this more in the next, I will discuss this more in the next slide as I go deeper into the Nepal case study. So to look a bit deeper into how demand side subsidies have been delivered, we looked at Nepal as a case in point. Nepal has been delivering subsidies since the 1970s. And there has been commendable commitment from the government national government in terms of allocating public funds for decentralized renewable energy subsidies, establishing a dedicated institution called the alternative energy promotion center, which led subsidy mobilization where donors could also come in and pool finance and align technical assistance support has also created quite a strong enabling environment with dedicated subsidy policies. So while we discussed quite in quite detail in our publication as a case study, I wanted to highlight three key lessons that we found here. The subsidy policy in Nepal define subsidies based on remoteness and included and it also included included an additional subsidy for specific beneficiary groups, which included groups such as the Dalit community who are often disadvantaged as the lowest cost across the country. These two diagrams shows results from a sample survey carried out as part of this research. They indicate that despite having an additional subsidy. Dalit communities were still at the bottom of access levels for electricity and clean cooking, compared to more affluent and higher cost groups. The results give some indication into when subsidies are allocated with restricted program timelines and with sort of a blanket subsidy that allow anyone within a certain geography access them. It can still exclude the most marginalized communities. Secondly, having systemic processes for quality assurance for technology verification or technician verifications qualifying companies really helped at the start to create a strong market for various intermediary technologies. However, complexities of these processes and delays eventually created various bottlenecks. So for example, energy companies ended up increasing retail price to end users to cover high administrative costs, or the delayed payments, if impacts of delayed payments, the sort of defeats the purpose of a demand side subsidy. While various steps have been taken over the years by in Nepal to address these issues. Identifying this risk sooner with periodic assessments could have potentially help mitigate some of these negative impacts on the ongoing programs and also weaving it better into newer programs. And finally, Nepal has benefited significantly with long term donor and government funding for subsidies compared to Sub-Saharan Africa. Nepal has contributed to creating a market with many local companies. However, it eventually also became very highly dependent on subsidies. For instance, some energy companies really struggled to survive following reductions of donor funding that was going into subsidies. Moving on into supply-side subsidies, we also looked quite closely at emerging lessons on resource-based financing or RBF as it's commonly known now. And it's becoming a very popular tool, particularly in Africa currently. It's focused mostly on geographic targeting. In the report, we present several lessons from end-of-RBA facility and recent programs such as the Kenya Off-Grid Solar Access Program. But to share a few key insights, RBF is indeed a very important financial tool to activate new markets. But alone on its own is not sufficient to establish long term market viability. It requires backing from affordable commercial finance where companies can rely on to pre-finance their products and services until RBF gets released to them. It has the opportunity to improve quality assurance, but there are also risks similar to what I mentioned from Nepal that could affect delivery. Most companies who have so far benefited from RBF are larger international companies with very strong track records and finance to expand to remote geographies. So if you want to unblock finance to smaller local companies, there is a need to really combine this with technical assistance support and even to look back at some of the processes such as qualification processes of companies and making them less stringent. Finally, we know that RBF is a market enabling instrument and is not often intended to reach the poorest communities. But there has been some really interesting examples that's been coming out from the piloted schemes. For example, in Malawi, RBF has been combined with a demand-side subsidy to target much in those communities. So moving on into some of the key recommendations there that came out. So lessons in this study so far really indicated to us that there is a huge need for better targeting. Because if you understand who needs subsidies the most, you can combine a mix of financing instruments for different end users based on their affordability and the context they come from. For this you need to define better, but understanding the context and the end user needs and to develop appropriate subsidy delivery mechanisms that don't exclude marginalized communities require more effort. Different end users have different marginalizations and they face different challenges when accessing finance. For instance, the Dalits in Nepal tend to migrate more frequently and this might have affected their ability to provide identities or certain types of proofs that are needed to access the subsidy. So for this, the elephant in the room is sort of data, the availability of data. Data often becomes a significant bottleneck when designing financing instruments. But there are different approaches that are already there to address data gaps. For instance, participate in localized planning approaches are key to co-generated to be to be able to co-generate data with local communities and local stakeholders. For instance, in Nepal, the FCDO funded Nepal renewable energy program is currently working with provincial level and local level governments to link with existing planning mechanisms. And local governments in Nepal already have and have been using a participatory planning approach every year. This provides an entry point to integrate energy into an existing mechanism. IID is working with various partners using the energy delivery model planning approach to engage with communities to understand their priority needs, build ownership and develop solutions that are tailored to the local context. The solutions eventually include delivery models and financing options specific to target groups. So collecting data using such tools can be a bit exhausting and it's expensive, it is time consuming. But what we see is that investing upfront in these type of approaches can really help to avoid failures and unintended consequences that could rise in the future. And at times you don't even need to reinvent data. The Malavi example, the RBF Malavi example for instance really showed that there are existing data options that you can use in Malavi. The RBF program is the government's social cash transfer program to target marginalized communities and it did it quite successfully. Secondly, identifying and managing risks early is critical when deploying time bound financing schemes. So just to go, I've touched upon these in in the past slides but just to very quickly look at one of them. So lack of end user awareness was raised as a potentially a very strong barrier that might have avoided marginalized communities such as Dalits to access subsidy related information and eventually benefit from them. Another benefit of knowing target groups better is that you can develop a mix of financing instruments, more affluent households can benefit from credit based instruments. And there are some examples that we have presented in the study such as work by UNCDF building partnerships and providing technical assistance to commercial banks, energy companies and microfinance institutions to provide end user credit. And finally, just to wrap up the presentation, we really looked at the need and the importance of commercial finance. As it is a challenge, it is an area that needs a lot more effort and understanding that is limited success stories and countries like Nepal are still struggling to blend in commercial finance into decentralized renewable energy. But we know that as donor funding is being more threatened due to issues such as the pandemic and global politics. This is really an area that needs further effort. I'm just going to end it here. This sort of gives an overview of some of our key publications. I'm just handing over to you, Rita. Thanks a lot. Very good overview of key lessons, best practices and good inspiration for the rest of the event. Now that we have listened to the presentations, I would like to invite our panelists to respond to the findings by Nikonika and Steven, based on the challenges and opportunities of public funded subsidies from each perspective. And each of the respondents has about three to five minutes to reflect on the presentations and findings. And I would like to ask Satish from Nepal, can I start with you? We know that in Nepal, government has always contributed significant proportions towards decentralized renewable energy subsidies along donors and is continuing to put more funds on subsidizing, minigrazing cooking solutions. Having listened to the lessons and recommendations highlighted so far, what lessons are useful for Nepal going forward? What, for example, might be useful for new frontiers in Nepal like clean cooking and productive uses of energy for some of the poorest? Thank you, Rita. And thank you, Nikonika and Steven, for your nice presentations. Let me trace back a little on how the subsidy mechanism in Nepal evolved over the years. And as Nikonika mentioned, like it started in 1975 as soft credit for agricultural, through agricultural development bank, basically for, you know, milling purposes, migratory was developed for milling. It was only in the 80s when the government de-licensed power plants up to 100 kilowatt. Then people started putting up migrators for electricity generation. And then the following year in 1986, the government started providing subsidy, 50% of the cost of electricity generation and distribution. This was basically to encourage electrification through these existing water mills. After that, a lot of innovations were seen in Nepal, like the local companies, local innovators, they came up with different designs of power units. One multi-purpose power unit was very famous and then they started using induction generator as, induction motor as generators, brought the technology for electronic load controllers and so on. And it went on for about a decade. And then the market kind of saturated and the government started using subsidy, this subsidy that was allocated for migrator for solar home systems. It was only in 1996, the apex government body called Alternative Energy Promotion Center was established. And then the subsidy and credit kind of separated before it was handled by agriculture development banks. So subsidy and credit were together. They were processed by the same personnel. When subsidy and credit was separated, it became very difficult for rural people to, for timely financial closure of their projects. It used to take years to come up with the fund, total fund, because subsidy only covered 40, 45, 50% of the total cost. And the project cost over and there were delays, some projects were like half completed and then they had to wait for years to complete it again. And so on. And what we found is that in a recent study carried out in two provinces of Nepal, that more than 90% of these projects are still operational. They are providing electricity but the quality of electricity and reliability is very poor. Because the people did not, could not make financial closure on time like they were using local materials, local laborers and you know local hardly skilled people to build canals and using like untreated wooden poles and so on. So, from Nepal's experience, what we find if you look at these micro hydros is that in it really depends on the community, communities past experience the communities capacity, financial capacity, also capacity to manage the conflict in the community, etc. is directly correlated to the condition of the power plant. If the community is well off you'll find the power plants are better run because operators are better paid. And there might be opportunities for productive end users in poor areas the opportunities for productive uses are very limited. The tariff rate is very low operators are not paid properly so they run away and there's high turnover of operators, and so on. So going forward. So, in Nepal now because mainly because of the rapid extension of the national grid. Over 95% of the population has access to electricity so those people who are left are like really in remote areas or they are under the grid very very poor people and these people definitely need subsidy not only subsidy but also a lot of technical support to be to be able to have access to electricity, both for lighting and for cooking also. Thank you. Thank you Satish. This is very insightful. I just name a few of the things you mentioned, because I think they're very interesting. And on the other sectors like the agriculture sector finance experience community capacity and as well as like subsidies as well as technical support in their combination. Just to keep in mind. Yeah, then we move over to Linda Linda. I'm a Canadian energy company and in all great solutions mainly. So can you explain a little bit more of what the role is of small businesses here. And what resonates with you, based on the presentations in terms of challenges and opportunities you face when accessing subsidies, like RBF and accessing finance in general. Okay. In terms of the challenges that we face as local businesses. The biggest by far is finance challenges and most small local companies do not have a not properly resourced, and they don't have the finances that would enable them to you know, take, take advantage of some opportunities that are offered in terms of, you know, like results of this financing. We have a program that's currently running the Kenya off grid solar access program course up, and it offers suppliers side incentives to encourage distributors to go into, you know, counties some 14 counties that are off the grid. And if we look at these counties, you know there's possibly populated very vast and the population there the people that live there are very poor so it's very expensive to go into those areas and unless you have other resources. It's very difficult for local for local business to, you know, even take advantage of such opportunities.  The other challenge is the technical capacity a lot of local companies, because of the financial challenges they have they also don't have that's, you know, and they cannot afford to hire top talents in terms of capacity building to offer that kind of technical support that they may require. So that's another challenge that we face on the supplier side I see a challenge even with the with this subsidy schemes because the current scheme, the course of scheme that has been designed targets, you know, those difficult to reach counties and it so happens that they're the counties with the highest poverty incidences so you know as high as 80% in some counties and targeting only supplier side incentives and not offering demand incentives really is a difficult balance because these people already don't have the financial means to pay for these products and we have, you know, like there's a category that I like to define as hardcore poor where you know people that sometimes even have only one meal a day such a person even with incentives and you know even if you give them interest free loans, they cannot be able to pay and it may, you know, give rise to the concept of probably they they require freely distributed products. How that can be done is another discussion that I believe local governments can come in and support in such cases. So, in terms of the opportunities that exist. So, you know, I would say local government supporting the extremely poor people that cannot afford products. We also have this extremely poor people in counties that are not classified as costs up so other counties that if you look on average people you know a wealth here but there's that portion of people you know that cannot afford so they are remote potential markets but they still have poor people we have people in informal settlements or extremely poor families that may need demand side subsidies. There's also more opportunities when it comes to partnerships. So I would say partnerships between these larger organizations that are better resolved with the smaller last mile distributor companies that have better local networks and can easily access this last mile counties without much requiring a lot of money. Also opportunities between you know other organizations such as local NGO that may already have networks local networks in this difficult to reach places and they can support with that. We currently have the course up to which is the second round of the result is financing. That's going on which is also aimed at it's an improvement of the first one this one is aimed at encouraging more local companies to take up these opportunities so the barriers have been significantly lowered. And there's an incentive to encourage you know initially even before you start claiming the results based financing there's an incentive to go into those markets so there's a there's a portion of the financing that has been made to incentivize local companies to take up this opportunity. And I believe there's also an element of more public finance, which may be both, you know, in addition to government may be larger local corporates. That's, you know, may have additional finances to being able to support these markets with, you know, supporting both the supply side and the demand side subsidies so through probably CSR opportunities where they can take up part of the costs and offer them. So the product cost subsidies that may be required for the extremely for people. So, I would say in a nutshell, that is from my experience. And it's a difficult market that we operate in as, you know, suppliers of these energy devices, because the cost of operations are really high and just, you know, operating in a market. The deal market scenario may not result in this organ in organizations being financially sustainable. And, you know, commercial financing may not be attractive where companies not making money. So, that's also part of the discussion. It was very good to hear your on the ground experience, so to say, and it's a pity that we short of time because I think you can go on and on and explain all kinds of things and it's very good to that you like paint this picture. And while it's good to hear the challenges as well as like new things that are developing like the course up to which you explained. So that's good news. Then I would like to hand over to Daniel for Daniel from and that from East Africa, Daniel, you are working with many many great companies delivering energy for all. So what lessons from Nepal are useful when thinking through subsidies in the countries that other members are operating in and how are the financing context for energy access difference over to you. Yeah, thanks.  You know, we are essentially looking at a fairly new industry, possibly, you know, five to seven years away from what the solar system sector used to be.  And also we are looking at lower consumption. So, you know, in average consumption across the continent is about six, six kilowatt hours per month.   But there's a lot of support in terms of incentives that, you know, African governments and, you know, with donor partners can play within the mini-grid context. We've seen a lot of progress within the RBF, you know, programs. But I think for where we're looking at in terms of the mini-grid model, there's, you know, another step the, you know, governments can go and in this case you're looking at a possible subsidy on the per kilowatt hour pricing, which is, you know, the tariff in which the mini-grid operator would charge the consumer. So we're looking at a situation where a lower, you know, kilowatt hour pricing can stimulate demand uptake from the consumer. And this, you know, this is not something that, you know, is new, but I think in some markets we've seen some prototypes being tested. I can give an example of Tanzania where, you know, one of our partners, Crossland Innovation Lab has been testing prototypes around how a lower kilowatt hour pricing can stimulate demand. And we've seen, you know, up to in some cases three times the growth in demand from consumers. So we are seeing a situation where consumers are really price sensitive. But also I'm seeing a situation where governments can support from the area of financing appliances, because a lot of these communities are, you know, rural and dependent on agriculture. So those types of agricultural inputs or assets that can actually consume with the energy the mini-grid provider is providing would go ahead and increase in both increasing consumer incomes, improving the quality of life, but also increasing load, which ultimately also benefits the mini-grid developer in terms of revenue. I think those are some of the situations I see. But also the other area that the government can support, not only in terms of the funding side is also around the issue of regulations because the regulatory environment in some ways may inhibit unlocking those types of capital. For instance, we are seeing in most markets it takes, you know, I mean an average mini-grid company up to a year just to achieve regulatory compliance. And in some cases they are in danger of missing out on the funding deadlines for some of these RBF facilities. So, you know, lowering that operational and administrative bottleneck in terms of licensing and approvals that can go a long way in terms of unlocking more capital for some of these projects. Yeah, I think I can stop there for now. Okay, thank you, Daniel. And thank you for bringing in a few other perspectives as well, like regulation, national government and energy prices. So I would like to hand over to Dana, our last speaker. Dana, the World Bank is exploring and putting forward funds for both supply and demand subsidies. And the discussion definitely shows that subsidy design is complex, but they are essential to reach everyone. So what are the key highlights you would take back from the discussion today and what can the bank do more to promote the use of subsidies. So over to you. And thank you very much for this opportunity. I will have to ask my daughter to be quite sure. And I can go. So, thank you. It's, I have so many insights that I'm taking from the discussion that it's, it's almost hard to know where, where to start. So thank you to Steven, to Nipunika, and as well as to panelists for these really useful insights. I would start by saying that it's incredible how much this discussion has moved that we had the, I think, as Steven alluded to it we had this discussion four years ago we would have, we wouldn't have this discussion because it was almost a bad word to consider and no one wanted to touch it. If we had it two years ago, we would have a very opposing view and fierce fights and you have it today I mean it's, it's almost to say it's getting a little boring because I think there's a lot of consensus that is being reached and a lot of agreement and I would also say it's getting more interesting because you know discussing if and when but we're discussing really how, how to do it in a way that these subsidies really are effective and reaching ultimately everyone and helping achieve the universal access. So maybe just, just starting few things that I found very interesting I found Steven's presentation, very, very useful in terms of bringing this, this larger picture of subsidies and, and I think what is important you know we sometimes are so much lost in the present that we forget the past and what he brings back is also the discussion that actually I remember when I started in an access space about 20 years ago there were a number of papers written about subsidies and cross subsidies for the poor by the time everything was focusing on the grid because the grid was basically at that time almost the only way how to how to how to reach people and everything was focusing on grid electrification but there are lots of examples that come from that time that could be used for off grid as well but there's been quite a bit of shyness to apply these principles to off grid electrification and partly for good reasons but partly also due to misconceptions and one of them has been that you know when the off grid solutions came in there was just a lot of enthusiasm because suddenly for the countries that have not been able to expand the grid like most countries in Sub-Saharan Africa there was an alternative there was a solution that was modular was cheaper and I think it was just a lot of believe that the solutions in the markets actually will reach everyone. What's happening is that we now know that this this is not happening and this will not happen the markets on that on its own are not going to reach everyone and therefore there is a case for subsidies but there is also a legitimate concern about large scale use of subsidies in the off grid sector and this is as Daniel mentioned I think in the minute with sector and it was the first one that has moved into embracing the need for subsidies because without subsidies the price to the consumers is just it's just too high and so this notion of viability funding or performance based grants that are needed to reduce the investment cost is very much accepted but in the standalone off grid sector the problem is that it's more it's more complex actually because it's based on the company selling products to the household whether they sell them on cash basis or through face you go you still you still you still selling them and if suddenly a program comes whether it's the government or donor this basically reducing the prices for these products for some households, especially if it is sort of subsidy that is targeting to reduce the cost of these these products for some households the other households may stop paying for the product even though they can afford them so be a little careful that you know in the end the subsidy programs do not end up slowing down the pace of electrification, but rather increasing it especially if there is just not enough money to subsidize everyone, which often is the case and if the subsidies may be short lived because then you know what happens when they when they expire and for this reason I think what we have seen up to up to now is that of good sector has much more embraced supply side subsidy as well, such as result based financing as I have been mentioned the World Bank has been building up the financing schemes for off grid solar and providing enabling working capital for for the companies and I was I was very relieved to hear from Linda that you know the second round of cost which is well bank funded program has actually implemented a lot of lessons learned from the from the first round and it's more accessible to local companies, but but the focus has been on on on eliminating the supply barriers. And that's that's a part of the story. And it is an important part of the story because if the products are not even available in the area where people live of course they cannot get it and they're then they stuck this either low quality products or kerosene or other things so it's an important step but it's not, I agree. This is not enough and in fact, in the in the research to be published this February in the of good market trends report we have estimated that to achieve universal electricity access by 2030, the upgrade sector will need to serve over 600 million this at least tier one product. And we have also warned that if a combination of supply and demand side subsidies is not available, then some 230 million people will not be reached either because they live in a remote locations that are too expensive to serve, or because of the affordability constraints and in many cases actually the because of the two combined constraints at the same time. I don't remember this was this research was done before COVID so now if you, if you add impact of the COVID but this number will be much larger, especially since it's estimated that you know that COVID could push as many as 150 million people back to extreme poverty. So, to reach everyone this affordability issue has to be resolved and demand side subsidies will be needed. And the question is how to design them so that they really are helpful, rather than harmful and it is something that really is something we are looking into and we are very happy to work with everyone who is, who is involved in the, in the similar research. I know I have to summarize so just quickly. I think what we need moving forward is to, is to do this right build more understanding and more evidence of what works and why and this is why the research by Nipunekata specifically looks into, into one country's team is so useful because it goes into those details that I needed to understand what worked and what didn't work and why and sometimes it's what design is the implementation issues, sometimes it's what's missing. Not just what the, what the, whether the subsidies were designed or not. I think second, we need to sort of get away from this silos how these subsidies sometimes are designed to offer people trying to figure it out on their own clean cooking on their own great minute rates and try to come up with help government to design the schemes that actually can work across different technologies for and this is where I thought that the students research was pointing in the right direction. More, more examples on the ground. More, more, more real examples and evidence and learn from that how to target especially how to target the men side subsidies. The key issue of the men side subsidies is targeting and if they are mis targeted they can do more harm than good so how do we actually learn how to use the data that exists as we discussed today, how to improve the scheme that exists to really get this targeting right. For something we need to look at innovative way into sustainable sources of financing for subsidies, you know, impact bonds could be explored more. The research that we have just published on clean cooking we have come up with the estimate that the combined health, gender and climate externalities cost 2.4 trillion dollars every year. So, you know, how to, how to monetize these benefits how to get sustainable funding out of basically these externalities is another area that that could really open up many, many new avenues and I think this is we need a good coordination among everyone who may be developing the subsidy scheme so that we don't undermine each other so so so thanks a lot for this for this forum because that's exactly the way to get there. Thank you Dana. Thank you for all your, your nice recommendations and overview of the challenges and also some history. So we discussed a bit but we would like to skip the second poll because we are a bit short in time and we would like to provide some time for questions from the audience. So, if all the panelists could turn on the camera and then we have quite a bunch of questions and if you have still have a few you can post them in the Q&A box. And I would just like to start from the beginning and you can vote them up, but the first question has quite a number of votes as well. So this is on cooking and knowing the big gap for clean cooking. I think it's, it's good to start with cooking. And the question is from Martin Price, saying, could you say a little bit more about the design of good subsidy programs for clean cooking specifically, and it's addressed to Steven. So Steven, can you answer that question. Really, and I'm certainly don't have any silver bullets on this.  So there's so many kind of interacting parts and it's in many ways I think much, much more complicated than electricity. Well, I would say from the examples we looked at is it again illustrates some of the challenges, though, that I talked about. I mean, we talk about, we looked at examples in Nepal and in India on a clean cooking side, and I'd be interested in that Satish has something to say on this, but in India, for instance, who saw subsidies for LPG stows in the first place, but then you're not really thinking about the sustainability again, and the subsidies for the fuel itself over the long term. So again, we've got essentially affordability gaps that may persist for some time that they were not properly addressed by subsidy scheme and it'll be interesting to see how the results from that scheme kind of evolve over time, especially in light of the economic challenges being placed now. But I would say one thing on what we could do. It's very difficult to start saying in the clean cooking space to really define what the system is, you know, a grid you can draw a regulatory ring around and define the set of regulatory and commercial arrangements to try and get the money into that system. With clean cooking, it's much more difficult to define the boundaries of that system. But I don't think it's beyond our sort of collective abilities to work out where we might draw that family. So how we define a set of regulatory and commercial mechanisms to get the money into that system. So think of the nuts and bolts you kind of think about it in the same way, but you know, we're a long way from cracking it. So it's a collective responsibility. So everyone would like to move over to a question from Johanna Galan. Again, sorry, for Stephen again. So you can continue. How would you envision a cross subsidy to work in contexts like Africa but operate solar is targeting very low income and utilities already under immense financial pressure and what could a long term model look like in your view. So it's a great question. And I think in my presentation, I sort of acknowledge that you certainly can't take the tie example that I was talking through, for instance, and just drop that into sub-Saharan Africa. It's not, it's not getting to work for precisely the reasons that Johanna highlights. So what I'd say, and I certainly don't intend to have all the answers on this either, but we do know from other sectors and from things we've done in the on grid electricity sector, for instance, generally how to create sort of credit worthy entities and to tackle specific problems that sit behind that credit worthiness and tackle them with the different financial instruments and so on and insurance products and all the rest of it that the advice have available. So it's simple and of course it varies from country to country how deployable some of those instruments are. But I would say that at its heart, we kind of need to find a way of perhaps creating some sort of separate vehicle that can be used to administer subsidy to achieve energy access goals. So it might be sort of, it might be regulated, but it might be at arm's length from government and the source of funds that goes into that vehicle will evolve over time. So as I was saying with the phased approach in one of the slides, you would move from a situation that initially is very much dependent on donor funding, but eventually has that target model in mind. And in some countries, you might get to that target model quite quickly. It might be on decadal timescales in other countries, particularly in some of the subjects. Okay, thanks. I would like to move over to question from Christina singer. I have a feel like answering. She says, I would appreciate the discussion around what is lacking in getting the commitment of public finance, national budget and donors to the policy and budget if necessary for subsidy subsidies to close affordability gap. So I would say Daniel and Nipunika maybe. I think for where I look at things is in some governments, I think there's still like a philosophical, you know, argument to be made around the need for, you know, utilities 2.0 in that, you know, decentralized grids are actually the grids for the future.  And we've seen a lot of pressure in some countries where, you know, the public grids are really pushed further and further into areas that are really deemed non commercial for the type of infrastructure they have.  I think on the, on the second, you know, question around RBS.   And so there's a need to, to increase the deployment rate, if I must say, and also, you know, the challenge, you know, aligned with that is, you know, some of the programs actually take quite long so that the lag in between, you know, is structuring some of the programs to actual deployment and actually targeting the deployment to move to more capital allocation to actual projects rather than other aspects of projects, you know, like regulations and others which also are important but also ensuring that that efficient allocation is quite key, if I may say from a minute perspective. Okay, thank you Daniel. I'm afraid we run out of time so we can't have the other panelists responding and do more questions although we had quite a nice set of questions, but that is promising for a continuation of the debate I would say. So I would like to wrap up by thanking all of you, the speakers, the presenters as well as the participants and all the interesting, proposing all the interesting questions for your participation, and I think it was a very fruitful exchange. And it's very promising to continue this discussion and to try to put our words into practice and see in the coming time how we can change things. So we did have a second poll so if you're willing to and you still have time you can take the time after I have ended the session by filling it in. And with this I would like to thank all of you and see you next time. Thank you.


Okay, thank you, Kenneth. This site presentation is a bit of a regular now for Compute Canada. I'm presenting this slide deck in cooperation with my colleague Maxime Vachonneau and also Ryan Taylor has in the past contributed to these slides, so I'm mentioning him as well, but he's probably sleeping because he's in British Columbia where it is around 6.15 a.m. at the moment. And we're speaking on behalf of the Compute Canada Research Support National Team, so that's a team that is mostly responsible for installing software, but there's some other tasks as well, namely the ticketing system and the user documentation. So our presentation outline is where we were, where we are in Compute Canada. There's supposedly a lot of changes going in that we're going to be reorganized in the coming years, but that's a completely separate topic. Our software installation, our Goal and Design Overview, the tools that we used, which is a combination of CVMFS Gen2, which used to be Nix, an easy build of course, Elmot and Python, and we do monitoring, a little bit about monitoring documentation. We don't have time for demo this time, I accidentally left that in. So what is Compute Canada? So Compute Canada, you could see as a collaboration between different regions and sites within within Canada that administers the academic supercomputers here. There are other supercomputers too, of course, that are outside the scope of Compute Canada, for instance. The weather forecasting organization, Environment Canada has big supercomputers themselves that also shop in the top 500, but we're looking at the academic site here. And Compute Canada in itself is composed of four regional organizations. Some have sub-organizations as well, but they're out west, we have Westcrete, we have Compute Ontario, for the province of Ontario, province of Quebec, we have Kaukou Quebec, and then in the east, eastern Atlantic provinces, there's ASNAT. About 200 technical staff distributed among many universities, and more than 15,000 user accounts, professors, postdocs, students, you name it, some people in government organizations as well, like research institutes. Across all kinds of disciplines, so we see here a diagram of where they come from. There's a lot of growth in non-traditional disciplines like HPC, traditionally is a lot of physics, computational fluid dynamics and things like that, but we've got a lot of growth now in bioinformatics, in health sciences, in digital humanities as well. So the ARC platform has these four organizations and there are some national clusters and some regional clusters as well. So the national clusters are accessible by all researchers in Canada, of which we have four HPC clusters, they are Cedar Graeme, Niagara, and Beluga, and a cloud system called Arbutus. This was also discussed in the previous talk by Felix Antoine. So when we were getting these national systems in place, replacing a lot more, what we call now legacy clusters, we wanted to have a system that was as consistent and easy to use as possible across all sites. So no matter where users log in, they get a very similar interface, whether they log into Graeme, Cedar or Beluga, Niagara is slightly different because it's not a general purpose system, it's purely CPU-only MPI for larger jobs, so they have kind of a separate stack, they want to go their own way, but our stack is also available in that cluster. So we want to have all software accessible on every site, it should be reliable, performant, and we want it to be independent from the underlying OS stack, so whether it uses CentOS 7 or CentOS 8 or some other Ubuntu or some other Linux distribution, it should still work. We want to automate as much as possible and we want to make it easy to use this large and evolving software stack. So we need a distribution mechanism, we chose early on to use CVMFS, this has been discussed extensively, so it doesn't need much more introduction than that. We have an independent compatibility layer which uses CentOS, it used to be Nix, this was also introduced in the easy presentation by Bob Trohe, we had a presentation from CentOS itself as well and automated installation by EasyBuild, and then we use Elmod as many others. So luckily all of these things were already discussed during the EasyBuild user meetings, so I can go quickly over these slides. We have a process that's mostly user-based, so except for the basics, we usually a software is only installed, software package is only installed if the user asks for it, so we have a ticket, I want to, can you please install software X, why is that for me? And then we create an internal software installation ticket where we install it on a build note, where there's a bunch of installers, about 10-15 people who regularly install software, where we share two build notes that are cloud VMs, we install on the build notes, we can install in our home directory, we test on the build note, and then when it's working okay, we can install centrally and then push it into CVMFS, for which we also have a development CVMFS and a production CVMFS instance, so we can publish it first on the development instance, then test it on the test note or on the clusters directly via a helper utility called PROOT, and then we can publish it to production and then inform the user, hey it's available on the clusters, please let us know if it works for you and then we're done. Sometimes there's software that's too specific or a user asks, can you install this, the latest snapshot, that's just a master checkout of Git repository XY set, and then we say no way we're not installing unstable software globally, you can install it yourselves and then the user, either we installed in the user's account ourselves manually or the user sometimes can use easy build themselves because we have that all set up to to install in the user's home directory as well and then it will just show up as a custom module. So this is the software stack is continuously growing, Maxime has been so kind to update this graph and we started this whole process in 2017 and right now we're, as you can see, it goes all the way into mid-January 2021 where we're now and there's been some some growth spurts when we added new architectures like AVX 512, you can see in terms of numbers, Bioinformatics is the king of a number of modules that should come as a surprise because of the enormous diversity of packages that are used by Bioinformaticians and so so you can see how we now are nearly at 8,000 software packages including all the variety architectures etc. So our design overview, it must look familiar because you've seen a very similar overview in the easy presentation by Pop Tröger, that's another coincidence because they used our overview as a template. So this year what we've done is to is to move from the 2017 base stack with Nix to a 10-2 base stack with the number 2020 because that's when that particular 10-2 version was released while you couldn't call it exactly a release. It's a step shot so it's just a rolling release where we took a random date from May 2020. So we have at the top the easy build layer so there's multiple architectures and we put that in a particular path modules software with 2020 and the older modules are for 2017. Then we used to have some middle layer with easy build generated modules around Nix profiles that looked like a good idea at the time but we got rid of them, it makes the system a bit simpler and we moved all these things that were compiled with Nix and then have easy build generated modules around them. We moved them into pure easy build so all these packages tc, eclipse, qt, pearl, python are all easy build packages now. We used to use Nix for compatibility layer but just like easy we're now using a 10-2 we can into more details in the next slide of why we changed from Nix to 10-2. Then under the compatibility layer there's a gray area which we also put in 10-2 but it can be overwritten using using all the library path if you want to that includes things like the luster client libraries and and lower level client libraries for for infinite path omni path and things like lip ib verbs for the infinite band stack packages like ucx and and lip fabric that are a little bit higher level than ib verbs they are compiled using easy build much like in upstream easy build and then at the bottom we have things that are local those are things that are very intertwined with kernel modules or the kernel itself and some things that we have legal restrictions on like like fast that are not allowed to be distributed even in a restricted repository. So the next part is why did we that like what are the difference between the old Nix stack we had in 10-2 prefix so so 10-2 as was introduced in the 10-2 talk it uses recipes much like easy build except they're written in some kind of best like syntax and they're called ebuilds but otherwise there's a lot of similarities between a 10-2 ebuild and an easy config file. We use them to provide dependencies for scientific applications. We provide ellipsey itself but we also provide many things that that easy build provides in the compatibility layer that are kind of boring and they're not scientifically interesting things like lip xml2 and and many many and many other things that are that that easy builders provide as modules because what they always provide is often too old. We put in the compatibility layer we added to filter depths. We provided abstraction layer as a module so when you install 10-2 prefix you you you get a best script that allows you to enter the prefix environment we use a module instead to to set a lot of the same environment variables and then we have most of dependencies with scientific software stack. So our 10-2 prefix layer is basically a checkout from the rolling gentle distribution namely a file that I downloaded on May 4th 2020 and then we we freeze that there is no version updates except for couple optimizations that customizations that we put in a special overlay repository attempt to overlay where we make we have to make some changes sometimes there are bugs in 10-2 sometimes there are just customizations that are more local. So swapping the module from the older Nix to a 10-2 feels like a Linux distribution upgrade it just makes everything newer it's like you do a YUM upgrade on a fedora distribution and just boom all the basic tools are certainly newer. So the big change is a new libc version change where in the older Nix layer we had 2.24 we now have version 2.30 and that actually has some implications for HPC because a lot of changes went into the libm library in Tlipsy I put a link here in the slide deck later on you can probably click on there when I make the slides public I'll send the slides to Kenneth. So one thing is that that a lot of effort went into optimized math functions like the exponential log power etc the older Tlipsy was extremely precise up to one half a unit in the last place which is the most correct you can get but it paid by switching to multi-precision arithmetic when needed it meant that for some input of the exponential function the exponential functions only went like hundreds of times slower than for other inputs with confused many people and now they made it faster but at a slight cost of precision for those edge cases and actually there are some examples there is one example from Damian who works at where he said that there was a big difference between the libm speed of GNU versus the one in libimf from Intel and those differences more or less went away with the newer libm so it helps it benefits everybody by having just this newer libm than what we had before and then many other things then change like we have a newer bash we have a newer git we have a newer vim emacs etc etc so why why did you switch to gen2 instead of nix and why kind of also reason why easy users gen2 is like in nix you have a very self-consistent system but when you put things on top of nix the whole system breaks down so so nix is very beautiful much like guix but you have to stay within within that environment to make it work properly and the reason is that in nix what you get is a is a sim link forest where you have a profile you you get things out of that profile but the profile itself is a sim link the version points to a generation like in this case 523 which in turn links to a hashed path and in the hash path you have the actual binaries which in turn point to the hashed location of the of the package in this case core utils for ls now we wrapped ls sorry the ld the the linker so that all paths point to this to nix user profile slash lib so the the first location here but what happens is that when you upgrade a package because some bug fix or something like that it gets the package gets installed into a completely new nix store location with a different hash and what sometimes happens that these hashes they they leak through the environment even though we only advertise these uh the the top lever top layer link top level link what happens sometimes that software packages they resolve sim links and we have these store locations uh leaking into user make files or something like that and and what what happens is that when you upgrade a package we can garbage collect the old location then the old location no longer there and things break down on the user level uh and that's fairly hard to control and we found out that tend to prefix it doesn't have any sim links it just stores directly in the location and we kind of eliminate that so tend to previous like the minimal solution for what we need whereas nix has a lot of extra things which is in principle useful but it sometimes hurts us now on top of the tend to layer we have what we call um software standard software environments uh put a link here right explain in our wiki um and so some combination of modules is loaded by default uh on clusters so that when people uh people can can use gcc and into compiler as soon as they log into the cluster and they don't have to to uh load any modules themselves before using a compiler uh there is some disagreement about that some people like it that that no modules are loaded by default and people should get a minimal environment and they have to load whatever they need but we had some majority that said okay we should load something by default which makes it a little bit more user friendly um and we call those standard environments it's possible to go outside that structure you can we also have like Intel 2017 installed and Intel 2019 uh but that's the default which usually uses uh c and and it's the what we compile the most packages for the other ones we install only small subset of net health packages for people who have a need of different compiler versions so so the workhorse for a few years has been uh 2016 compiler and tc 5.4 on the cluster c there in graham that were born into production in 2017 uh on the newer beluga cluster uh and partly on the agrar as well we we use tc 7 3 and Intel 2018 uh with open mpi 3 and and that's still using the same nix stack as the 2016.4 standard environment and now in uh we're working on the newer gen 2 based environment that we're making the default in april this year and that's that uh that actually corresponds to upstream easy build iomkl 2028 so it's kind of a mix between fos and intel uh it's mostly the intel toolchain accept that we use the open mpi instead of intel mpi because we find that that the runtime architecture uh works a lot better first with open mpi than with intel mpi and so we have 10 to cc 9.3 uh intel 2020.1 and open mpi 403 uh we have multiple flavors for architectures except for the system toolchain for which we uh which we have architecture independent so it uses generic ssc 3 uh one thing we've also we've changed versus the um versus the 2017 stacks like the 2018 and 2016 standard environments is that we now have fat binaries using the intel compilers uh that helps from clusters that are heterogeneous uh we now have particularly the cedar cluster uh it has login nodes which are haswell or brodwell and it has a lot of newer worker nodes that are skylake and so we want to make sure that the binaries are optimal and sometimes users submit jobs and they don't know if those jobs end up on skylake or or haswell processors so we want them to to have optimal performance no matter where they end up um some other things that we what we do a little bit different from upstream is that we have a lot of modules at the core level we compile them with tc core 930 but they end up in the core um part of the module hierarchy and then we even have that for things like r julia many b bioinformatics tools basically anything that doesn't use either mpi boost fortran asia 5 f ftw is heavily factorized that can benefit from the intel compiler we compile with tc core and it ends up at the core uh module hierarchy so that's kind of different from upstream because upstream has a separate hierarchy for tc core uh with a version and we can make that work by everything by a central lip standard c plus uh library and lip t fortran library so by doing that we basically take advantage of the fact that lip standard c plus plus has first and symbols so it is uh it is backwards compatible and and this this makes this makes it possible to collapse different tc core versions into a single location it's a little bit tricky but it helps users who compile their own software will just use it always use the same lip standard c plus plus so so we don't have things that accidentally try to load different standard c plus plus versions from different locations because they were compiled at different times another thing we do differently from upstream is that we install intel mkl at the core level which means we get all the functionality except for the mpi wrappers of ftw and we even have a toolchain that i i stole from from julie which is called cc core mkl which combines cc core but it can use mkl because mkl is installed at the core level we can combine the two for for the packages r and in particular we use our path but we don't use easy builds our path support we use our own linker wrapper so it actually uh it's also used by if users compile their own software if the the wrapper looks if it links to things that are installed via easy build and if so it injects the r path in the binaries we use our path now instead of run path so we can no longer overwrite this with all the library path however the advantage is now we have our path inherited by a runtime runtime plugins that's a bit complicated but run path sometimes is not inherited and that means that we lose the information in in in shared objects and that gives us very annoying blocks of time so our path has that order nice so um with standard mf 2020 we have a bit of catch up to do we had 800 different software packages installed in the older stacks so so we had to recompile pretty much everything except that we thought okay let's put in some priority we we made a list using module logs we identified which packages were used the most often and then we made like a top top 800 list and we started from top to bottom and say which ones needed priority and recompilation because there's often users who ask for software packages to be installed and then they use it for a bit of time and then the user graduates or disappears so there's no reason to install it to install we install everything we just need the ones that users use um we have some unresolved issues that are mostly about just upgrading software that that's what happens over time uh we found some some some bugs like saying oh we use a tech number in open mpi mpi sent an mpi received that was like a tech number of one billion and suddenly didn't you didn't work anymore and actually that was not permitted by the mpi standard it was not it was out of spec but it just happened to work with with older mpi which uses open ib and now if you see actually it doesn't work but anyway this is kind of the things that we we keep running into when software is upgraded people find that the software for some of your reason doesn't work anymore it's just what happens as software gets upgraded we still have some some things we need to work on too uh parallel IO doesn't work optimally for some people that we that's always a complex issue there's there's some library that can speed up communications collective uh on collectives using the age call library uh we installed in the compatibility layers the melanox library but we found some issues with it so we work on that and we also have virtual gl in the compatibility layer uh for using um gpus on working notes that's uh so the minor things we still work on um another thing we we've used a lot more in the newer stack is to have python extensions with multi depths so if a python support exists we install with python 3 6 3 7 and 3 8 so it's not just upstream easy build is one python 3 version plus python 2.7 we have multiple uh we installed for multiple subversion of python 3 so in this case 6 7 and 8 we're we don't we don't have 3.9 installed yet uh because we haven't had anybody asking for it yet I'm sure it will happen in the future um so examples are things like boost to us uh to kikis and open cve many other packages so what people do is that they load the module qt 5 for instance and they uh get by qt installed as a python binding so so we we just make that easier for the user and to say okay we want to to to load python module and have some python functionality of the other module and then we don't say okay we have to load another module on top we just have it in a single module um some people disagree with that approach we we found it it helps the users um however for many other python things that are pure python and don't need another module we have an extensive python wheel infrastructure uh where we have our own wheelhouse and so uh the standard of python distribution are our wheels and because our environment is a little bit different from um what the python world calls um uh what's the word again the uh the the binary packages that we can install from pi pi they often don't work on on our clusters so we provide our own wheelhouse so users install those instead so with no users um when they want to install most python packages like tensorflow uh set up your own virtual environment and use pip install python package uh like pip install tensorflow it gets installed in their own virtual environment and then they can they can use that so that gives a lot of flexibility in terms of which python packages people want to use we have some some python packages via modules like these extensions we also have a sci-pi stack module for the most common uh scientific packages London by sci-pi and upload lip uh things like that this is an example of the wheels so we have at the moment as many more than 8 000 wheels installed it's quite a lot of python packages and we can use the command afail wheels to to see which ones are installed so these are tensorflow for instance you can see that we have two three zero installed for us for python three eight three seven and three six so what models are users using we we send that to sit syslocks and we have a nice critical interface uh we can see uh what what people are using using craft banner so here this shows uh which models are used as a snapshot of a couple years ago um we can see uh we can exclude users we can see we can exclude some modules and we can see which then given the description um what models are used so this was kind of the things that we used to also identify uh which models were used more often in the oldest text to see what needs to be compiled for the newest standard environment so this brings me to documentation and resources so we have our list of modules on the user facing biki but we also made uh last year we made the technical documentation public so there's now a github uh based technical documentation about the compute scanner software stack i know that the people on easy refer to it quite a bit also one of the reasons why we made this public and basically anything that's not confidential and we have to keep in our internal google docs based documentation is has been put in that software stack so if you want to know more about our software stack uh internally it's it's it should be all there if not please let us know um you can use the public side of our software stack as well using this uh wiki page access in cvmfs uh so that's uh that allows you to to mount it much like bob showed in his talk for easy you can access our cvmfs as well even on windows with wsl to and and i can use all the software like so that's the end of my talk i hope i didn't go much over time a little bit thank you bard um does anybody have any questions for bard you'll get a question so i will pass across to him thanks for a very interesting talk um one quick question you've mentioned that you had problems with nix and the python virtual environment do you happen to know if anaconda has a similar problem if you're using nix anaconda and nix um i i i'm sure we've seen many we've seen many issues with anaconda so if you if you would combine a python from nix with with anaconda i'm sure you'll find all kind of weird issues going on there thank you because basically basically when you install the over the underlying issue is when you combine when you put something on top of nix uh you are getting outside the nix infrastructure so nix doesn't know what's happened what happens on top of it so whether you use easy build anaconda uh spec whatever in combination with nix it's slightly to break down somewhere yeah that was my suspicion as well from the problem you described with the python environment thanks very helpful maxine has also raised his hand i wanted to add to this the breakage we we had with python is virtual and copies the python binary which was from nix and included uh our path to the nix store and so when we updated something and garbage collected the nix store those our path would stay in the user's environment because the binary was copied in there i'm not sure how this would happen with anaconda but mixing anaconda with anything will give you headaches yeah we've we've when you go on our on our python page the wiki page about python on compute kenda we basically tell users please don't use anaconda it's entirely at your own risk because we've got so many issues with people installing anaconda i have to say thank you for that page because we send our users to your page but i'm not using anaconda yeah so i'm just seeing that somebody is typing and i'm just wondering whether we're about to get a question or in the slack uh question is um for the python packages in the wheelhouse do you have multiple copies built with different architecture optimizations yes we do we have um we have very sub directories um and so uh the the great great great majority of of wheels in the wheelhouse are architecture independent and are just pure python and so they are in generic and there's a small subset that is uh that is an avx2 or avx512 sub folders and it automatically detects where it should copy from