Good morning. Good afternoon, or good evening wherever you are. Welcome to the food transformation system, food system transformation session and land use. I will be the moderator for this session. My name is Hiroko Kunia, and I'm the executive director for the SDGs project for the Asahi Sinbun. Food is about everybody. Currently our food system is unsustainable for the people and for the planet. COVID-19 exposed to us very clearly how the food system is vulnerable to sudden disruptions. It is a situation that we're facing is what World Food Program calls a hunger pandemic. So we are regressing on what we have achieved on the SDGs. Globally, 2 billion people do not have access to safe, nutritious and sufficient food. Millions of people go hungry and go to bed hungry at night. And also the food system, the impact on the environment is very serious. Roughly a quarter of greenhouse gas emissions is from food system. It is the largest single driver of land degradation and massive deforestation and biodiversity loss. Also, if I could add one more point is that about 30% of the food is wasted or lost. Now looking ahead, we are facing unprecedented population growth. Right now the global population is 7.7 billion and it is growing at towards 10 billion in 2050. The planet must feed 10 billion people. So the challenge we face is that we need to produce more at the same time, protect and restore the environment, create a sustainable resilient food system that is inclusive and improves livelihoods of the farmers. The food system needs to be therefore radically, radically transformed very quickly. How can we put effective actions on track, bring solutions to scale, to reshape the food system? Now let me introduce our panel. We have a fantastic panel. First, the Prime Minister of the Netherlands, Mark Rutte. Netherlands is a leading exporter of agricultural products and services. The country has been in the cutting edge in transforming the efficiency of the food system. Next, Jeffrey Lu Mingfeng. He's a Chief Executive Officer of China Menu Dairy. It is the largest dairy company in China and China is expected to see significant increase in the demand for food, including dairy products. Next, Geraldine Mechacht is the Co-Chief Executive Officer and Chief Financial Officer of the Royal DSM. Royal DSM is the leader in the food space, working to advance sustainability and the nutrition agenda. Last but not least is David Beasley, Executive Director of the United Nations World Food Program. WFP was awarded the 2020 Nobel Prize and it stands at the forefront to support people who are suffering from hunger. Welcome, thank you all for joining us. Now, to put it straightforwardly, the food system is extremely complex. Experts say if we want to transform the food system, we need technological transformation, we need behavioral transformation, and we need policy transformation. But I think I'd like to focus a little bit on the potential that we have on the technology and innovation in the beginning of the session. If we are going to achieve agriculture sector that is good for people and the planet, Mr Prime Minister, I'd like to ask you the Netherlands, the agriculture sector has been the leader in transforming efficiency of the food system. What role must innovation play? And which are the areas where collective action can be transformative in the food system? First of all, good morning from the Netherlands and thank you all for participating. And Mrs Hiroko Konija, thank you so much for moderating this very important session. I think we could easily say that COVID-19 has shown that if the scientific community, but also the public and private sectors and civil organization individuals, if they all feel this sense of urgency and perform global collective action that we can tackle these emergencies together. And that means we have to, I believe, to focus on three things. One is, of course, the UN food system summit in September. We simply have to make sure that it will be a success. And I believe it should be a moment we can look back on as a turning point. Secondly, to your point, Madam Moderator, the science and data, because they have proven key in tackling the crisis for COVID-19 and play a crucial role in addressing the climate crisis. And similarly, science should be more visible in a way to inform us how we improve policies, how we improve food production and markets for food. In the near future, as we all know, we'll need to feed 10 billion people worldwide. And then third, the role of businesses in the agri-food sector should be stimulated. And I'm so happy that the company we're very proud of, that it is based here in the Netherlands. Royal DSM is also part of this panel. So the role of businesses in the agri-food sector should be stimulated and able to create scalable solutions. And here I'd like to highlight a World Economic Forum initiative in this regard, the World Economic Forum Food Innovation Hubs. And these hubs in Africa, in Asia, in South America and in Europe will allow businesses to connect regional stakeholders to skill innovations, because this is key skill innovations that can address food systems challenges. And here I'm particularly proud to announce that the Netherlands will host the global coordinating secretariat of the World Economic Forum Food Innovation Hubs, which will connect all other food innovation hubs. And I believe this is important because it will be facilitating to create the partnerships we need. So it's all three together, the UN, science and data, but also the role of the business community and how to leverage and scale up what is happening there. So that will be my first reaction, Madam moderator, to your question. Prime Minister, I think for many of the viewers there, it's the first time they are hearing about the innovation, food innovation hubs. And so if you may explain a little bit more about what the objective is and what the significance is, why do you are making regional innovation hubs? What is the objective? To be very clear, what we need to do here particularly has to do with the role of yes, science and data. It has to do of course with the UN system, but particularly here what we need to achieve is that the role of the business community and the way they can help us to create these solutions. But then particularly the question, and I had this discussion many times with the big companies here in Holland, Unilever, Royal DSM, others in the food sector. How can we not only make use in our countries of what is happening in these companies, but these companies themselves have a worldwide reach. So it's not just the world community or governments which will enable the scalability. We also need here the impact of the business community. And to support this, it's crucial that we have, and this is a particular new initiative by the World Economic Forum, that we have these innovation hubs. But then if you would have them only at a global level, we all feel that we would miss out in terms of what is needed in particular areas. And that is why we need them in the regions, we think in the big geographical regions like Africa and Asia, but also South America and Europe. So that we have the business community to connect within these areas, the regional stakeholders. And that is particularly important because otherwise these innovations cannot get scaled. Maybe we'll hear later from Geraldine Machette from Royal DSM on what they are doing in the area of food security. They're doing some fantastic research. And when I'm in New Zealand or wherever I'm in the world, I am confronted with these innovations by Royal DSM. But then the question is how do you make sure that they get not only scalable in the restaurant world, but also in Africa and in Asia at large. But also when you have these regional hubs, of course, it's also crucially important that we somehow connect what is happening there. And that is why we are so proud here to host this global coordinating secretariat. Mr. Prime Minister, I think I will ask you about the importance of these hubs a little bit later on. But since you mentioned Geraldine, I would be very much interested in hearing about what DSM is doing in trying to improve sustainability in the food system with the technologies and the researchers that you're doing. Thank you very much. What is the role of innovation? Absolutely. Good morning, everyone. And I have to say, we're extremely excited by this announcement that's coming out today. It is the culmination of quite a lot of work that's been going on already in the pre-UN food summit system summit that is coming up in 2021. And to give you a few examples, here are the three areas where I think these innovation hubs are going to be critical. In our view, firstly, the farmer needs to earn more when producing food sustainably. Now, this is very, very much doable, whether it is through carbon sequestration, regenerative agriculture, no-tilling technologies, more biodiversity. These are all examples where the science is there. And we have a lot of colleagues that are more upstream in the food system, whether it be Syngenta or Yara who are doing a lot of work on these things with the ministries, with the organizations, civil society as well. So there's a whole part of, if we truly want to change the food systems, the farmers need to be better off when they do the right things for the planet and for the consumers. So that's one area. And we mentioned amongst others, there's a lot of talk about carbon sequestration in soil, but there's also, of course, reducing methane from livestock, one of the most impactful greenhouse gas. And the science is there, but connecting all of the players to scale it has always been the biggest challenge. So that's one area. Farmers need to be better off when they do the right things. The second area is consumers need to have more choice and realize that diets matter. And you mentioned, Madam Moderator, in your introductory comments, the impact of the COVID crisis. Now, one thing we have certainly noticed in this crisis, and it is a sad reflection of maybe a lot of societies, is that parts of our societies have been hit much more than others. Financially, by losing their jobs, limited access to medical care, but also a lower nutritious status. And the relationship between general health and nutrition and resilience when there's such a crisis is critical. So the nutritional content of food and having the right diets is not just an option. It's not a luxury. It is something that countries need to help in putting in place. Now, all the players in the food system need to collaborate for this. But it has to be both a policy decision, a supply decision and an innovation decision. And here again, the technology exists. So the nutritional status of people, the ability to change that exists. So that's the third area. And then you need to connect it all. And this is all about transparency. And here there's again a lot of innovation and technology. It's about connecting the supply side with the demand. It's about having the form to fork transparency so that people can truly make the right choices. And we can help the different players across the value chain. But for that requires legislation and technology put together. So these hubs are absolutely the right way forward. And we certainly very excited about the announcement. But also the fact that we feel that we can truly contribute with this in place.  I'm sure David Beasley will be very eloquent in highlighting the different needs of different geographies. Because the mindset and the needs of Europe, for instance, or Asia are very different from Africa. And there is technology innovation available for all. But it needs to be custom for the region. They really different challenges in different places. Thank you, Geraldine. Before I go to David Beasley, I think I'd like to ask Jeffrey. You're the largest dairy producer in China. And what is your reaction to this initiative? And where do you think you can play a role? Great. Thank you. Thank you. Good afternoon for those who are in Asia. I'm Jeffrey from Mongolia Dairy, one of the largest dairy company in China. Now, I probably will take the COVID-19 impact as the example. This is a typical disconnection of the entire supply chain, which is actually, you know, from farm to desk. The waste and efficiency loss are largely, you know, increased, which resulted in even bigger impact on the sustainability of the food system. So COVID-19 is probably just an amplification of the problems that we already have existing in the for quite some long time. Now, in our industry, actually, you know, people who've been very, very well known that the dairy industry have a very long supply chain, as well as, you know, even on the finished product perspective, we have a range of big products. But if I just to focus on one thing, which is as mentioned by already deriding the transparency of demand and the supply and to bring that into a data platform so that you will significantly increase the efficiency and also reduce the waste along the chain, taking a very simple example. In the system we have been starting to work on is we know exactly, you know, how much of, you know, the fees we have to grow in the very beginning and how much we have to produce at the farm side as well as in the feeding system and then in the manufacturing system. So when you have all these data available, and we are talking about also a large scale. Today we are produced roughly around 9 million ton of product each, you know, each year. Right. If you look at that scale, and if you look at if we can already put some of the data transparency in the system, we can reduce a lot of the waste. We can reduce a lot of the seal to admission along the logistics system where we are moving around almost 10 million, you know, ton of foods. Right. So these are all the things we have, you know, being doing in a large scale to shorten the distance between demand and supply by implementing the data system and the transparency along the entire chain so that, you know, you will see a significant improve in the efficiency on this system itself. And then obviously the other area we are also very focusing on is on the nutrition system. What is the right nutrition? Through education, because we are reaching almost 90%, 90% of the Chinese family, you know, with, you know, 1.4 billion people. So along the way that we are moving a lot of the effort to the education of what is the right level of nutrition and also about food waste, etc. But all these will have to do based on a data and technology improvement based on a large scale application along the way with all the partner who are working with us today.  And the efficiency will significantly being improved. Right. David, thank you for waiting. I have two questions for you. One is Jeffrey just talked about the potential of technologies, how that could improve sustainability or the efficiency. What have you seen on the ground through helping people in this very difficult situation, how technology has provided better outreach for the people. And also secondly, this initiative that the minister introduced gears to custom made approaches for transforming the food system. So having been on the ground, why do you believe custom made approaches are necessary. The two questions. What technologies have you utilized in helping people throughout COVID and why custom made approaches. Rocco, great. I'll give all that to you about an hour and I'll try to do it in about a minute and a half. But you know, as the Prime Minister was saying, you've got to really look at this at a geographical regional context because what may be working in the mountainous areas of Afghanistan won't necessarily work in Niger or Mali or Burkina Faso. So you've you really got to look at the global context and what works in different places. And that's why the innovation accelerator that we do using e-commerce cell phones to connect farmers to markets as well as buyers to different markets as well. We've got that going on. We spent about $300 million last year in just this type of technology and innovation. And so the innovation hub, I think that's just rolling out and it's really going to have a major major impact. But you know, in some places, and this is what we need to think about because it's two very different contexts. One is you look, as you said, Rocco, the population of Earth will be about 10 billion people in a couple of years. We're struggling now because 700 million people go to bed hungry, 270 million because of COVID are on the really on the brink of starvation. Now imagine what's going to happen when we get to 10 billion people. And so right now the struggles are mostly in the poor countries. What do you think will happen? We don't have enough food in Brussels or Chicago or Berlin. So you've got to look at the poor countries and the wealthy because we're talking about supply chain disruptions along with better technologies, e-commerce, hydroponics, what we're doing now, like in Algeria, blockchain technology, satellite imagery, all the new technologies, better seeds, better fertilizers. But in some of the countries, in some of the innovation, what we're doing now, we used to just bring food in. Now we're putting a couple of billion dollars worth of monies into the local smallholder farmer market opportunities and creating kind of priming the pump to get things moving along. And so here's what I think has been one of the greatest innovations in the last 20 years, 30 years, is in my opinion, the United Nations shunned the private sector. But it's an extraordinary new spirit at the United Nations in that how can the private sector help us solve the problems out there? And so what we're doing now, whether it's with the Unilevers, the DSM, Geraldine, and I have had several conversations just in the past couple of weeks, along with Bayer, Yara, and the list of companies that goes on. My question is not how can the private sector help the United Nations, but how can the United Nations help the private sector be more strategic, more engaged in countries around the world? Yes, we're working on post-harvest losses, better hermetic practices, and we can delve into that to a great degree, but not enough time here. But one of the things that I was talking with Geraldine a couple of weeks ago, along with some other CEOs, is the private sector, particularly in small holder and developing nations, how can you bring your practices there, integrate, but not displace the small holder farmers, but actually integrate them more strategically? Because if we don't, if we continue the same path, we're going to have destabilization when we displace workforce. So we've got to be more efficient, but more strategic and politically sensitive at the same time. And let me make this one last comment, because people say the food systems is broken. It's not broken. 200 years ago, 95% of the people on earth were in extreme poverty, and that was with 1.1 billion people. Today, 7.7 billion, less than 10% of the people are in extreme poverty. Now, so we built and designed system that really is improved and substantial success, but try telling that to the 10% that aren't getting an appreciation of this great asset sharing. And so we've got to continue to work the system. We got to make certain that we are more less vulnerable to COVID type impacts. They're going to be more of those. And as I've told folks, if you think you had trouble getting toilet paper in New York because of supply chain disruption, what do you think is happening in Chad and Niger and Mali and places like that? So we do need to work to improve the system, but we do need to make sure we don't tear down the system that are reaching 90% of the world's population. We've got a lot of work to do, a lot of innovation, a lot of creativity. And as the Prime Minister said, it's not just one size fits all. And Hiroko, we've got to continue to try all these different dynamics because I can tell you, and I know I need to wrap this up. But just in the past few years, with our beneficiaries and ecosystems and trying to environmentally improve the planet, our beneficiaries who reap the consequences of the wealthy nation's carbon consumption. Our beneficiaries just in the last few years, literally planted over 6 billion trees. We rehabilitated over three and a half million acres of land, using half moons and just basic fundamental practices that are just absolutely critical to the people surviving in some of these areas because of flash floods and droughts. And just in the last couple of years, over 53,000 holding ponds, reservoirs, wells. And so one of the things that I tell donors, if you will work with us and what we can do, because we have about 10, we assist about 100 million people, but 10 million of them are engaged in building water wells and road feeder roads. And I can keep going on because quite frankly, anyway, Hiroko, let me throw it back to you. I'm going to cut you short now because I think you can list all the achievements and we'll run out of time. When we talk about the food system, you're right. The humanity has been extremely successful in being able to provide more food in the 20th century. But that is now giving us a backlash of, you know, it is the environment food, the carbon footprint is being hard. It is soil degradation and so much, it is not sustainable. And experts say, and I want to ask perhaps Geraldine, they say that we know what to do. And I think the understanding that the food system needs to change is spreading, but action gap is the big problem. Action doesn't get scaled up. So what is the missing link of this issue? Is it the sort of the sense of urgency is not being shared or is it the finance or is it the policy? If you put your finger on the problem, what do you think is the problem of not being able to scale up action? I love your question because this is really where I get a lot of energy looking at 2021. And the reason I say this is if we look back to the year of the COVID pandemic, what it has brought to light is that these crises don't stop at borders. Everyone gets impacted. And when we discussed with David, the number of people on the brink of starvation, it's the combination of the issues that we knew before. Plus climate change, plus COVID. And when you put all of that, you see a tripling of people on the brink of starvation. Now, I truly believe that in 2021, the sense of urgency is finally going to come out that when we talk about climate change, the first place it's going to hit and it will hit everybody is food. When food goes wrong, there's social instability, there's economic issues, there's migration, there's all sorts of disruption. However, this year we have COP26, we have the UN Food Systems Summit. And when you put those two together, it is the perfect platform to first and foremost actually create the burning platform realization that this is right now. And you're absolutely right. The food issues, yes, it's about quantity, but it's also about how it's produced and whether it's nutritious. And a lot of the focus has been about quantity of food to meet the number of people. But how we produce it is either a spiraling downward issue of degradation of the ecosystems or it can be a spiraling up of doing it in the right way for the planet, but also for the consumers. But it all starts with awareness. I truly believe that the human race actually once it really realizes the crisis is there, it can act very fast. And we saw it last year, the speed at which everybody across society reacted to the threat of the pandemic, whether it be on innovation, new policies, new behaviors, new technologies. It was a matter of days. It wasn't even weeks. So I think our biggest priority for all of us listening to this is how do we not in a kind of exaggerated way, but very factually based, raise the awareness that climate change is first and foremost a challenge to our sustainable food systems. And that is where efforts need to go. Finance technology can be arranged. Mr. Prime Minister, as Geraldine said, if this present situation is compounded with climate change, we should be very aware that there might be a rapid collapse of the food production. In this sense of urgency is spreading, but not probably enough. What, from your point of view, what is missing in trying to scale up the efforts in the transformation? Well, I think that is the core question which we need to address. And I would like to build and amplify on what both David and Geraldine have been saying. And we all have seen the cartoon, I think in the pre-meeting material where we see the waves of COVID, but then the recession, the climate change and the biodiversity collapse. And I think that's exactly what we are facing at the moment. And I would argue, but I would love to hear the views of the other members on the panel that we need to do a few things at the same time. Crucial here is, I believe, a whole of society approach. And that means that, of course, as I think we are all saying, the private sector needs to be on board. And there we have the scalability. It is the entire value chain from farm to fork, so to say. And there, I think it was also the executive director, David, building on mentioning this, the small, the medium sized and the large businesses like DSM working together because they collectively feed the world. And it's not governments feeding the world. It is the whole of the value chain doing this. And that means that I believe there are some great promises when we look forward, and which will also help us in getting the rest of society on board. And one always has to be the technological innovation. Just to mention one example, we have here in the Netherlands the project GeoData for Water and Agriculture, where digital tools help small holders to learn how to sustainably raise their production. So it is a very practical tool which helps small holders in all parts of the world. And a second one I'd like to mention is how can we work on generally sustainable agricultural practices, circular agriculture, acroecology, and above all climate-smart agriculture, as we have discussed so many times also at the World Economic Forum. And we have seen a recent report, the International Finance Corporation says that climate-smart agriculture has the highest green job creation potential of the 10 sectors studied. 40 jobs per million dollars invested. And my vision for the future is that anyway we'll see in many countries that for example in the Netherlands we have 10 million jobs in the Netherlands. In 2030, probably 2 to 3 million of those jobs we do not know of at the moment. We do not know where these jobs will be. But we do know that many of these jobs here in the Netherlands will be linked somehow to the food industry, to the whole of the supply chain in the food sector.  So in closing I would say it is yes and the urgency of now that we have to fight these waves like the cartoon is putting on the table in the material for this meeting. But at the same time we also have to stress the positive that so much future growth potential and new jobs will come out of the innovations we are putting on the table now. And that will help us to come out of the COVID-19 crisis to fight recession, it will fight climate change, it will prevent a biodiversity collapse. So there are so many opportunities here at the same time. Yes, so it would be wonderful if we could create a positive cycle of the positive environment and positive employment and growth and so forth. Jeffrey, your sector is dairy and I think the major issue is the greenhouse gas emissions. And what you have to do is to improve the nutrition of the Chinese people by promoting dairy products. But at the same time you have to decrease the carbon footprint because animal protein is providing so much burden on the environment. What are the technologies and innovations that you are trying to adopt? Thank you. I think if you look at our industry I have a couple of things to share based on this issue. The first thing I think is a leadership issue. I think that you need to require a very strong local leadership that share the same vision to achieve whatever we have agreed to. So we just said by 2030 China will be reaching the highest in terms of the CO2 emission and then by 2060 we will be getting into neutral. So that is important. Once we set that mission we will have to drive into that. Now come back to particular what we are doing today is actually just some of the speaker have already shared. The technology we are now already start to share with our farmers is to help them to grow feeds, use less water and also use less fees when they feed the cows. We have implemented already a digital system on each of the cows. So we know the health situation of each cow and how much they eat, how many water they eat and what are they producing. So you know exactly that in the level of consumption at each key point of the supply chain. So that will actually reduce already by 25% of the feed that go into the farms. And then obviously to produce one liter of muc you have much less of a CO2 emission. So just taking that as an example at the same time we are also working on new technology to work on plant based as well because as you know as we are also working on a food nutrition platform for the consumer. At the same time when we do education with consumer we also talk about packaging green packaging. We also talk about the waste. So we along the way and for example the recycling of our packaging having also a significant one when we do a testing in the case like Shanghai, we are able to achieve 95% of the recycle of the package. So that's also a very significant move. But again coming back to my point is has to be a clear leadership, a share vision, and then along the way a roadmap for every sector. I think every sector have private sector have their own issues about food system. But if we have that and if we have a roadmap of actions, I'm sure it will significantly help it for the transformation of the entire food system. Thank you. Jeffrey mentioned leadership. My question is, is there enough leadership, I've been hearing a lot about transformation of energy on climate change. And not, I haven't heard leaders speak about the food system change so much. So you, David, and as Prime Minister you've mentioned about the food summit and that there's a nutrition summit also and there's COP. We are lacking leadership in this field. I wonder how you can mobilize more leadership action. David, or Mr Prime Minister, who wants to take this question. Well, I always yield to the Prime Minister. But let me say this, one of the things that bothered me in the last few years, because we had crisis all over the world from climate extremes, war, and the number of people marching to the brink of starvation when I arrived four years ago at the World Food Program was 80 million people. And then pre-COVID it spiked to 135 million. And the question was why? Man made conflict, wars and conflicts and dozens of places all over the world. And then climate extremes on top of that.  It was Trump, Trump, Trump or Brexit, Brexit, Brexit, now COVID, COVID. And we literally had hundreds of millions of people on the brink of starvation and the media just wasn't covering it. But now people are beginning to wake up and step up. COVID, I think, exacerbated a vulnerable supply chain system and food systems of which so much success has been made, but it's very vulnerable. And we've got a lot more work to do because what leaders are now beginning to recognize. And this is this is the reality we're facing in 2021 with 270 million people on the brink of starvation. If we don't receive the support and the funds that we need, you will have mass famine, starvation. You'll have destabilization of nations and you'll have mass migration. And the cost of that is a thousand times more. So I believe you're beginning to see leaders step up and recognize we need to address short term and long term solutions. And like the Prime Minister stepping up here, I think this is just a good example of the future that leaders because the United Nations Security Council for the first time passed a resolution recognizing clearly the correlation between food security and political stability and safety and security of nations. So I think we're on a new path now. Now we've got to act on it and move forward. Mr. Prime Minister, can you mobilize more leadership? I will do whatever I can. And if David helps me and Geraldine and Jeffrey, then we can do that. But let me just underpin that because when you look at the numbers, 840 million people worldwide are not assured of access to enough food. I mean, this number is staggering. Secondly, the various crises the world is faced with, not just COVID, but coming after COVID, the recession, the climate change and the biodiversity collapse. And at the same time, the huge opportunity in terms of future job creation, which will be necessary to fight off the recession, which will be necessary to fight off climate change and therefore the biodiversity collapse. And that's why I think the three themes coming through in this meeting are so crucial, which is yes, we need the world on board, the UN system on board. Yes, we need the technological innovations. And at the same time, we need the scalability power of our small but also our medium sized and bigger worldwide operating companies. And when we do that in conjunction, the case for change is enormous and is compelling and is in your face, both on the dramatic side as well as to be a bit more optimistic on the opportunity side. And I think we can collectively do this. And then finally, when leaders are talking about that, it's crucial that they are not just pontificating these things in meetings like this, but also show, and I will see what I can do more in this area, also show at the national level what they do in their national dialogues to make this happen. And that will also add to the, let's say, whether it is believable what they are saying, whether other people believe what they are saying, because it is putting the money where their mouth is, they are putting words into action and that also has to be done at the national level. Of course, that is where national leaders are most effective and from which they will get their authority to talk about these issues in panels like this. And I'm not saying I'm the perfect example here because we are doing not enough on this issue, but I will promise you that we will do more on this. And also at the national level so that next time I can speak with more authority on this issue. Okay, thank you. Maybe this is the last question I'd like to pose it to Geraldine. What will be, we need more business participation. We need to stimulate the business activity in the field. What is missing? What is missing? I think what is missing is a collective belief that we can make the change happen fast. I think there is a certain view that in the food system things take a long time and that therefore it's a very gradual progressive process when in fact change can happen fast. And when we align all of our technologies end to end, we can make the difference. And what really is missing, if I think practically about what's been in our way, it's the ability to connect from end to end from the planetary impact of producing the raw materials of food to the processing to the consumer diets choices and the transparency across that. Our biggest challenge as an industry is to see how do we connect all together with the help of the public sector with the help of the NGOs to make it happen that we have this transparency across end to end. And I would like to add a little wink to the work on the ESG disclosures, because I think the more companies report as a private sector, how we contribute to the SDGs, how we contribute to the non financial impacts of our activities. The more this will help us connect both upstream and downstream to work together. So the convergence on ESG disclosures is also a very important step that is happening right now. Okay, we're out of time. I'd like to thank all the panelists for joining us. If I may, at just one point. Sorry, I was, if I were to add one point on what Jaredings said, I think it's also along the along the long chain of all these partners who should take it positively. When you talk about this, you know, people might think it's a negative thing, but there's a huge of opportunity in this area on technology on efficiency on creating new jobs. So all these partner when they are joining, we should have a common belief it is positive. It will generate more positive impact financially, but also economic socially. Great. Thank you so much. We're out of time. Thank you all for joining us. And goodbye.


So good morning, good evening, good afternoon, depending on where you are in the world. Thank you for joining today's CNCF webinar. I'd like to thank you all for joining us. And today's webinar is how to migrate databases into Kubernetes. I'm Christy Tan, and I'll be moderating today's webinar. We would like to welcome our presenters today, Alex Chirkov, CEO and Founder of Storage OS, and Peron Castel, Product Reliability Engineer at Storage OS. A few housekeeping items before we get started. During the webinar, you are not able to talk and as an attendee, there is a Q&A box at the bottom of your screen. Please feel free to drop in your questions and we'll get to as many as we can at the end of the presentation. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of all of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinars page at cncf.io slash webinars. With that, I'll hand it over to Alex and Peron to kick off today's presentation. Take it away. Thank you, Christy, and good morning, good afternoon, and good evening, depending where you're dialing in from. A little bit about myself. My name is Alex Kirkup, I'm the founder and CEO of Storage OS. I also hold two hats and I'm the co-chair of the CNCF Storage SIG. I am very focused on building a candidate storage solutions at Storage OS, but before that, I spent 25 years engineering a number of different infrastructure platforms, primarily for financial services. Always happy to hear feedback from the community, so feel free to DM me or join our slack and interact. Peron, do you want to quickly introduce yourself? Sure, Alex. Hello, everyone. My name is Peron. I'm based in London as our company Storage OS is. I'm a platform reliability engineer with a background on infrastructure and platform engineering. We usually run a lot of infrastructure on clouds and bare metal, so my background is mainly on systems and a bit of DevOps as well. Brilliant. Thanks, Peron. Before we start, I just want to give you a bit of background on Storage OS, where we come from. At Storage OS, we are building cloud-native platforms for customers and users who are running platforms in the cloud, on-prem or in hybrid environments. One of the things we come across very often is customers wanting to run databases in Kubernetes, so we're going to share some of the experiences. As I mentioned, I've also got my other hat, so I'm going to do a little bit of SIG advertising here. The CNCF Storage SIG is a public group. It works with the CNCF TOC team, and we help create content for end users as well as review projects and provide technical storage expertise to the SIG, to the TOC. The calls are open and are open every second and fourth Wednesday of the month, so we'd love to see you there. I talked about what we're seeing in our end users' communities. Of course, everybody begins their cloud-native journey, typically with containers. Containers have changed the application landscape because they've broken the dependencies between an application and an individual-specific server, so this concept of cattle versus pets, which we hear about quite a bit. Containers allow applications to be portable, and of course, the minute applications are portable, they can be orchestrated. This is where Kubernetes comes in, so Kubernetes being that container orchestrator allows developers to be able to compose and declare what they need out of their applications by defining, for example, compute requirements or network requirements. Kubernetes plays like an expert Tetris game with your applications to abstract away your infrastructure and fit the applications in the most efficient way to the available resources. Of course, because it has the automation, it can do lots of advanced services like automating, scaling, and healing, and connectivity. But of course, all applications store state somewhere, and one of the amazing things with Kubernetes is how, with the power of new standards like CSI, Kubernetes can interface with different storage systems. Now, developers can use cloud-native storage functionality that will integrate with Kubernetes to not only automate their compute and networking, for example, but also to automate storage provisioning, availability, and scaling. Now, what does that actually mean in terms of an end user? So effectively, now that you can specify storage requirements, it makes it very easy to move stateful applications into Kubernetes. And because those storage requirements and those applications can be defined with a standard set of YAML, we move to the scenario where it's possible to build anything as a service. And one of the things that we see happening very regularly is the concept of building databases as a service. So why would you want to automate the deployment of databases in Kubernetes and what are some of the advantages in this? So one of the key things with Kubernetes and cloud-native storage relates to this key concept that the containers and the applications are portable. So in this case, when I refer to an application, I'm talking about your database because every type of database is now mostly available as a container too. So popular databases like Postgres or MySQL, for example, as well as databases like same Mongo or Cassandra, and more distributed systems like Vitesse, for example, are all deployed as containers within Kubernetes environments. One of the key things here though is that those applications, those databases, are inherently portable and they benefit from the ability for Kubernetes to dynamically scale down a cluster and to be able to upgrade nodes on the fly, whether it's because you want to upgrade the resources of nodes or you want to upgrade nodes to make sure that software is current and security patches are applied, for example. So one of the key things to look out for here is to look for storage that is portable and supports the cloud-native attributes of that type of application and database running in Kubernetes. So effectively, we're moving away from an environment where servers and storage are tightly coupled, where the storage is locked into individual server nodes and we're moving to an environment where the data and the storage is locked into the database and the application and Kubernetes has the ability to move your application and therefore the storage is portable too. We talked a lot about the fact that in Kubernetes, the environments are declarative and reproducible. So what do we mean by that? In much the same way that you can specify the containers that are needed, in this case the database and the compute and memory and network requirements, you can also declare the storage requirements and items like scaling or availability, for example. And the reason why this is so incredibly powerful is because it makes your environment recoverable and reproducible. So you no longer are nurturing individual servers, but Kubernetes has that power of extending the database and recreating that environment wherever you are, making it very easy to have pre-production or development and production environments replicated very easily. One of the other things which applies to databases in Kubernetes is the way you scale the deployments of those databases. I will talk about that detail in a little bit, but effectively what we're recommending is that you co-opt the same sort of concept as microservices where effectively a database is no longer deployed in large gigantic instances, but different databases can be deployed in separate smaller instances and can be scale out horizontally with highly available or dynamically available database endpoint subtractions, allowing you to tune the workloads and to right size the environment appropriately. So we talked a bit about how that dynamic provisioning works, so I'll give you a little example of what we're referring to specifically for a database. So within a database we start off with a storage class. A storage class is a way to abstract away the definition for accessing a storage provider that Kubernetes is going to talk to. Now we mentioned CSI. CSI is the container storage interface and this is the standardized API which is used by a number of different storage providers to provide the interaction to dynamically provision volumes, but also to dynamically attach detection and mass volumes into different nodes within your environment. Effectively using a storage class which is an abstraction point for your storage provider, you can now define a persistent volume claim. That persistent volume claim uses the storage provider to dynamically provision a volume and is identified using a friendly name which you can then refer to in the application itself, typically defined in a pod or a stateful set. So in this case, for example, we're showing the crunchy data example to define a Postgres container that will use the database volume that we defined in the persistent volume claim. The idea behind this is that we now have a very simple way to define the way the databases run and the volume requirements that are being used. And if you were using a software defined storage system, you can also have the benefit of being able to use the same YAML configuration, whether you're deploying, say, in a cloud instance or on-prem or perhaps in different VMs or developer environments, for example. I'm going to caveat this and say that many databases actually have a superset of automation nowadays called an operator or some sort of capability to actually provide additional functionality around the life cycle of the database. For the purposes of this particular webinar and this example, we're restricting ourselves to providing examples that are related to starting a database using a simple pod or a stateful set. And that's mostly for simplicity, but for completeness sake, I'm mentioning the fact that operators are available for most types of databases to provide that database life cycle. So what does it look like in terms of what we've seen happen time and time again in different enterprises and then user environments? So typically, we start off with a large server or a service that has potentially multiple databases hosted in a single database instance, typically on a large node, and that may or may not have dedicated storage or may have some existing other storage provider. The point that I'm wanting to emphasize here is that most of this definition is now very, very server centric. And it's based on a scale up architecture, which obviously gets more expensive based on the capacity of those individual servers. But more than that, the functionality becomes operationally more complex and prone to failure, simply because of the fact that we're talking about running multiple databases within that large instance. So every time a new database is required, it requires a fair amount of manual effort. Perhaps it involves database administrators or DevOps teams to get involved. And of course, you know, potentially operational change windows for your database environment. So what does this look like in the new world? So starting off with that large database instance, we then are looking at how we deploy this within our Kubernetes instance. For the sake of this particular example, we're using storage OS as the underlying storage layer, but of course, other storage providers are available. And we start off by moving a single database instance within a container that is now runnable and accessible within the nodes, within a node in the Kubernetes cluster. And this is typically a pod or a stateful set that might be managed by an operator. We can then continue to break out the additional Kubernetes, sorry, the additional Postgres databases into their own mini instances, effectively making them portable standalone products that can be distributed across the different nodes in the cluster. So effectively, we're using the functionality of Kubernetes to turn each database instance into a declarative self-contained container. One of the aspects of using the provisions of the cloud-native storage within Kubernetes is that a lot of the storage providers provide the capability of having primary volumes and replica copies of that data in some form that are distributed between the different nodes within the environment. What this means is that, and again, using the automation that's available through CSI, that allows Kubernetes to talk to the service, to talk to the storage provider. When a database is working with a copy of the data, the data can actually be replicated and data protection can be applied such that transactions are being replicated to more than one node within the cluster. And again, there are a number of different software options that allow you to do this. And the CNCF-6 storage has created a white paper to describe the storage landscape in the CNCF, which is a good reference point that I'd recommend that you read. This type of data protection means that if a database commits a transaction to the storage environment, that data is now available on multiple nodes within a cluster, which means that if a copy of that data fails, say, due to a disk failure, resource failure, or a node failure, that the database can continue to run transparently because the storage system is handling this in the background. So this gives additional options to some of the database providers by providing storage-level replication that's managed by Kubernetes to protect the databases within that Kubernetes environment. Of course, there are other projects like Vitas, for example, that provide other methods of doing database-level replication, which can also apply in these environments. But suffice to say that the whole concept of allowing the database within the Kubernetes environment to benefit from that transparently replication means that you get automated failovers within your environment, especially because the database remains accessible via the service accounts and the service IPs that are defined as part of the as part of the pod or the stateful set. So just summarizing, databases are, of course, stateful workloads. And in order to take real advantage of these Kubernetes workloads and Kubernetes environments, it's important that you are able to apply the same declarative construct and composable construct that you apply to your applications to the stateful workloads to like databases. Containers are, of course, ephemeral, and nodes can be ephemeral, but using the capability of software-defined storage within Kubernetes, we can ensure that there are multiple copies of your data and data is accessible anywhere. And just re-emphasizing that CSI as a standard interface that Kubernetes uses to talk to storage providers has now been around for over two years as a GA function. And this provides all of the benefits to provide flexible storage for these environments to allow the databases to be portable with dynamic provisioning. One other, a couple of other aspects that I'd like to cover is that by splitting out each database instance, each database into its own separate containerized database instance, we have better performance and better throughput, typically because Kubernetes can now balance the load more effectively into multiple instances. We have the concept that each one of those database instances can be tuned and resources allocated individually per database, so you're not having to worry about one large instance for multiple databases. And of course, by scaling out horizontally rather than focusing on a vertical scale-up with Kubernetes, you reduce the failure domains and you reduce the blast radius and the impact of any one component within the cluster failing. All right. So with that, I'm going to pass on to, I'm going to pass the battle on to to Ferran, who's going to give us a live demo, talking, providing a demo to show us how to actually move a database from a standard server into Kubernetes. So this is probably the exciting part of the environment and I'll pass over to Ferran now. I'll stop sharing my screen. Thank you, Alex. I'll start sharing mine. Second, great. Cool. So as Alex mentioned, we're going to go into a live demo. First of all, let me give you a bit of introduction what we have. We have a Kubernetes cluster. Four nodes in Google Cloud could be somewhere else, but Google Cloud is as good as any other. And we're going to do actually what Alex explained about sending the main database into multiple ones in Kubernetes. So we have one main server with Postgres. Let's actually have a look. It's a standard Ubuntu box with a Postgres running. You can see just a simple process. It's just a standalone machine with a multiple Postgres databases. So let's have a look. We can see actually that we have multiple databases. Those databases are schemas. I created with ordinal numbers those databases for simplicity so we actually can see what we are migrating. But keep in mind or try to understand that this is something that we are creating a database for each microservice or for each component of our application or for each part of our holding, our company, etc., etc. So there are different ways of distributing our data to the store. So I'm going to move all these databases which I will understand each database as a microservice into Kubernetes. So I'm going to use k instead of kubectl to save everyone's time. Just bear with me. So I created the namespaces. So we have one namespace per microservice. Very simple. We want to keep isolated them. Even they can access to the network. One component to the other. We keep them separate. By the way, everything I'm going to use today is available in GitHub and we're going to share the links for the YAMLs, etc. And I want to encourage that everything you see today is something that I highly recommend to use into CI CD environment. So we're going to do it by hand so you actually can see what is under the hood or in the Kubernetes constructs. But you would use a CI CD or Jenkins server, whatever tool you prefer, to actually run those automations. It will make your life way, way easier. So first of all, I'm going to deploy a Postgres server on every single one of these namespaces. So we're going to have 10 databases or 10 instances of Postgres itself. To do so, I'm going to use these YAMLs that, as I said, they're in a GitHub repository completely public. And then we're going to migrate the data. So I'm going to iterate over them and just create the resources. In a second, I'm just going to show what actually are we doing. So to be able to provision databases, as Alex mentioned before, we're going to use stateful sets. Every stateful set is a Kubernetes controller that will enforce certain requirements. For instance, the amount of instances of the application. So let's have a look at the stateful set itself. The set is a Kubernetes controller that will enforce, in our case, to have one instance of the application always running. We are using a fairly standard or fairly common Postgres instance. And very importantly, we're going to mount a volume into PG data, into slash PG data. That is where Postgres will write the data. If you want to use and split the write ahead log and all other components that Postgres allows for speed or whatever, you can do that here, create different volumes. Because this is the clarity, it's very easy to just provision your volumes. Where are the volumes coming from? These volumes are coming from a template. That template is a persistent volume claim template. So for every application, for every part that this controller starts, it will have one persistent volume claim always associated to it. In this case, we're just referencing the storage class that we want to use for our software defined storage backend. But there are actually different ones in this cluster. So we have multiple. There are two of them that are software based. And there is one actually that is the world cloud volumes. And actually you can add more and create different storage classes that will actually have different capabilities or options for your specific use cases. Now, let's actually have a look at the pods that we have created. I'm going to use K9S, which actually is a very cool tool for live information. I'm searching for app. We can see that for every name space, there is one pod, one positive spot for every single one of those microservices. So now every microservice can have its own database. Cool. Every one of those databases have a volume associated. Let's say app three to say something, get pbc. And we can see that there is a pbc associated to it. So now we have 10 instances of Postgres running and they have 10 volumes associated to each other. Cool. Now we have Postgres running. We have the main server running. Now let's run the migration. Let's see. Let's see. For instance, let me show you in the instance itself before we did the migration some data that actually is in this table. So actually we can relate later about them. Let's connect to one of the applications. Let's say app two. Let's see there is one table. Actually it's just random data and there is just block files and block information. There is no real use, but we will be able to see that this table actually is migrated along to the database itself. As I said, it's very important that you guys use CI CD for these kind of tools or for these kinds of executions I'm going to do now. There is one thing that we actually need to have before we do the import itself and that is a way to connect our Postgres servers, our many instances in Kubernetes to the main server. We do that using the external service. It's a standard YAML. If you're not familiar with external services with Kubernetes, don't worry too much. Just for context, what we are doing is we are telling Kubernetes that there is an IP that we want to access from inside the cluster, but we want to give it a name. Instead of accessing the IP itself of the Postgres server, the main server, we are creating an endpoint and a special service that allows us to access that main server from a DNS name inside the cluster. We keep good practices and an elegant distribution of concerns. Actually, I have a name space that has that service. We can see that a service called PG, so we're going to access PG.name space, so PG.PostgresExternal, and that will forward into that specific Postgres database. If we see the endpoints, actually, it's mapped to a specific IP where my Postgres server is running. Cool. This is important because when we want to do things from an automated point of view, we want to change configuration while we don't have to change anything else. How are we going to run the migration? I'm going to use a job. A job is a Kubernetes constructor or a Kubernetes controller that will execute my container or, in this case, my task once. What this task does is a simple PG done and pipe to PG restore. Of course, if you have big databases, I have a small database today, but if you have big databases, you can add the size of the migration in here or the size of every chunk of data that is migrated so you can control a bit of the flow on the data and be more sophisticated with that. Essentially, it's very simple. There is a source and a destination. Where these source and destination variables come from? They come from configuration, as actually it's quite common and quite obvious, but sometimes we just want to keep these jobs as agnostic as possible. Through configuration, we will be able to run this. We will create one of these jobs for every namespace, so we will do an independent import for every database that we have. Let's have a look at the config map. The config map itself is again quite simple. We have a database. This is only one database, so we will have to use some templating and some patching to be able to specify the source database that we want to migrate. User and password, of course, use secrets for this. This is just a demo. The source and the destination, and this is very important. The source and destination are those DNS names I was mentioning before. The source is this external name that points to the main Postgres server, while destination is the name of the pod dot the service, because I'm using a stateful set along a headless service, which is a service that goes along with stateful sets, I can always access my application with the same name, podname.service name, because this spot will run in the same namespace as every database. It will always connect to the right one, because I don't have to specify the namespace itself. With this, the only thing we need to do is a patch. Let me create the config map, and then I'll show you how to patch it. Let's iterate once again over all the namespaces, so one config map per namespace, 10 config maps in the end, and now let's apply a patch. A patch is a simple Kubernetes, I could say construct, but it's a way that we can alter the YAMLs themselves. Of course, you can use, for instance, customize or any tool that you find or you prefer for your own templating. I just want to keep it simple, and on the other hand, you can do that from a CICD point of view. Very easy to do that. You have your applications or microservices listed, you simply iterate over them, and you tell every configuration or every application, which configuration has to apply. So we applied the patch, let's have a look at one of them. So we have the main postgres and the postgres migration. Great, we can see that in namespace app one, we have the database app one. Quite simple, but actually this allows us to be agnostic and run that as many places as we want. Now we have to just create the job for every namespace. Let's iterate over the namespaces once again and create the job. While they are created, let's go and keep looking at the real-time information. We can see that the population itself, you can see there are different ones, and for every namespace, we keep having one. Actually, because obviously the data is quite small, they're finishing fairly fast. Let's have a look at the logs of one of them. Let's say up six logs, and we can see simple pg-restore restoring our data. That database is my data. So now let's go into one of the applications themselves. Let's get a shell into them, and let's get a shell. And we can see that in app two, we have only one database. Instead of having 10, we have one. Let's connect to it. We have the same data as before. Something for the trivial, it's just a pg-restore. In our case, pg-restore is not blocking. So for production, it's pretty all right. For other databases like MySQL, if you use MySQL, don't be careful because that's blocking. There are different tools. So adapt to your specific use case. As always in engineering, we have to always find the best solution for our use case, not for someone else's use case. So pretty much we have now 10 databases with their own applications, and they are segmented. So we just migrated in a matter of 14 minutes while I was talking databases from one big instance that can be difficult to maintain, difficult to upgrade to multiple databases where their concern is a split. Let's say that app one, for instance, is very intense on reads. That application, that database, can be scaled for reads, can be improved to scale better for reads. Let's say we can add read replicas, or we can scale the size of that stateful cell or the pot itself to scale horizontally to make it bigger, or we can specify different configurations. While maybe app two has a different concern, for instance, is the microservice for user logins, where latency is not as important, but consistency is really, really highly important. Backups don't have to be the same time. We have a different concern. We have different tuning options. We can improve our availability and actually the blast radius of that, our databases. If our main big instance of Postgres goes down, even if it's a cluster itself, if we have a problem with that cluster, we will end up with a situation where all our information, all our services are down. With this model, of course, it has caveats. There is an overhead of running a database. That depends on your use case and how you want to split that. You will have a better scalability and a better microservice architecture, in my opinion, at least. Cool. The demo is pretty over. We can keep going and start doing things to databases, but I think it's a good time to pass to the Q&A. If you have any questions about the demo, feel free. I can reshare my screen once again and answer them online or on the command line. That's fantastic. Thanks so much, Ferran. We've had a few questions come in, so we can go through them. One Olivier asked if we can share the repo that we're using for the use case. Perhaps if you can send that to the chat window and I can mention that. We also had a question around what is the opinion about local PV for Postgres with replicas, such as Crunchy, or with Stalin implementation? This is a question that comes up fairly often. We often have a discussion about the pros and cons of different instances. Effectively, if you are thinking of a database like Postgres, availability can be done at the Postgres level with replication at the database level. It can also be done at the storage level, or perhaps it can be done with boats. Each of those options has perhaps different use cases, so using a local PV, which is effectively a disk that's only available on an individual node, means that Postgres instance is very tightly coupled to that node. It does make services like Failover a bit more complicated. If the disk on that particular node fails, it means that it's not straightforward to move the database to another node. Of course, you can do database level replication in that instance and then have and then recover to another database. That said, you are now managing two sets of databases and two sets of compute resources, because both databases have to be up to do replication. On the flip side, if you're using a storage system that's providing storage level replication, and again, there are a number of different options how to do that, you tend to have better portability, so databases can database instances can move around horizontally across the different nodes within the cluster. Additionally, you have the benefit that you have an extremely low recovery time if a database instance actually fails, say a node or a dataset fails, because one of the things that we do come across quite a lot is how does a failure affect the deterministic performance or the recovery process of a database? For example, if you have a very small database and you're doing database level replication, then if a database fails, it might be quite quick to sync up another replica of a database. If the database is large, that's creating another replica instance of the database could take a long time and potentially that impacts performance from a disk and from a network resource point of view while that recovery is happening. That's also another reason why we've seen end-users deploy both database level replication and or disk level replication in both of those instances. We have had another question. Sorry, go on. I was about to ask one of the questions actually for you, Alex, because you are here quite a bit about this. What's the preferred storage selected to deploy storage for stateful databases? What's the best recommendation? That's actually a really good question as well. It is a good question. The answer is it depends and I'll sort of expand on that slightly. In the past, we've often taught of the attributes of a storage system in terms of how the access method for that storage system is defined. So SAM being sort of block-based devices or NAS being, for example, a file system-based device or NAS being directly attached to storage to an individual node. The reality is, though, in a Kubernetes-type environment, we have a lot more layers to a storage system. I guess the important thing to understand first is what are the attributes of your database or your application? Often databases may not necessarily be optimized for throughput, but they probably are. They probably do require lower latency if they are to handle transactional performance, for example, and also databases typically have storage attributes that require strong consistency for data integrity and strong data durability. All of these different attributes, by the way, are well defined in the CNCF storage landscape, which we publish on the CNCF storage site. We define these different attributes and explain the different layers within the storage system that contribute to these attributes. One of the things that changes quite dramatically in the cloud-native world is that, because of those layers, it's no longer safe to assume that the way you consume the storage, say a distributed block system or say a file system, defines the attributes such as performance or latency. For example, we see file systems that are sometimes built on top of object stores. It might have the file sharing attributes of a file system, but the latency attributes of an object store, for example. Therefore, what you really need to do is just understand the attributes of the storage system in relation to what you need out of your databases. Typically, databases, as I mentioned, require the low latency and the high consistency. Therefore, you're looking at a storage system that can provide strong consistency and deterministic latency within these environments. That would tend to naturally select distributed block systems, but that doesn't always have to be the case. May I answer one of the questions, Alex? Will you go through the other if that's okay? Of course. Go for it. There is a question talking about the size of the persistent volume in the context of databases in Kubernetes. That is definitely a situation where it's not easy to know which kind of size you want to put. Obviously, persistent volume claims are declarative and when they are declarative, you have to set them at bootstrap. When you create the PBC, you set that image, that size. Of course, you don't know how it grows. You cannot just go like this. In fact, most or some of the storage providers for Kubernetes implement resize of PBCs. If your database runs out of the space because you put 25 gigs and it's not enough, most of or I would say most, but some of the solutions, mostly the software-defined storage solutions, allow resize of PBCs. It couldn't really be a problem for that. Of course, the answer is it depends on the size of your database and your data for each dataset. But in the end, you can just resize based on the storage provider. Yeah, that's brilliant. It's fantastic to see the continued development of CSI and how this enables the data to operations like resize. It's key to the day-to-day operations. Hey, Fran, one of these questions is for you. A user has asked, is it possible to replicate data from an on-prem database to Kubernetes databases in order to require minimum downtime? Yeah, 100%. What we did today was no more than connecting one application to another. We didn't care about the infrastructure. We just needed, obviously, the plumbing of or the network in between. As soon as your Kubernetes cluster can access, whether if you have VPN access in between instances, they are in the same subnets or they're visible through the network, from your bare metal through VPN, for instance, to a cloud environment or wherever it is, you can just configure one of these databases to pull that data to replicate from that master. From a Postgres point of view, that's not problematic. Actually, it's fairly easy. It's asynchronous. When it's ready, it's ready. The only thing you have to do is set up the configuration. We created a config map while I didn't show the config map today, but when we created a stateful set to create the application itself, the Postgres application, that has a config map, which in there you can specify all the configuration and you can configure there as lathe or more commonly called a passive instance of a master. I could highly recommend, though, if you do that and you have higher latencies, more than five, six milliseconds, to do asynchronous and to do a transaction log replication through Postgres, for instance, is good enough for that. If you create synchronous replication in between a data center and a cloud provider, or something that is far apart, you will struggle a bit with performance because the latency affects the input-output operations on synchronous replication. Brilliant. Thanks for that. We've also had a follow-up question on the database sizing. The question is about what is the recommended size of database storage to be effective in a Kubernetes database? I think that's a fairly open-ended question. I don't think there is a specific size that we would necessarily recommend. It can be as small as a few gig all the way up to several terabytes based on the actual size of your database. However, that said, one of the things that is worth understanding is, again, how size translates to the different attributes within your storage system. For example, depending on the storage provider, you may find that perhaps there are IOP thresholds or input-output operation thresholds or megabyte per second throughput thresholds that are correlated to the size of the volume. Sometimes you may find yourself having to over-provision the size of the volume to ensure that you have the correct number of IOPs available to run your database. That is definitely a factor which is very specific to the particular storage provider. Alex, if I'm going to shoot a question to you, what is the overhead in terms of memory CPU to deploy the storage OS on all the nodes of the cluster specifically in Kubernetes? Specifically with StorageOS, StorageOS is built to be extremely low overhead. StorageOS can typically run with a single core and maybe a gig or two of RAM depending on the amount of activity. StorageOS will largely coexist with additional workloads which are running on the cluster. Typically, it's often deployed in a hyper-converged type of topology where nodes are used to provide storage to the pool but also for those nodes also run the same applications that run on the pool. Of course, the amount of CPU that any storage system utilizes will be tied to the actual amount of activity, of course. If there's lots of IOPs or a high number like hundreds of thousands of IOPs, you'll see higher CPU contention by the storage system. We had another question around container-attached storage. Container-attached storage is a term that's used by a number of software-defined storage providers where those storage providers are actually providing storage, are actually deployed as a container and providing storage to a cluster that way. I think the answer to this question is, in terms of does container-attached storage lend itself to databases, the answer is obviously yes. There are a number of CNCF projects like Longhorn which falls into this category as well as a number of different vendor-supported projects like StorageOS that work in this method and effectively create volumes out of the available storage on individual nodes within the cluster. I think it is a particularly good fit for databases. It's a very common use case. Things like databases and maybe message queues are one of the most common use cases in terms of stateful workloads that move into Kubernetes first. A lot of the services offer some more advanced functionality, like which we haven't talked about today, but things like affinity and locality which provide additional benefits to databases within the Kubernetes environment. We also have another question around is it possible to run a Postgres cluster like a primary replica set up in Kubernetes and also how would you load balance to it and shard it? Is that something you can cover? Yeah, one second. I was answering one of the questions on the chat. You mentioned about the primary replica set up, right? Yeah, cool. Let me re-ask the question a lot. Is it possible to run Postgres QL cluster like primary replica set up in Kubernetes, also how would balance and sharding work in Kubernetes? Kubernetes, I would say, that is no different from running that in a different server. What happens is that you have an orchestrator that will leverage the startup of applications anywhere in the cluster real fast. That's fantastic. You no longer have nodes tied to applications, you have applications tied to resources. Storage is a resource, no different from CPU or memory. When you're asking for storage itself, if you have a system that gives that storage as actually what I was doing today with the PBCs, you're just asking for capacity and the storage system is giving you that capacity. Running the application on top of this is no different than running it somewhere else. In fact, because Kubernetes gives you the networking interface and the DNS itself is actually very easy to configure primary replicas or read replicas or active passive model. Actually, I would say read replicas is easy, active passive, use a storage system because that will be way more difficult. It's done for you, so you don't actually need to do. When we're thinking about primary and replicas, you can configure very, very easy a service that is for the primary. Then another service, let's say a stateful set with a headless service, then it's very easy to configure another deployment or a stateful set that holds replicas. I would say stateful set. You can have a stateful set and scale that to 10 instances if you want, so you would have the primary stateful set and the replica stateful set. Tiny bit of configuration to sync all the data or to put configuration for the syncing using DNS names that they never change, whether even though the applications, the pod restarts here and there, that doesn't change. Kubernetes handles that frontend DNS for you and then you can just access the replicas through one service called read replicas dot posgras and the main one to main dot posgras to say something. Then when it comes to sharding and things like that, you can do exactly the same. The load balancers that run in Kubernetes, it's a level for load balancing, so it's based on networking. It's not really, you can configure that if you want, but you don't need to. You can configure a load balancer itself physically, but you don't need to because all everything is handled, whether it is with IPVS or IP tables or different components of the CNI, the access to the network you can do by name. When you have to shard, you will hit any of the pods that serve a service. When you have to shard, you have to keep the sharding in your own application or posgras can do that for you because the routers are in front and they're frontended. If you, for instance, elastic search, elastic search does something similar, you hit some frontend part of the application that knows where the backend data is and it does all the aggregation when it comes to sharding, et cetera. It's fairly easy. In fact, I would say that it makes your life easier to run these infrastructure components in Kubernetes, at least in my opinion. Fantastic. And I think that covers all of the open questions and puts us neatly just a little bit over time. Yeah, perfect. Thank you again both for the great presentation today. Is there maybe your Twitter handles or a Slack link that you want to share with folks in case they have lingering questions after the webinar? Yes, indeed. Drop those in the chat. Oh, or your slide works too. Yeah, by the way, I shared the Github repo as well on the chat and in the questions. So we're all of our engineers and Ferran and myself are available on Slack. StorageOS.com and all the demos and use cases are available in the documentation on our website. I would happily discuss use cases and different options for moving databases into Kubernetes. Thank you very much. Great. Well, thanks again for attending. We hope to see you at a future CNCF webinar. A reminder that the slides and recording will be posted later today to the CNCF webinars page.  Thanks. Bye. Thank you very much.