The panel is a keynote conversation led by or moderated by Candace Rondo who has been a part of the Future Security Initiative for many years. She's Senior Director of Future Frontlines and Planetary Politics here at New America and she's a professor of practice in the Future Security Initiative at Arizona State University. She'll be speaking with Ambassador Oksana Markova of Ukraine. Thank you. Well, Ambassador, thank you and welcome to New America. This is your first time. Yes. So I'm here physically. That's right. Virtually, we've had many moments, but also you and I have had a few moments here and there so I'm excited for this conversation. I wanted to introduce you, if you don't mind, to our audience here at New America and online. For those of you who don't know her, I would say that over the last two years, two-and-a-half years really since her appointment as Ukraine's ambassador to the United States, Oksana Markova has been one of the most active and energetic diplomats on the circuit in Washington, D.C. and I really count myself as very lucky to have been able to sat across the table from you on many occasions to talk about the future of Ukraine, how to respond to this war of aggression that Russia is waging against your country. I've met a lot of diplomats in my time and I have to say very few have impressed me as much as you in large part because not only do you work tirelessly for Ukraine's interest, but you really are working one of the toughest towns in the world to be a diplomat. I don't envy anybody coming into Washington for the first time trying to figure out how it works because we all struggle. Those of us who live here struggle with it, but you have really mastered the arcane art of navigating the State Department, the Pentagon, the White House. I have seen you throw out a first pitch at the National Stadium and now you've got a pretty good arm, I'd say. Actually, pretty impressive. I was trying not to embarrass my son. Well, you did well. You did well. But prior to your time here, you were also at the Ministry of Finance from 2015 to 2020. First Deputy Minister and Government Commissioner on Investments. And then since 2018 as Minister of Finance, you know about money, apparently. And you have degrees in Environmental Science from the Kyiv-Mohila Academy in Ukraine, an MPA, sorry, in Public Finance from Indiana University, Korn Huskis, let's go. And you also have a head for numbers. I have seen that also at work in some of our conversations and science. And I think you know a lot about what it takes to run a business, having spent 17 years in the private equity field. So I think you also have a sense of what's going to happen next in terms of reconstruction. And we're going to talk a little bit about that today. So again, thank you for coming. Thanks for making the time. Our time is unfortunately limited. I'd love to stay up here all day. But I want to start out with, I think, one of the big challenges that's probably on everybody's mind here. Ukraine is fighting a war. The defense alone must cost, who knows how many billions, right? But we also know that there is just a huge reconstruction need ongoing and in the future. And I think one of the biggest questions most recently has been around the need for reconstruction in the area of grain exports and production and where we're going to go next with that. So I want to just ask you sort of what is your impression of what happened with the Black Sea grain initiative in terms of its impact on Ukraine's export capacity? And so what's Ukraine's perspective on how to bring that back together or move forward? Thank you. Thank you, Candice. And thank you for having me. It's a great pleasure to be here and to be able to have this discussion. And thank you for keeping on the agenda. Something that is not only important for us in Ukraine, but actually has a global meaning. And the result of this war and hopefully the victory of Ukraine soon in this war is going to be a prerequisite of all of us, all countries that have the same values, but also countries that want to deliver better to their citizens to restore as soon as possible the supply chains to address the food security needs, energy security needs. And I think food security and in general the agricultural issues that you just mentioned are key issues. First, the key issues for Ukraine. Ukraine has been a very heavily agricultural country. And when I say agricultural, it's not just growing, but it's food processing. It's all the value chain. We own 30% of the global black soil, the best type of soil for the growth. And actually the productivity gain that could happen in that field is huge. This is something that before the full-fledged war started, we wanted to develop as one of the key advantages of Ukraine. We are top five exporters of the majority of crops from wheat to barley. We're number one, used to be number one in sunflower oil. We are very high on honey. So we can feed the world. And once called the breadbasket of Europe, we can definitely be a global breadbasket. Now Russia specifically, in addition to the aggressive war, in addition to their war crimes, horrific war crimes against civilians, women, children, in addition to just having this unjust war, they specifically target the food.  The only reason Ukraine is not able to deliver the food, which we grow for so many regions, especially for Middle Eastern Asia, especially for African countries. We cannot do it simply because of the Russian actions. Now with the help of UN and Turkey, for some time, they were able to broker this grain initiative, which Russia agreed, but then sabotaged, of course, every month. They were trying to delay the ship inspections. They were trying to scare the ship companies. So it wasn't going perfect, even when it was there. But Ukraine always stick to what we wanted to do, to get the grain, to get everything out so that we can end. In addition to just selling it, we even donated. We have this program called Grain from Ukraine, where we donated grain and other countries, US through the USAID actually helped to pay for the shipments so that we can donate it to countries in need. So right now we are in a situation when Russia decided to stop it and block it. They're trying to create or put out all kinds of unreasonable additional demands. We are ready to continue, of course, but it looks like Russians really would like to weaponize the food again. Now we're trying to export as much as we can through the land borders using other ports. That's why you see during the past weeks increased attacks on Odessa, Odessa region, closer to the Romanian border. That's they're trying to prevent any type of shipment of the food, which will affect not only the shipment of what we have in the storages, but also the harvesting, because we are in the process of harvesting, which is I think remarkable that Ukrainian farmers have been able to plant, care, and harvest now the products in this situation when they're not only under constant attacks, but we are also one of the most mined countries now, the unexploded ordinance, not only in the residential areas or mines, but also in the field. So look, we will do whatever we can. Soon there will be again another General Assembly of the United Nations. This issue is going to be discussed, of course, and raised by Ukraine. We are trying to communicate with all of our friends and allies, especially in, you know, what people call the global south, you know, essentially saying we have to be very vocal. We have to press on Russia. We have to tell them that we know who is behind this. And they have to stop not only this aggressive war against Ukraine, but they have to stop threatening half of the globe with the food crisis because it's serious. Yeah, it is extremely serious. I mean, as you were talking, I was reflecting on a long ago visit to the Museum of the History of the Holodomor, which is a remarkable place and key of very striking. If you've never been there, I will just tell you that it lies kind of on this open sort of square where there are tremendous monuments and historical museums of great value. But the Holodomor Museum, to me, is interesting just in the context of food security, the weaponization of food, this constant refrain of Russia to return again and again to the food as the weapon historically is, well, it's tragic, but also I think it should be a lesson to us all that actually as the conflict continues, we should just expect that to continue on some level from Russia. So there are some challenges ahead, as you say. I know that other people are gonna also have questions about just kind of the humanitarian crisis because of course this relates not just to food security, but human security. And I again just wanna remark on how struck I was, many war zones, you name them, with the exception of Iraq, I think I've been to all of them. And so I'm just curious to sort of, for me it was interesting to see how well the civilian response was sort of coordinated. I wonder if you could talk a little bit about the humanitarian situation now, what you might predict for the future, the winters coming up, what's needed, what are the challenges that you see? Well, yes, you noted the Museum of Holodomor, which is striking that this year is the 90 years mark of that tragedy when Russians denied food, took all the food and people were dying from hunger. And this is probably the most cruel this and children when you live in the country where literally food grows everywhere and to create artificial hunger in the place which was the source of food for so many neighbors, not only for itself, is a very cynical and very cruel war crime, but again, not surprising what we see now because unfortunately we have a history of war crimes of Russians, whether it's Russian Federation, Soviet Union or Russian Empire against Ukrainians. But with regard to the humanitarian situation, on the one hand it's a very unique, I would say war, because I don't think we will find many wars when the government continued to execute its functions and never stopped it, not even for a day. So yes, where it was under occupation, even there our mayors tried to execute their functions and be with people and try to deliver food and organize something, that's why so many of our mayors have been kidnapped by Russians tortured or even killed, especially in the areas which they occupied but in general, all the areas where we have the control and as soon as we liberate, there is Ukrainian government there, both national and local. And all the society, and that's another key element, the civil society that work together with the government, so we are trying to address all these challenges with the help that we are receiving from the US. But that's why for the first time actually in our 32 years history, US is providing us with the budget, direct budget support, the grant direct budget support and this money are actually humanitarian money. They used to pay the salaries to educators, they used to support the IDPs, they used to provide the basic needs for people and it's done by the active government. Our banking system never stopped working during the war. During this war, Ukrainians, especially when there was the Russians were advancing at the beginning and occupying, people were actually putting money on their accounts and throwing their cards and trying to move to the territory which Ukraine controls because they knew that that's how it will be kept safe. Our digital system, DEA, which we have on our phones, so you don't need to have passport, a driver license with you, you have it all in your phone and that also allows us to communicate directly and to send money directly to people through this app, essentially governmental app and we have more than 20 million Ukrainians that are communicating with the government like that. So this is unique that we are trying to use the digitalization, the innovations and the government is adapting to this situation. On the other hand, of course, it's a humanitarian crisis of massive proportions. So yes, we're able to do and to use the help that we are receiving from the US and other partners to utilize it but then any thing you look, not all of our schools have bomb shelters, not all of them had and we need that in order for kids to continue their education. The majority of our kids are behind in their vaccination schedules, of course. The majority of people with very like rare or the diseases that require daily care, the different type of cancers, the people who require, who are on the different type of daily supports like dialysis or something like this. They cannot get it and not only under occupation, under occupation it's a total disaster. That's why people are killed and tortured and not have the access to basic needs but even in the places where either close to the front lines or everywhere, the need is pretty much in every sector whether it's healthcare or education or transportation and plus not to mention every day, Russia is trying to shell the infrastructure, Russia is trying to attack civilian objects but also others. There are more people that are killed or wounded. We have extreme number of people who lost their limbs and it's not only our brave warriors but it's also civilians throughout and of course preparing now for the winter. We remember what happened last winter when Russians specifically targeted the energy system in order to create the blackouts and to create the situation when either there will be additional waves of refugees because you cannot stay on them and unfortunately Ukraine has cold winters and when you don't have the electricity or energy supply it's not just about cold. You don't have the water supply, you don't have sewage, you don't have any other basic needs and that's a very, very difficult situation. However, and I just came back from Kiev five days ago, I was there when Secretary Blinken visited, the resolve of Ukrainian people is still there. I've seen it when I came back after Bucha was just liberated. It was April, 2022. I've seen it in September when I was there. I've seen it in December, 2022. It was cold and dark and I've seen it now and it's been throughout the 17 months that people say it's difficult. It's horrible, it's many losses but nobody would say that we shall surrender. We all know that surrendering for us, it just means that we will all die and fighting is at least we have a chance to survive and this resolve to fight is still there. Resolve to do everything possible and sometimes impossible. Yeah, that resolve certainly comes through and I actually was gonna ask you a little bit about, just a small follow up on the work being done I think with the help of the US and maybe some others on kind of building more resilient infrastructure for electricity in particular, if you could talk a little bit about that. Yes, we actually are working very actively with the State Department and Department of Energy and USAID is of course part of this group so we have like we have Rammstein meetings on weapons, we have literally weekly meetings on energy coordination and what we're trying to do and that comes a little bit related also to the reconstruction post-war is and to steal your president's quote build back better. So when we're looking at what Russians are destroying we're trying not just to rebuild or repair what was there but already think where shall we be in 10 years? So during this 17 months of war, as surprising as it is our cabinet of ministers adopted the energy strategy for the next 10 years. We agreed how we will replace the coal mines and everything else which had been destroyed with the renewables and we will change our mix and we will continue developing our nuclear stations which is the base for Ukraine so we a little bit more than 50% of in our mix comes from the nuclear and Ukrainians are very professional in the nuclear energy field and when we are talking to all of our friends and trying to get this additional transformers and generators and everything else of course I mean it's not something that is on the shelf and you can pick whatever you want it's something that you, it's rare commodity even without the war but when we're choosing between different options we already are thinking is it in line with the 10 year strategy? Is it in line with our post war reconstruction vision because we want after we win and there will be big need to rebuild and the destruction is really big especially in the areas which has been occupied for a longer time. We want to do it in a way that we can leapfrog from where we have been in 2022 ahead already so in all the sectors which are critical for Ukraine like agriculture, energy, IT and digital to do something that is 22nd century to do something that we can get more business to come to invest because let's face it just the recent study of the World Bank and they are doing this rapid damage report so it's called our DNA so the damage, just the physical damage not the losses of the profits not just the physical damage of what they have assessed was destroyed by Russians during the first full calendar of the full flashed war as of February 2023 amounted to 411 billion US dollars it's actually, it's not taken into account a big environmental damage it's not taken into account into the mining that has to be demined it's not yet, it doesn't include the destruction of the dam which actually is not just a simple environmental catastrophe and the losses that were right there for the water that was rushed down south but that reservoir was the source of the water for the agricultural district and for many towns and cities so the overall long-term effects are going to be also big so the damage is huge and in order to repair we of course would welcome any help from our friends and allies but the question whether we'll be able to leapfrog and do it is going to depend on will we attract business compliant, large business that will come and will do it together with us and that's why we have to open the door for all these innovations and do it in a very inspiring way so that we can become a hub for these innovations in our part of the world So interesting, I mean that's one of the paradoxes of war isn't it that I mean that it forces so much loss but then after the fact oftentimes there is a leapfrog in all kinds of ways sometimes social, sometimes technological, industrial but you mentioned something actually that you touched on Ramstein which maybe people in the room don't really know what that is but if you've spent any time in Germany you'll know that there is a base called Ramstein it used to be an expo for a lot of folks coming out of Afghanistan probably still is for most of the Middle East and a very important tie up, lash up for the collaboration and coordination of weapons transfers to Ukraine's frontline fighters we know that there's been a big decision recently on the big list of items that you are looking for in Ukraine we're getting close to two in fact one we have the F-16s Are you talking about this one? Yeah that's the one When did you pin that on? Before the decision Yeah that's interesting it was like a magic talisman So Denmark I believe has come through Netherlands has come through What do you think, what are you hearing from your colleagues in the defense side about the impact of the F-16s what will it take to get everything up and running what do you predict will happen in the next year what should we look for? Well first of all as you know we have been since February 24th working on the very long list of capabilities unfortunately we are fighting with not only brutal and criminal enemy but also enemy that is much larger than us so with all the support that we are getting Russia is still people wise and the methods they use to conscribe they can pretty much force any amount of people whoever they can catch and whoever did not leave the country to send them on the frontline but also they have a lot of weapons yes bad Soviet type you know sometimes not working but the quantity sometimes is a quality in itself so the list was large and first we were using both our old Soviet type of equipment and some Ukrainian equipment and then starting with the javelins and stingers and then everything else and now we are almost a hundred percent using the NATO standard equipment and to get more capabilities is very important because you know as military people say you know it's on the one hand it's a very World War I type of war you know like it's an artillery doles on a very long period very mined you know Russians simply are destroying whole villages you know the city of Mariinka the city of Bakhmut unfortunately when they were advancing they did not think about even the civilians they were just destroying inch by inch by inch moving forward and you know so we need to counter that we need to fight with that but we also need to do something with their supremacy in the air which is also very important so we were discussing a number of capabilities from the beginning and we are very glad that now we have the political decision on the F-16 because it's a very important part of the air defense it's a very important part of it it's an additional capability that has so many so much use and as you know our pilots are already at the training and we are working with our friends and partners big thanks to Netherlands and Denmark for agreeing to transfer the platforms and we are working on that so again it's a very complex capability it will take you know some effort to completely get it but you know it's all capabilities that we were getting for the battlefield today throughout the 17 months but we always were thinking also about building the army of the future how the Ukrainian military will look like after this because again with our aspirations as the future member of the European Union and future member of NATO we do have now the largest the most capable the battle tested army which will be an asset for the future transatlantic family so for us any capabilities that we add now it's not just the equipment that our brave defenders can use now but it's also it's also shaping through the battle the future force of Ukraine which again inevitably will be the eastern flank of NATO so it's a very we already have shown and our brave defenders have shown how quickly they can learn the US Patriots I think are showing remarkable performance saving lives, saving children in Ukraine but also showing how superior they are to any of the equipment that Russia has so it's not only a great capability for us to have and we are very grateful to the US for that but it's also a signal to so many other countries that whoever relied upon Russia to provide them the military support or the SAGs from the Wagner group they can no longer do either of them and it's a wake up call that with the size of the economy that they have with the lack of values and the aggressive nature and the imperialistic thinking which is totally outdated in the 21st century but they cannot even do the evil things as they did I mean they're still doing them to us but so it's a big I think geopolitical question for a number of our friends in the rest of the world and UN is such a great place so to talk about it and discuss it and see that we have to reform too we have to move forward we have to address the problem with the country that doesn't respect the UN Charter those conversations at the UNGA coming up to be a fly on the wall there would be quite something I have to say and I'm glad that you mentioned our friends the Wagner group thinking about those guys way too much time and so is my colleague Ben Dalton who is in the audience with us today my partner in crime a lot of folks in here actually have been very much part of the conversation about the Wagner group not only in New America but certainly I think in the United States and other places where people are paying attention so we've had some big action pretty recently I was planning to come to Kyiv actually but a certain friend of ours got in the way Yevgeny Progosian and Dmitri Utkin the operational commander of the Wagner group and of course Valery Chakalov who was kind of the money guy all killed along with four bodyguards and apparently a set of innocent crew members as far as we know we don't know a lot of questions about what happened there read my book whenever it comes out you'll know I may have some thoughts on it but I will say the question that I get asked now continually and I would like to ask you is okay they're dead but the Wagner group we all know has been linked to a number of war crimes countless really actually in the Ukraine context and we can't really talk about Syria and so forth so what's next with kind of accountability for Wagner operatives what do you think has changed and then where are we going with accountability and justice where they're concerned there is no lasting and just peace without justice and as you know President Zelensky when he says talks about the peace formula justice is such an important element of that so there we will we will not leave any stone unturned so I will come back to Wagner but in general we are doing the criminal investigation in Ukraine the prosecutor general is doing a remarkable job we have more than 100,000 individual cases that already are opened by the Ukrainian legislation US is helping us a lot and by the way Netherlands on data collection and evidence collection and also how to talk to victims because a number of these crimes are the sexual crimes or the brutal executions of civilians or tortures and you have to do it also in a way not to re-traumatize the victims so we are doing this and we are not only investigating we are also already in the courts and some people are not only indicted and some people are indicted in absentia and some people who we caught the prisoners in Ukraine but they are also sentenced and believe it or not and it's a big challenge which our prosecutor's office is paying very big attention to is that the due process of law is there and that all Russian perpetrators actually get lawyers to represent them and to defend them and this is the most difficult issue to find enough attorneys, Ukrainians who would defend but we have to do it, you know this is what differentiates us from Russians as well in addition to that a dozen of countries already opened their own criminal investigations we are fully cooperating with all the evidence to them and as many countries that would do it, we are very grateful there are three international courts all of which have cases whether on genocide or other one of them already indicted the ICC, both Putin and Lvova-Belova and rightfully so for the kidnapping of our Ukrainian children which is part of the genocide we are working very actively on the crime of aggression which is the mother of all the crimes and it's a very well documented crime because Putin did all of that online, literally he was publicly documenting every decision that he took which led to announcement of this special military operation, i.e. war and then he acknowledged the same actions in 2014 and 15 after he stopped pretending that it was some green man in Crimea of course it was us so this is very important and frankly of course the prosecution in Ukraine goes faster, the international courts will take some time the tribunal for aggression will take probably even more time but it doesn't matter people need to know each of this crime will be in the court of law of accountability now coming back to those who are dead yes, the leadership of the Wagner group is brutally executed in a very public way on the two months anniversary of their work to Moscow but they were not the only ones who were committing the crimes the thousands of Wagner operatives which again very documented in Ukraine we have a very big base of the evidence but all Russian troops as well so it's not just some rogue units it's not just some private type of military units which again we know they are not really private it's the Russian army and the Russian armed forces and the Russian president who created them and who were providing them with weapons and who were giving them instructions and yes there were some disagreements between but it's more a quarrel between the different lines of the same kind of organized crime organization rather than really state and the private doing it separately so we'll continue investigating, we'll continue indicting them there are other private groups not as successful I don't know whether successful is the word to use of course but not as capable but there was attempts by Russians to create these groups there are a number of them which I didn't hear they still are operating outside of Russia I mean hopefully without capable of the evil and criminal head like pre-Gorzhian we will see decline of the use of the Sugs in African countries and others but if not then justice for everything they've done in Ukraine will also help us to get them out from so many criminal missions throughout the globe so I think it's a very important issue and to get justice regardless of how much time it will take is as important as winning this war because that justice wherever it will be served fully will be the final victory in this war together with the reconstruction No justice, no peace that's how it goes we're getting kind of close to time and I know that there's probably a question or two in the audience so I want to open up the floor and maybe get back to the question on the ICC and the question with the children and what's going on there but let me go to the audience first and go with my colleague Peter Bergen Thank you Candice mentioned all the amazing work you were doing in Washington the central problem of course is the presidential election of 2024 quite a number of the Republican candidates sort of saying that they implying that they would reduce or maybe even end A to Ukraine I think 71% of Republicans now say A to Ukraine should stop so how do you deal with this sort of American political scene where the Putin is surely looking at these polls and making his own conclusions Well first elections are very important in any democracy and this is something for what we are fighting in Ukraine, for democracy for the ability to choose our government and to change it on a regular basis and I would never call an election a problem to be honest this is what people should do and this is the basis of all of us continue developing whether we like the results or not so first of all I have big trust in American people regardless of party affiliations when we explain it to people I always feel the support so when we tell people why and we tell people the more information about what's going on and we tell people that we are fighting for our homes for our loved ones that we were attacked by a brutal bully with no pretext with no reason whatsoever it's very much our own war for independence and freedom I think the majority of Americans understand and feel it because this is what Americans have in them this is what this country is built on these values so when people either do not support it or say it's not in the American interest it just means that we didn't explain it well we have to do more we have to go and talk to people for more information we have to provide them with this knowledge and that's why and again I always thank the journalists because their work has been a game changer during this phase of the war in 2014 exactly this happened Russia attacked us exactly the same way the Shem referendum in Crimea was no different from the Shem referendum now but our voice was not heard it was just our voice it's the cameras and the journalists who have been showing the world what's going on so it's very important to continue to inform people by informing people we also have to tell them more about how the American help to us work that yes there is 113 billion that Congress very generously appropriated to provide for the Ukraine and related to Ukraine support but not all of that money goes to Ukraine so yes we are getting the direct budget support which is about slightly above 20 billion for which we are very grateful of course again I always say how grateful we are so that is the money we are getting in order to be able to continue the fight and sustain the effort all the defense assistance which is much larger than this we are not getting the money we are getting the goods and a number of resources that Congress provided goes to replenish the stocks of the Pentagon it goes to increase the production here I just recently last month visited the Lima Ohio plant which produces and I have to tell you it's additional jobs there and the majority of people in that plant I mean I felt like I'm visiting friends there because they are proud that they are producing this excellent quality and that we will be able to do it similar with the plants that produces breadless and then you see the videos how when our brave defenders liberated Robotina in the south and there were still people there civilians you know of the age of my mother and they were put into this American Bradley to be evacuated to safety because everyone in Ukraine all defenders know if there is a shell in you go inside the Bradley not outside because this is the place we will be kept safe so we just have to explain that it's you know we are very grateful again for the weapons but they are produced here and we are doing it together and they are developing providing jobs here in the United States and it's also for the benefit of both of our countries you know we need the goods in order to defend our families but you are producing them here and third which is also very important we are defending not only us we are defending the whole European part which Putin has been very loud and clear that he wants to attack and he wants to he has problems not only with Ukraine he has problems with everyone who was able to get out from this empire the Russian, the Soviet or whatever you call it so he is threatened sometimes Poland he is threatened definitely all Baltic states he was talking how the Finland and other European friends and stuff like that so you know right now we are defending other including NATO countries at a very modest I would say military budget and we are doing it ourselves we are not requesting any of our friends to fight for us we don't need other boots on the grounds we just need the weapons but if God forbid we fall and Putin occupies completely Ukraine and kills us all he will not be stopped he will be emboldened by this and he will inevitably go further and then unfortunately a number of NATO countries will have to help to defend other NATO countries so the fastest and the most efficient from the financial standpoint this is the minister of finance is actually to help Ukraine to help the war now while it is still in Ukraine so whether it is the moral argument whether it is the shared values whether it is the effectiveness and efficiency argument because it is in the US national security interests to defeat an aggressive autocratic regime that not only attacked Ukraine let's keep in mind that they have attacked Georgia in 2008 committed horrible crimes in Syria that they have poisoned people in the streets of the United Kingdom that they have interfered and send their Wagner sacks into so many other places not to mention all the crimes that they have committed while they were in the form of the Soviet Union so many wars and so many so it's an all of our civilized people interests to live in a safer world which will return to the security architecture that we had after the World War II because that peace and again I do not imply that the peace was everywhere but at least the lack of a great war which would involve European continent has been a basis for the prosperity that we all enjoyed in the collective western countries and in some newly developed economies and we have to get back there as soon as possible if we want to continue delivering to our people to our citizens so I think when we explain it clearly to the American people they understand and they support so it's a task for all of us to explain it better so I see I keep getting these high signs over here I wish we could just keep going on I really hope you come back let me just say that because this has been a wonderful conversation I was going to ask you how many states you've been to but maybe we'll have that bet later how many states have you been to actually well not too many because I'm trying to stay here and I'm trying to travel for sure but I've been to California I've been to Ohio I've been to where did I go Massachusetts I've been to Pennsylvania I've been to Florida and New York of course but that's not we're going to Illinois I still have to come back to Indiana that's right you've got to get back to Indiana well listen I want to ask one tiny question but it's going to have to be a short answer I'm afraid I did want to come back to this question on the ICC decision to charge Putin to charge the High Commissioner on Children's Affairs big decision and obviously seems to have constrained Putin's movements what's your response and how has that decision affected the the efforts to repatriate Ukrainian children from Russia well first of all we are very grateful to ICC for taking that case for moving ahead with that case this is probably maybe for me as mother is the most horrendous crime of all so 19 more than 19,000 cases registered already in Ukraine the children we know have been abducted to Russia according to the estimates of our commissioner on these issues it's actually 200 to 300,000 the Russians themselves themselves claim is even larger amount of children they rescued as they say but you know killing of Ukrainian children abducting them to Russia putting them into this one day adoption and they change their own laws to be able to put them for the speedy adoptions indoctrinating them putting them through what they are telling us the teenagers that we were able to get back through the re-education camps which sounds like from the book about the World War II and what Nazis did to children is horrible everyone has to be punished for that and the fact that ICC ruled and indicted them is such a notion of justice an understanding of this problem so we are grateful for this government that is working with us on this issue we are trying to get back as many children as possible very difficult unfortunately we were able to return a very small number of them we are very grateful to everyone in congress there is a number of resolutions that submitted on this issue not only condemning but also calling for some actions with regard to how to do it it's a very difficult issue of course we need to win we will know first of all for sure what is the situation on the occupied territories but also this is what we will start working more actively we are working as active as we can but to have the win this of course is going to be a big part of our victories to get all our children back our first lady, our president and everyone in Ukraine says we will not rest until every child is back let's hope that happens thank you again ambassador for joining us such a pleasure and an honor I hope the audience will give you also a warm applause


Okay, no. Maybe it's the other one. Should I try the other one? Yes, yes, yes. Okay, no. Still not. Yeah, okay. Okay, this one too. Okay, yeah. 15 minutes, 15 minutes, 15 minutes, 15 minutes. So what I have to do? 6, 10. Okay, great. Okay, good afternoon. Can you hear me okay? Yes. So, I promised you a Kubernetes tutorial. So first, Mia Culver, I suggested I would give you hands-on access and so on, and I just realised when practicing with people in a two-hour situation that this wouldn't translate to a one-hour session. So I really do it in mode tutorial of doing things and showing you what I'm doing. But if you want to follow along yourself, either you've got your own Kubernetes cluster already or you do it after the event. Okay, I hope that's okay for everyone. Okay, so I'm Michael Bright. I'm a developer advocate at Containers. Containers are the creators of traffic. Some of you may know it. It's a reverse proxy load balancer and can be used as a Kubernetes ingress controller. That's something I'll be showing as a demo. Just out of interest, what's people's experience here with Kubernetes? Are you all fairly new to Kubernetes or are some people... So who is totally new or pretty well to Kubernetes? Okay, good. You're the people I'm aiming at, that's fine. And just out of interest, is anyone here using traffic? Do you use traffic already? No. Do people know about traffic? No. Okay. Good. Thank you. I just say this tutorial is open source. The initial version, which was swarm, Kubernetes and mazars, we've run about three, four times. And now this is the third or fourth time after I've run the purely Kubernetes version. And it's all on GitHub. And you're welcome to open any issues on it or just send me feedback directly. I'm keen to have this evolve. Okay, so this is what we'll be covering. Basic concepts of Kubernetes, the basics, the command line client. Then we look at running pubs. So if you know about containers, and you know that... The basic unit of execution within Kubernetes is a pod, which is one or more containers. So we look at that concept and running pods and what we call deployments. We look at how to perform and rolling upgrade across our cluster. And then how to expose our applications as services. There are several ways of doing that. And I will show just one way, which is the ingress controller. And this time, which we'll very quickly just talk about Helm, which is quite an interesting tool for Kubernetes. If you want to do this afterwards, then there is a page that talks about a set of options for just simply running your own cluster. The simplest way, of course, is to run Minicube. Minicube being... It used to be just a VM for running Kubernetes as a single node cluster within a VM. There's also a non-VM option as well, so you can just run it with Docker to simulate a Kubernetes cluster. Just one thing, so you'll hear me talk about traffic and so on. I mentioned that I work for containers, for the creators of traffic, and we're hiring, by the way, for remote workers too. Okay, let's get started. Okay. Don't be put off by the fact that whatever we're showing you when I execute stuff, mainly I'll be doing it within the Jupyter environment. You don't have to worry about what that is. It's basically a notebook where you can mix markdown and executable code, and so it's quite a nice way of documenting things as you go along. So I'll start with some slides and then go back into the notebook. Kubernetes, first of all, what is it? It's basically a cluster manager managing a cluster of containers. It could be a whole data center that you're going to manage with it. The principle, of course, is that as we move towards microservices, we've got more and more containers running in our data center, and it's becoming impossible for operators to take care of all the little details of deployment and you really need some sort of deployment platform which is going to allow you to say declaratively, okay, I want these services running. Kubernetes, go away and take care of where they actually run.  When there is a problem, I imagine maybe a node of a cluster that goes down, that would be a physical node or a VM, or a pod, a container, goes down, you don't even want to be informed. You just want Kubernetes to relaunch neural resources in the place. So if you know about Cloud Native and the discussion about pets and cattle in the old world, all software and VMs tended to be pets. There was a problem with the VM and you would go in and try and fix it and do your best to care for it like you would for a pet, whereas here we're much more in a scenario of cattle where a container dies and you just launch another one in its place. Okay, very different philosophy where we tend to declare what we want to be running on our cluster. So historically, Kubernetes, it's an open source project that was created by Google. It was based on their experience with deploying containers within their infrastructure, so notably the internal projects of Borg and Omega. And they really wanted to bring those same principles into an open source project. They donated this to the CNCF, Cloud Native Computing Foundation, in 2015. And a lot of companies bought into that as a container orchestration platform. And it's currently at the 1.9 release, surely 1.10, obviously. Okay, this is just a Google Trends just showing the tendency of the different container orchestration engines. There are more, but I just compared Docker Swarm, Apache Mesos and Kubernetes. We can see quite easily that the blue line, which is Kubernetes, has just taken off completely. And in 2017, we've really seen the effect of that with some surprising or not announcements from Amazon. Everyone was expecting them to launch Kubernetes services and they did that with their ETS and also Fargate services. To me, more of a surprise was Docker, when they announced at DockerCon that they would support Kubernetes as an alternative container orchestration to Docker Swarm. We've seen new managed Kubernetes services. So this is where you will pay to use basically a Kubernetes control plane, a dashboard to deploy a cluster in some other infrastructure. It could be out on Amazon EC2, for example. It could be on-premises, actual physical nodes on your site. So we saw new services launched from Microsoft and HPE and no matter how many others, they managed Kubernetes space. There are many other distributions. One of the early adopters of Kubernetes was Retat. Their OpenShift pass is based implicitly on Kubernetes. Others have been, well, CoreOS with Tectonic, who have just been bought by Retat. Canonical also have their distribution. And we see new tools. There are many, many tools appearing around Kubernetes. There's one which is looking particularly interesting today, which is Helm. She essentially is a sort of, allows you to do, create whole stacks of applications, a bit like a stack with Docker swarm today. But also with a catalog of applications that you can pull down, or full stacks that you can pull down. Okay, so if you look at the Kubernetes architecture, this is a bit small, this diagram, but it's really just to show three things. Basically we have master and worker nodes. Normally, in production at least, it's on the worker nodes that you actually run pods with containers in them. And the master node is the master nodes, are the control plane nodes that will monitor the workers and distribute tasks to the workers. There's also some sort of distributed key value store, so ETCD, which is used to manage the configure cluster, and in particular to say which of those masters, there's only one here in this diagram, which of them will be the leader. And so this is something that, I don't know what size Kubernetes clusters scale today, but essentially, as we're talking about, you might have a whole data center, or you might have several clusters in a data center. So if you look just a bit closer at the master nodes, there are three main components within a master node, apart from the ETCD demon itself. The API server, so most things you will do with Kubernetes will be either using command line tool like QCTL, or the dashboard, or some other application built from the API. That is the way that an operator will interact with the cluster. There's a scheduler and a controller. So as I said, Kubernetes is very much based on the idea of declaring what you want to be running on your cluster. It's very much a declarative approach. And in the same way, the controller is actually, think about 50 software control loops, each of which are checking some particular aspect. And they notice a discrepancy. Let's say you declare that you want 10 replicas of the Apache web service. For some reason, there are only nine. A part has died or a node has gone down and so on. The controller will detect that and it will tell the scheduler, okay, we need one more Apache web service. The scheduler then, based on constraints, if there are any, will then decide where that new Apache web server should be deployed, on which worker node. Are there any questions about that? So if we look at the worker nodes, a few more elements in there. So the main contact point is the Kubelet. So this manages the API that the master node will use to talk to the worker nodes. Obviously, there's a container engine. So typically today, it's Docker, but Rocket is already supported. We're seeing the creation of new container engines. So there is a specification, the CRI, container runtime interface, which is also being used by Docker already. And Red Hat have created one instantiation of that CRIO, which then provides a new container and time. So who will see choice in that area? The Kube Proxy, so... I'll open up the web pods first. The pods are one or more containers in each pod. They are on a separate network. So to get access to the actual applications running in those pods, the Kube Proxy is the one that will provide the routing onto those internal networks. There are various add-ons like Kube DNS to allow service discovery and so on. And there's the dashboard, which itself is an add-on. And in fact, all of those elements themselves can run in containers on a Kube system, as we will see. So the pods. I said one or more containers. So the philosophy, just as like with containers, the philosophy is really to have one or more containers in a pod. In a pod... Sorry, I meant one or more process in a container. In the same way, a pod will have one or more containers in it. The idea is that you should have, you know, one principal functionality provided by this pod. And if there are other containers, then they should be providing some supporting function to that main container. So for example, I don't know, could be a Apache web server again. Maybe another container, which is maybe Fluent D, maybe doing some distributed logging to elsewhere. Or another good example for a web server. You might have a container, which is doing a Git repo sync. So it will be pulling from a Git repo any updates, which are actually the static content of the website that you're serving up. So those are the two quite distinct functionalities, but the Git repo is supporting the main web server functionality. And those containers, OK, share namespaces. So these will have different process IDs within them. They will have the same IP address. They're sharing the same IP address, and they mustn't use the same ports and so on. So it's an abstraction, which is a little bit like a machine, conceptually. Oh, an important point is, as I say, a pod will be one main functionality and then some supporting functionality, but really tightly linked. And the idea is that as you scale, this is the minimum functionality that you want to scale. I want to scale from 10 to 20 web servers. What I need, let's say, the Git repo associated with each of those. So we'll see in a moment the command line client. I've heard about having it working from the command line all the time, but I should try and come back to the dashboard as much as possible because it's a pretty nice dashboard. So let me, rather than show you slides here, let me actually launch that dashboard. So, OK, Kubernetes dashboard. First thing to notice is within our cluster, we have three namespaces. When we run, we can create new namespaces, of course. When we run new pods by default, they will obviously go into the default namespace, so which is empty at the moment. I've not created anything. Other namespaces are Kube Public and Kube System, but if we can change into Kube System, and there we see already that there are some things deployed Kube DNS, Kube dashboard, so the dashboard is showing itself in the system there. We can interrogate the number of, well, replica sets, which haven't presented as a concept, and pods. So in fact, all that is a set of pods that represent our Kubernetes cluster itself. And in this example, I'm running an A3 node system, one master and two worker nodes, default. I'll go back there. Looks like I didn't do cleanup. OK, so as I mentioned, for this part of the lab, I'll be working with this three node cluster. It's actually a complicated environment, but later when doing ingress control, I will switch to another mini-cube-based environment, but I wanted to show a bit of working in multi-node. OK, so let me just do some cleanup. I see I did still have some things running. The advantage of the notebook is that it keeps it historic of everything I've done, so it allows me to archive with the whole tutorial, but to run it, it'd be better than I likely have stuff. So I mentioned the command cube-ctl. Normally I'd spend, or cube-cut all, and normally I'd spend time just showing a bit of the option so that, but I really want to try and keep them to one hour. So I do a cube-ctl get all. So basically with get, we can interrogate services, deployments, replica sets, and so on, which we'll be seeing in a moment. If we do all, it will give us those types of resources from the start. So if we look at the most basic way of starting pods in a Kubernetes cluster, the quick and dirty is cube-ctl run, and what this does behind the scenes is it creates something called a deployment, which itself creates a replica set, and then will create pods that will actually implement our application. So what are these concepts? A deployment represents essentially a deployment, so a deployment of a particular version of an application, and we see that when we do a rolling upgrade, this is the resource that will take care of upgrading from one version of an application to another. It creates a replica set, which is responsible for obviously creating the number of replicas that we've asked for. And in this case, I specified replicas equal to, so we have two pods. Of course, at a lower level, then each pod will have one or more containers. Okay, so I'm going to do a live-satellite hold. I'm going to do a cube run of this image, of this image, K8S demo, colon1, so colon1 is standard Docker image notation, saying it's version 1 of this image. This is basically going to create a deployment called the same thing, K8S demo. I'm going to apply labels. I haven't talked about the concept of labels. It's something that's very important in Kubernetes. Essentially, Kubernetes already applies labels to various resources, and you can explicitly apply labels, and it's a way of marking things. For example, if you're running on the same cluster, you're running development pods, test pods, staging pods, and production pods. I don't recommend it, but if that's what you're doing, then you could label, okay, I'm launching this just to test, and I would say, you know, I don't know, status equals test or something, and it would be a way of, later, if I just wanted to clean up, I could delete just the test pods, for example, not touching my production environment. Generally, labels are really useful. Another thing is, imagine the nodes in your cluster. You might have nodes with particular physical properties. You might have nodes that have a GPU graphics card, particularly good for the high-intensive compute operations. You might have nodes that have SSDs, others that have hard disk drives, and they would be useful for different types of applications. So you could use labelling to then select and say, when you run a pod, say, okay, I want to run this, only on nodes that have a GPU. Anyway, just to explain labels, as before, I will say, we want two replicas, and I want the pod to expose port 8080. I'm going to do a kubectl get-all immediately behind that, just to grab it in an intermediate state. Okay. So that first line, deployment K test, demo created, that's the output of the run command. And then the get-all, curiously, lists the deployments twice, and the replica sets twice, but I'll start looking from here. So it's launched deployment. It says here, desired two. It's saying that basically we want, I think that's pods rather than containers at this level. We've declared that we want two replicas of this pod. There are currently two. It's already launched them. They're up to date, but zero are available. Similarly, at the level of the replica set, there are two of them, desired and present, but zero are ready. If you look at the pod, we can see why. It's in the status container creating. So typically it's actually pulling the Docker image off the Docker hub. Probably not the case in this example, so I'd already done that, but still there's a little time of container creation. And here, the ready, we have zero of one. It's actually telling us that's zero containers out of one, desired. A particular pod specifies only one container. Okay, oops. Okay, I'm going to run that command again, which will be basically the same, except this. Now, we see two available, two ready, and of course, both pods are running with one container out of one. Okay. Just to show an example of the sort of input we get with kubectl. We can also explicitly interrogate the pod's deployments and so on. And with get pods, taking the option wide, we get just a little more information we had to hear before. This is the IP address of the pod that's created, and this is obviously the node on which the pod was assigned. So normally you run pods only on the worker nodes, though if this was a single node cluster we only have a master. So for development purposes, you can taint a node, allowing that to run pods as well. But if you were to scale to 10 pods here, we would see that they would all be distributed across the worker nodes, node one and node two. Interesting as well. I can see the nodes we have here. What I'm going to do, so there's a command called describe, which gives much more detailed information, a lot of information. If I look at those nodes, we can see the actual IP address of the nodes themselves, and we can see that, well, I know that these are slash 16 subnets. Knowing that, we can see that these are a different subnet than the pods. And in fact, I think that every deployment we create creates a new subnet specifically to those pods. Okay. So how can we access our applications? I'm going to use that information about the IP address of the pods. Again, this is not how you do it, but I'm working through to the different ways of accessing your application. Given that we have the IP address of the pods with a bit of bash magic, we can pull out those IP addresses. Okay. And then we can do a curl on the port ATHG that we were exposing. Okay. This is what we get. And the other one can't see both, but basically that's a different container name from the other one. Okay. What happens if a pod dies? I say, well, you know, we will automatically launch another one. So I just get to get the ID of the first one of those pods. And I'm going to delete that pod. I do quickly get pods behind it just to show, okay, we're in this intermediate state. So that first one, and then in S5, is already in the terminating state. It's not quite finished yet. And we can see that obviously the controller has detected that that pod is going down. Well, I actually would even know from the schedule. And it's creating another container in place.  And we'll see that with rolling upgrades. Yeah. Okay. So now we just have two nodes again. But one of them, the new one, has only been running for 49 seconds. Okay. So we've seen how to launch pods via a deployment and replica set. Actually, cutting down the tutorial to one hour, just one point that I didn't mention is. So I've been creating those from just a kubectl run command. That is the quick and dirty method. The way you would create resources and we'll see it a bit later, is we tend to put those in a YAML file to actually, again, declare the overall set of resources that you want with your system. And we would actually do a kubectl create or apply of that YAML file. Which is quite nice because you can also, and when you want to stop things, you can do a kubectl delete. Or if you want to modify parameters, you do a kubectl apply. Okay. So when we start doing stuff here, I'm going to use a different cluster. It's no longer this multi node cluster. It's just a mini cube, a single node mini cube. And I'm going to show other ways of accessing our application. Obviously, what I just showed, is I'm going to use a particular pod directly. That's no good. I had to know the IP address of a particular pod to access it. And also I was accessing directly an internal network. So that's a no-no. So let's look at how we can use services to expose those pods. So the basic principle is you will have a service which is going to expose one or more IP addresses, so that some external user can get to those pods. Okay. And there are different ways of doing that. And it's important as well, not just because users shouldn't have access to a pod which is internal, but as well, because of this whole nature of when we change things, pods will come and go. We can't keep track of those pod IP addresses. Okay. So there are many ways, actually. At least a couple I haven't listed here. These are maybe three most interesting ways of creating a service to access our application. The first one is node port. Wow. I'll skip to the following slide. So the idea of node port is that you will expose the IP address of each worker node at a particular port that will be used to access one application. Okay. So, well, yeah, but that means you don't have to know the addresses of the pods anymore. You don't have to access that internal network, but you still have to know the address of the worker node, which might go away and be replaced by another one. And it's also no doubt an internal network as well. So that's really something that can be useful, maybe, for internal testing or something, but probably not something you'd want to use in production. As it says here, then, yeah, you'd also be using one port for each service to expose. It's also in this 30,000 upwards range. There are some disadvantages, but it can be useful for testing. Another way is load balancer. So typically, if you create a cluster on GKE, the Google Kubernetes engine, you could use this method, and an external load balancer will be created for you automatically, and you can use that. That's sort of like the disadvantage of this. It's great when you're running in some cloud provider and they will provide the external load balancer for you. And this will provide you an IP address, a known IP address, and this will be on some safe subnet that you do accept to expose. And then it will load balance between the worker node or between the pods, I should say. So this is good for deployment. But there's a third way. That's the one I want to present to you, which is to have a Kubernetes ingress controller. And this one provides more functionalities. So it will provide public addresses you can use to get in to the services running in the pods. The thing is that you will no doubt be having a large number of services in your cluster, and it becomes complicated to route between them. The concept of ingress controller allows us to do different types of routing to get to those services, like path-based recruiting. So it's an example where, based on the host name that we attack, it will be directed to a different service, and it's the ingress controller that will provide us that functionality. So traffic is an example of a reverse proxy load balancer that can be used as an ingress controller. That's how I'll use that in a demo. Traffic allows quite extensive combinations of host name, path prefix, sort of a URL path, and port-based routing to be able to direct to different services within your cluster. It also allows to do hot reloads of configuration. You don't have to restart traffic if you reconfigure it to add another service or to change an existing service. And it also has less encrypted support, so it can actually do automated encryption. So I'm going to go away from this previous dashboard. I'm going to close that. Okay, so this is Kubernetes dashboard of another Kubernetes cluster. So I said this one is based on Minikube, so we should see here it has just a single node. I think I have nothing running. Let me check that. Okay, so I'm going to show how to deploy traffic itself as an ingress controller. For the moment, I didn't show you explicitly, again, if I cleaned up properly. No, I didn't clean up properly, apparently. Okay, let me... Okay, so I just got a scripted demo. I'll talk you through the different steps. So I'm going to install traffic itself. And we see that's quite simple to do with the appropriate YAML file. I'm going to deploy some pods of an application that shows Jesus, as you have seen. Traffic is really into Jesus. And I will deploy a service for those different applications. And then I will set up different paths to those services. Okay, so the first step, I won't show the actual YAML files there. I've actually applied the configuration to install traffic. And now I'm going to create my applications, my services. Okay, so we have... I'm looking for the image. We have an application called cheese. I'm going to deploy that. So I've deployed three services, actually all based on the same image, called Stilton, Cheddar and Wensleydale. And we can see that those three services are running now. Okay, I actually still have... Okay, something wasn't cleaned up properly. So I have an Ingress running, but that's not going to create this problem. So I'll get an error here, I think, because the Ingress is already created. Okay, so it actually might work better than before. So let me come clean. I have a working version of this demo. And it was the first time I installed it myself. So this may or may not work. If it doesn't work, I'll switch to the working version that I already had. But it would be really cool if this works. Well, now let me change, because that's not... That's the working version. And this is the maybe working version. Okay, so just to say the address up there, which is tokyo and readall, is trafficui.miniCube3. And I have an ETC host sentry with the IP address of that particular node. Okay, so this is a traffic dashboard. I didn't mention that traffic actually can work with many different back-ends, so, obviously, Kubernetes. Docker Swarm, Apache Mesos, Nomad Rancher, and also some other back-end services like Consul, ZooKeeper, Eureka. There are a lot of back-end services. What we're seeing here is, okay, in terms of providers, we have one configured, which is Kubernetes. So when we configured it as an ingress controller, basically enabled this tab. What we're seeing is a number of front-ends. So these are actually the ingresses for the services I created. Now, this first one actually is an error. This was me trying to set up the system earlier. And then these following ones should be good. Okay, actually, I'm a bit surprised by the address, but okay. And these front-ends are going to basically root through to the back-end services. Okay, yep, HTTP, it always helps. That's fine. I don't normally use that address. Service and available. Okay, never mind. So I'll go to my working version then. So I'm cheating. Whoops. So I'll start with the dashboard of the other system that I have. Okay. This is more, to be honest, this is more than I was expecting of cheddar.mini-cube, Stilton.mini-cube, and the last one, Wensleydale.mini-cube. So these are the host names. Again, it's the same IP address. I have entries for these in my et cetera hosts. So now, if I try to access cheddar.mini-cube, I will be directed to the cheddar service, Wensleydale. Okay, and I think you get the idea. But I fancy that it's Stilton anyway. Okay. So that's, I just wanted to show you the idea behind an ingress controller where basically you can configure different routes. In this case, basically if the host is cheddar.mini-cube, then we're going to route through to the back end cheddar.mini-cube, and this is going to direct through to the pods. What traffic is doing here is, there are different ways it can be configured, but it's actually interrogating the Kubernetes API. So we don't have to configure traffic to say, okay, these are the services. No, it's traffic which has detected that those services are running. If I normally delete stuff, delete the pods, then those back ends will disappear. That's one of the advantages, at least of traffic, that it has this dynamic configuration reloading. And generally this is sort of functionality that reverse proxies ingress controllers will provide, but maybe not with such dynamic reloading. So in this case, it's very simple filtering. We're basically saying it comes in on your cheddar.mini-cube hostname with path slash, then go to this service. We could do things more extravagant. We could have a regex in there and this sort of thing. So there's a lot of possibility for quite flexible routing of different services. Are there any questions on that before I go on? Yeah. Traffic behind another traffic. Behind another traffic. So you have one load of reverse proxies and then you need to email in their own reverse proxies for their service. I guess you could. I'm not sure to understand the use case, but yes, a lot of the use cases we have are treating where a head is already been modified because they passed through different systems. I guess you could have a chain of reverse proxies. Does that answer your question? Yeah. The use case was more if there was a multi-tenant nice cluster, and each team controls the routing of the service. But then there's still a global. So, yeah, I think that would be the fifth possible. I tried to remember it as an idea for a demo. It would be pretty cool. Okay, finally. Go back. Let's say this is really the shortened version. This is the cheap one-hour version. Speaking of which, this is the first time I present traffic. The second time would be this evening. And maybe think earlier of, say it never happened to you. I got in a plane once and the pilot proudly announces I'm trying to put people at ease. Don't worry. This is this plane's first flight. So everyone's happy, except the engineers. I think, what? Engineering, reliability curves. It's the same. My first time presenting traffic. So a few hicks. Hopefully you'll go a bit better for the second time. See if I can debug in the MRT later. Okay. So I mentioned, I mean, there are tons and tons of tools coming out for Kubernetes. I even bumped into a guy this morning, the speaker room, who presented three new tools. Something that he's working on. Pretty cool. But there's one in particular, Helm, which is particularly interesting. So it was a company called DICE who started working on this and some of the tools. And DICE got bought by Microsoft, Plastium. And that continues to operate as an open source project. But there's real momentum.  And already, in January, I think, there was a Helm summit. Wow. Okay, so what is Helm? It basically allows you to specify in a YAML file a whole application stack. And then those YAML files themselves are available on the hub. That's pretty nice. That means you can actually browse. I go, do I have it over now? Okay, let me go there. This is the Helm website. Down here, there's the option to get charts. I go over there. Cube apps. And explore apps. So if I want, I don't know what you're with this, I could even, I could even install traffic using Helm. But, you know, but okay, you can see a whole load of application stacks. There's a command line tool as well. It allows you to do quite simply Helm search. Helm fetch, I think it is. Then Helm install. Maybe that's not quite the plan. But it's really nice to be able to install complete applications across your Kubernetes cluster. Okay. So I'm going to stop there. So in the end that was a 55 minute version, not the hour version. So, any questions from people? Yeah. So let me see. So jithub.com, container orchestration, labs. Actually then it's Kubernetes, orchestration Kubernetes. And it's actually on a branch across Asia. And yeah, I'm interested in any feedback, any issues you want to open against the tutorial. This is a one hour version. I mean, could actually run for anywhere between one and three, before hours next time. Okay. Thank you for your time.