My name is Douglas Shield. I've been working with C4 since I think the first time I came here was 1996 Played a lot of tennis with Jeff Sayer and decided to stay so then I was here for ten years Up until 2008. I'm still a senior associate. I've been coming backwards and forwards more or less every year up until last year I didn't come For about four of the last five years. This was the view out my window in Uganda looking out at the rain Often falling on this forest here in the middle of a continent in Africa thinking about where does it all this rain come from it kind Of plays in your head when you're looking out the window. Where does this rain come from? We're in the middle of a continent Yeah, how come there's so much moisture in the air? So a good question do forests attract rain? So I'll try and explain why I'm enthusiastic about this question It's not very hard to justify an interest in water. Of course water is absolutely Basic do you want to move around a bit? I'm worried people can't see if I'm waving my arms there Water is absolutely essential for lives and for livelihoods life on the planet requires good water supplies Declining access to water is associated with as a threat to food security and of course the human health you need water to live well If you look up the official statistics There's been 11 million people killed in drought 7 million people killed in floods in the last century or just a little bit more and Many many people 5 billion people needing emergency aid so reliable access to water is a really key thing Whatever you're interested in if it's the environment if it's people if it's livelihoods if it's food security. It's fundamental So bringing this back to forest. What do we know about forests? We know that they're a major source of atmospheric moisture You can read the literature and find out the deforestation is implicated in declining amount of cloud cover a decline in reliable rain And in various parts of the world there are good correlation studies showing that a declining forest cover leads to a declining amount of rainfall I should say that I know I'm talking to a kind of a cross-disciplinary group So if I start using jargon or I start talking too quick because I've had too many coffees this morning, which may have happened Do interrupt because I'd like it to be informal if I see that people are lost, but then I'm not doing my job properly So please interrupt Too quick So what do we know a bit more about forests and the evaporation? We know that a closed canopy tropical forest will generally Evaporate amount of water, which is equivalent to about one to sometimes even more than two meters a year So this is a huge amount of water So you can imagine Everything flooded in one meter depth of water and that's the amount of water going into the atmosphere from most tropical forests every year huge amounts We know that forests are much more efficient at evaporating water into the atmosphere Maybe around ten times what you would get from grass your shrubby vegetation and even more than you would get from open water So if we talk about a lake versus a forest we're getting about twice as much From the forest per per unit area as we are from the open water Why is that well if you really look at a tree it's kind of a machine for evaporating water It's really Has this high leaf area that are all these things hanging out on the ends of the branches these leaves Biologists here at leaves on the end of the branches. You have these tall things sticking up to carry it in the air So why a tree so special well forest themselves have a lot of water in the soil They have water up soil that's very good at accumulating moisture these deep organic soils Trees also have incredibly deep roots in some cases, so they're able to access moisture deep in the soil profile Even when the soil is maybe very dry on top They may be able to access moisture deep on the ground to keep this transpiration going into the atmosphere You also find the trees I like most other plants have a big store a big reservoir of moisture in the trunk So it can keep transpiring even when the amount of water it's releasing is much higher than it can pull in through the roots You find that the tree will actually shrink through the day As it pushes the water out there's a store in there So it can keep transpiring water a long part of time that herbaceous can and then the stem is retained at night The average is it out and you also find that forests are a big source of aerosols That's a technical word for particles Any kind of particles in the atmosphere That can encourage water to condense now if you want to have clouds if you want to have water condensing out of the atmosphere It really helps to have impurities in the air sounds strange But that's what you need and forest are a really good source of these impurities often its biological particles appear For example an image of the Amazon Basin in the afternoon You see the cloud is starting to come here over the forest but not here over the ocean so much and a lot of This is to do with the chemistry of these particles There's a lot of research we've done Okay, I thought I talked you through a few just recent Articles that are not so controversial before I get to the more controversial stuff There's been a lot of work recently and talking about the amount of evaporation from land that returns to land So how much of the rainfall and land is really dependent on moisture that's evaporated from land surface from forest Land cover. There's a nice thesis on that too And the rent he says about that saying more than half of the evaporation that comes off land That she returns for this rain, so we're talking about really considerable There's also a nice study in nature just a year ago or so that showed that the amount of moisture They're coming off the land services really dependent on vegetation vegetation is responsible for more than 80% And that's much as 90% of all the moisture that's evaporating off the land surface This was kind of a stunning study at the time because people before that Much lower maybe around 20 to 25% now. We know that it's much much more So we're really dependent on vegetation for the amount of moisture that's evaporated off the land There's also these advice and technology in terms of how we can actually monitor this with satellites That can actually look at rainfall models that can actually tell us where air is coming from and telling us that really rain depends on tree cover meaning where the wind is blowing from How much forest cover the wind is coming through Effects very much the rainfall you get on a given day Where is the wind coming from how much forest does it come through if it's a forested area? You will generally get twice as much rainfall If as the wind it's come across non forest areas over the last 10 days my sticking to pick So we know that where the wind is coming from I think we have good data on this. This is not controversial So I put this picture in here just to remind myself I'm not just talking about really what super wet forest you get the same results hidden in deserts They're actually forest cover or plant cover It's really the primary source of evaporation even in the arid parts of the world We're not just talking about wet human reasons And that's because trees like this in the Kalahari Desert with roots down to almost 70 meters are able to access brown water They are a major source of evaporation even in these really arid parts of the world If we look at the Amazon basin, it's a nice example of a lot of people studying the Amazon basin We know how much water is coming out of the mountain of the Amazon every second We have one of the estimates here 209,000 cubic meters of water every second Huge amounts of water coming out of this region out of the river Amazon Now if you think about it over a long period of time What goes in sorry what comes out has to be going in there has to be a balance So we can think of it that all this water that's flowing out also has to be flowing in from somewhere somehow It has to be coming in so we're talking about huge amounts of moisture. So how does it get inland? This is this was my fun bit honey. She wanted a prize for a question So I read a paper recently where I could use article in science talking about Swedish researchers who like to quote Bob Dylan tracks in scientific papers So if anybody can name me the Bob Dylan track is the answer to this question you get a prize How does the most you get in land? Blow it in the wind So to actually understand how rain gets in land We have to understand how wind is actually carrying all this moisture So if we go back to how climate scientists look at this We have a fairly simple idea of how actually wind carries moisture in land Here's a cycle. We call it the sea breeze cycle is thermal explanation. This is pretty standard explanation Basically as land warms up you get the air over the land area here warming up The air rises. It's the same idea. It's hot air balloon, right? Spans it's lighter the lighter air rises as it rises to moisture near condenses You get rainfall here air has to circulate. This is a problem for the atmosphere You can't all air ending up somewhere has to circulate so it circulates back to where it's cooler and goes from the sea The cool place flowing in so you get this cycle as the air cools comes back warms Cool. Yeah carrying moisture off the sea In land. Yeah, so that's pretty much the standard idea and it's based on our communities principles So we're going way back to 250 BC. This is a fairly basic idea way back in time. I Didn't think of asking that as a question So we can also see this at a larger scale The early ideas of how I spread circulation works as best as the same idea We get an equator in the bottom of the poles. So here is where the air is warmer Here is where the air rises and here is where the air drops. You get a circulation going like this So most of the rainfall is here in the territorial zones the air dries As it rises it rains the air dries come back cool Yeah, you get a cycle like that and that's pretty much even though it goes back to Edmund Haley the guy with the comment by the way To just the end of 17th century obviously got a little bit more complicated people like had they came along people came along and added The circulation of the site the rotation of the earth and the formation of these cell systems It's a much more sophisticated idea, but it's basically the same idea the engine the energy Circulation basically it's hotter than the equator and colder towards the poles. It's just a thermal explanation of how this all operates Okay, so what do we know about climate from our models, so we have these really really sophisticated impressive models We can talk about forest loss in global climate models You know the kinds of ones that the IPCC are using how is it that they represent for us and they have quite sort of Nice physical ways to look at leaf area index declining Decline in routine depth as you go from parts to conditions for crops or whatever a change in mechanical roughness There's always different variables including their systems But there are really really big uncertainties if you look at these IPCC reports, you know They're impressive, but a lot of the documentation a lot of the discussion is about the uncertainties because the uncertainties are really big It's really hard to do an experiment on the earth. Yeah, so a lot of this we really don't know except that we're looking at physical principles If we look at these models and what they tell us about forest rainfall, of course, it's really analytical complex many unknowns There's lots of strange things going on For example, I just tell you about the thermal model, but that doesn't explain why the Amazon is often colder than the surrounding oceans I'm still getting huge amounts of wind going in If you look at models without actually making them fit and just ask what the basic physical processes will tell you You'll find that the amount of water in the Amazon should be about half what we see So that's the best fit models that you have We're actually not explaining the entire flow of the Amazon and current models. So there are acknowledged problems So this was a review last year in science I'm just trying to show you this to show that there is an acknowledgement that there are problems with the current views and models So what are climate models missing? This was a review in science by experts who were looking at the whole idea of these kind of models What is it that the modelers still needed to work on and they said rainfall ever land is largely determined by unresolved processes That's a pretty strong word unresolved processes and they really highlighted that this was true particularly in the tropics They couldn't really model rainfall in a realistic way This is the main limitation in current representations of the climate system So these are climate models talking about their models and they're acknowledging and have this huge problem This is a major roadblock to progress in climate science. So they're really telling you that this is a problem They really are acknowledging they don't know how rainfall works. Yeah, that's what how I read it anyway Okay This is a graph from a sorry figure from a paper. I did with Daniel a few years ago But the nice thing about this is just to point out Coming back to my initial question before us to track rain. We're about to get there If we look at the globe and we look at transects across forested areas or non forested areas Going inland from the coast with the direction of the wind So rainfall generally will be coming in from coastlines with the prevailing wind direction in land We find that if we look first at the orange sites here These are non forested areas you get pretty much an exponential decline And this is what you expect the rain falls in an area some of it recycles But a lot of it goes away on rivers and drains out to the sea eventually So as you go further and further in land, there's less water to recycle so it has to decline at some Way, these are best fit lines. Of course the real date is a bit noisy This is the best fit. So as you go in land here across the bottom kilometers in land up to a thousand meters Sorry, thousand kilometers, two thousand kilometers You get a pretty dramatic decline in rainfall as you go in land But if we look in areas where we have rainforest or a boreal forest up here We get a different pattern. We get more or less continuous rainfall Yeah This is an observation. It's just a fact But then it raises the question well which came first The forest or the rain, right? So as an ecologist, I kind of like these questions It's kind of fun, you know, so if I go to Daniel and say Daniel, do you understand this stuff? So then that's kind of how I was getting into this And I was reading the work of these two colleagues Physicists in Russia and it stays on the river and make the bush top and they've written some really interesting papers and Few years ago, I guess I was still at sea for I was reading a lot about different aspects of tropical rainforest And you know, you have different chapters on different topics. One of the chapters is on climate and weather and stuff So I just was reading a whole set of papers And I found some of their work and then I started to say well if what they're telling me is true This is really interesting. I want to write about this So I went to various colleagues who know about hydrology like Daniel and me helped me talk to other colleagues who know about hydrology to really find out Whether what they were saying is true because they're saying that yes forest do attract So this is pretty pretty different way. This is not about temperature anymore They'd come up with the way to explain how forest actually play a role in driving winds and driving rainfall pattern So my role in this would basically be the bewildered guy on the sidelines initially again I didn't know I didn't know it's true But I do want to try and understand if what they're saying is true And I should say he's a physicist a lot of this work has been published in really good physical journals This is an example of physical that is a they've been publishing we start the physical journals peer reviewed the physicist except it's true They've also published a little bit in some of the environmental journals This was the example is the first one. I was reading hydrology and earth science systems Trying to address environmental scientists, but these authors have had a real trouble getting this out to climate scientists And now I just want to tell you a little bit about the controversy kind of as it currently stands I've actually been working I'll jump a few steps here because we don't have all day to do this talk But I've actually been working with these physicists to try and make sure that their message reached the climate scientists for me It's not a question of that being certain that they're right But if it's a viable theory if it's a viable hypothesis then we should either be able to test it or say that it's wrong You know, it has to be one or the other it has to be something we can validly test Or there's a logical flaw in what they're actually presenting so it's really important in science that we present this to the people who feel best place to judge it the climate scientists and So I worked with them to actually produce this paper I've been collaborating them with them for the last six years to try and do this and in a sense It's helpful that I'm not a physicist and I'm not a climate scientist because I'm always trying to say What are you set trying to say? What does this mean a lot of it's about simplifying the message and making sure that we don't just write 17 pages of differential equations But actually give clear English language statements about what it is that we're saying the problem with working with Physicists is they think the differential equations are the argument, but for a lot of us as an audience we say, okay I don't know, you know, I can't I can't judge this But we do have to come up with testable ideas testable things that we can actually bring data to to actually test the science The science behind this so we have this paper in atmospheric chemistry and physics, which is quite a nice journal Generally, they have a pretty quick turnaround on most of their articles. It's a public online system You put up your draft paper It's public the comments are there in public and we had the debate going on and it was quite controversial It had a lot more comments than normal from outside Generally, it would take about three months to be approved This one was hanging for over two years But finally finally we had a lot of discussions Back with the fours of various people, but it was published with an editor's comment Which was a bit unusual and the editor said in contradiction to common textbook knowledge The handling editor and the executive committee are not convinced the paper is wrong Okay, so that tells you that they were pretty scared or they were pretty worried about publishing it, you know All scientific theories are Provisional, you know, they're not right in it in a in an ultimate sense, but this one is not They were not convinced it was wrong. So, okay, there's obviously something going on here So what I want to do is just kind of fill you in a little bit more on the background Okay, so it's a very basic idea early I talked about the thermal explanation for what's going on in the atmosphere and If you did these sort of ideal gas gas law at physics at school, you'll see the pressure is Related to temperature. This is the basic ideas temperature goes up The pressure goes up and the and the air will rise because the volume expands This is how we understand it normally But there's also another one in here in the ideal gas law, which is the number of particles the number of molecules in the gas Now what happens when you get condensation you go from having more particles in the air to having fewer basically It's a very simple idea, right? So you also get a change in pressure as If the moisture condenses now there is temperature release there is energy released in this process And this is what most of the debate is about and I'm not really qualified to tell you what's right and wrong But I can say that there's no clarity about what really happens So I'm going to skip over the physics a bit I have articles I can share with you if you're really interested in the physics But I think what's really interesting for this audience is what it really means And if we look at the theory itself, what does it tell us? It tells us the areas with really high levels of evaporation develop the lowest atmospheric pressures So that means that those are the areas that draw winds into them that draw moist winds into them draw air in And there's also a positive feedback because if you draw in moist moist air This is also providing moisture for rain so you can provide more moisture in that point It provides a positive feedback to actually keep keep the system running So this is really important in oceans with big tropical storms and the likes but that's not so interesting for this audience And on land one of the most powerful sources of evaporation. I told you earlier its forests So it's really telling us that forests are playing this really important role in sucking air in So just to try and talk you through a little bit what we think is going on with this new theory This is not the thermal idea. This is the idea that comes from the Russians Anastasia and Victor and the rest of us so we've published this this year What we think is going on if you look at this top one up here you see The ocean natural forest maybe I should use this The ocean and the forest higher evaporation over the forest the pink here is dry air So we're starting in a position here where this is a landscape that has dry air everywhere, right now the forests Evaporate air more quickly So we go to a situation where this bluer air here is now more moisture in the air So here maybe clouds start to form Because you're getting more moisture forming here. There's no wind in this yet. There's no wind We're just talking about naturally on its own This is generating more humidity And of course after a certain point we get to this side starts to rain and as soon as it starts to rain We get condensation happening We get a reduction in air particles particles in the air the moisture is condensing and then that leads to low Pressure here and this draws in moist wind off the oceans. So we're actually carrying air in land Yeah, this is the basic idea and we've actually looked at this as a basic idea So this could all happen in one day. So you could imagine this is the morning No winds no winds and then in the afternoon you get your big thunderstorms, right? This is what happens And if we look at this in the Amazon Basin we look at wind directions and we look at pressure changes Etc. Etc. I don't have time to go into all the technical details, but this pattern more or less fits So just to go back to what does this all mean? With this idea where you get the most evaporation you get the wind being drawn towards there You get the lower pressure so here for example on a desert or maybe an open landscape Versus the ocean the evaporation here will be lower than open water So the air will be drawn this way because condensation will happen here more regularly You'll get rainfall here more regularly the air will move this way drawing the moisture off the land because any evaporation coming off The land here will be drawn towards the ocean. Yeah If we have enough forest then the evaporation over the forest is greater than the evaporation over the open water Because the forest has a more efficient evaporation per unit area So the pressure here will drop because we get more regular Rainfall and condensation occurring here and that will draw air in And if you look at the physics and I won't go into it But it also allows us to look like at the Amazon basin and say yes This is a sufficient process We look at the physics to actually draw the winds right in so we can actually water the whole Amazon basin with this mechanism I'm trying to see if people are still awake. Yeah, good This is kind of a fun a figure it's a bit of a messy data But of course we can't cut down whole rainforest or whatever to test an idea like this What one point would be if you cut down the entire Amazon rainforest and make it into a parking lot You would expect to see a very rapid decline in rainfall as you go inland That will be one way to test it but quite difficult ethically to do that So what we can do is look for parts of the world where the forest switches on and off Which is going to the boreal systems So during the boreal the winter time the forest up in the boreal to northern latitudes Get all snowy the trees more or less shut down. There is no moisture Traveling through those trees the ground is frozen. There is no transpiration going on switched off You've turned a knob going from transpiration to off. Yeah, so good from having a much higher transpiration than the surrounding Ocean you've switched it off and what we see here. This is a transect looking bottom graph here is distance in land This is the zero. This is in the Siberia as you go inland Obviously, there's terrain effects. This affects the data. So there's little Changes in the landscape that have an effect on rain, too, but as you go inland During the summertime you see not much change. Maybe a little bit decline. It's a bit noisy But in winter time we see well, I think it's reasonably convincing But anyway, we can do this many times in different places But basically the pattern fits what the theory would say the overall when the forest is switched on you get much Better transport in land when the forest is switched off you get much more rapid decline in rainfall Versus the ocean. Yeah, so I don't really have time to go into everything on this. So just point out that This this theory has certain evidence to support it I think this idea that rainfall declines into interiors when there's no forest but not when there is forest It actually resolves this problem. I was telling you earlier about the cold Amazon paradox and the idea that runoff is Is I think I've lost something there the idea that runoff in models is too low Because we actually have a different mechanism that explains why a cold Amazon still draws moist air in produces high rainfall It explains why we get a decline in the effect over the boreal forest in winter time And there's a whole list of other things including sort of power estimates of global circulation stuff If you're really interested in this stuff, we have a literature on it. I'm seeing people are still awake. Yes, good Okay, so Probably if you're going back to my original question when I started coming to Daniel all those years ago about six years ago now saying Daniel Do you believe this and going around different people who actually know about this stuff whose opinions I respected Does this make sense? Is it because if it does make sense then it's really important So of course I got various different responses from people and I've had a lot of email communication with climate scientists and the stuff And one of the main responses initially was dismissal. No, no, no, no We've been studying this for hundreds of years. You know, we have tons of data. We have incredibly complex models. It can't be right so that was one response and Of course, that's a bit frustrating, but in the sense. It's also exciting if you think well Maybe maybe this isn't right, you know, maybe the dismissal is you know, then you say to them Have you looked at it? Do you can you say where it's wrong and people are often just saying well, I'm busy Yeah, so it's hard to get a good reason, but a lot of people were dismissive a lot of people were dismissive Other people said yeah, yeah, we've known about this for a long time. We've known about this for a long time It's a tiny effect. It's it's not it's not important and some of them even said it's in the models already At this point, I'm starting to get a little bit curious because I think well, they can't both be right Some people I talked to were kind of cautiously interested and I think that was a Kind of made sense, you know people who are saying well the physics looks to be right. It's in these physics journals I can't see a mistake Yeah, it's interesting. It seems reasonable response, but there were people in that category Even a small number accepted that it's true. I think particularly in the environmental kind of groups It's actually quite exciting to have this whole new Potential for arguing for forests that it actually really makes this huge difference to regional rainfall So I think particularly amongst environmental people they want it to be true You know, this is an amazing story if it's true forests are really generating this huge amount of rainfall are really necessary for maintaining rainfall a lot of discussion right now is is People who've accepted that it's true, but are arguing about the magnitude. How important is it? It's not that the thermal argument is wrong. The temperature argument is not necessarily wrong but How do we balance this if we don't have models that take it into account? It's very hard so far to really talk about how significant this this forest effect might be So I think that's where we are right now is on the magnitude So just to kind of come back to why it's interesting hopefully for you One aspect of this theory that's really important is to highlight that we need for us to Maintain rainfall in continental interiors So if we're like living right in the heart of the Amazon basin and people start Deforestation all around the Atlantic coast You could be in trouble with your rainfall your rainfall is likely to decline and it's even possible according to the equations We're just talking equations here. It's even possible to go from a situation with lush rainforest to more or less desert Just by switching because there's a positive feedback involved We can go from a system that sustains itself with the positive feedback to a system where we switched it off I think that the positive feedback has lost We have no rain there anymore So this technically is possible So that's a pretty big risk. I Think for this audience. There's also real opportunities If you look at the equations it tells us that if we actually could plan to forest across the Amazon across the Sahara that they would actually be enough Power drawing in wind to actually to actually water those forests. You wouldn't need irrigation. This is this is what the equations say I'm not saying it's necessarily true But it's an intriguing idea or at least you wouldn't need irrigation everywhere all the time As long as you think about the right places to grow the right weather systems to work with We could potentially use a system like this to green the deserts. Of course, we would have to do it on a big scale We're talking about climate systems. So the magnitudes there are sort of hundreds of kilometers across We can't just do a few hectares and expect it to water itself We're talking about expanses of hundreds of kilometers Earlier I was saying it's a whole new value for forests Of course if this is true Then it means that people who live in the middle of continents have to start talking to people who live on the edge of Components are going back to where I was in Africa in Uganda looking out the window at mountain gorillas and rainforests If the people in Kenya or the people in Congo lose all their forests Then my forest up in the mountains in Uganda may also dry out So it provides a whole new regional value that we have to start thinking about yeah potentially And of course a lot of what I'm saying is hand-waving. I mean, I think it's really interesting It's really profoundly important, but we don't really know that much about it We don't know that much about the ecology what the implications are for the management of forest When is it that you would lose these properties? How do you maintain them huge policy implications potentially and this idea that you are dependent on Forest in one place for rainfall in another So I think there's yeah a lot of opportunities there. I was going to stop now because I didn't want to go on too long But you're welcome to talk to me more if you're interested We have a lot of publications or I have a lot of publications my my own but also my colleagues or physicists If you like the technical stuff, I can share those too. I Do have a few more slides at the end if you're interested on on the arguments that people put forward But I think I've already yeah enough people are Paying attention. Yeah, okay, we'll see I'm ready for questions. Thank you


Good morning. Good afternoon. Good evening and welcome to the level up hour here on Red Hat live streaming I am Chris short host and showrunner of Red Hat live streaming as You can see the usual suspect Langdon is missing We wish Langdon the best of luck. He has taken on his Professorship duties at Boston University starting this week and if you weren't here last week yet Langdon has no longer at Red Hat so Just to let you know.  So me calling him illustrious and a professor on the show Might have done him some good. I don't know. Maybe I'll take credit for it though So today I'm joined by the new usual suspects The one and only Randy Russell Jafar Chorabi and then we have special guest Eric Nelson here Let's just go around the horn real quick. Let's do some intros, you know, Randy I'll toss it over to you as the quote new host of the show What? Yeah, so yeah, hello everybody I'm Randy Russell. I'm the director of certification at Red Hat and really excited to be joining the level up hour and Having these discussions with you approximately every week or ish. Yeah, you know when we're not dark for various reasons Jafar Hey, so my name is Jafar. I work as a tech market market manager for OpenShift and I'm really glad to be here. I'm trying to grow my verb So it looks like a bit like Langdon, but it's gonna take a while. So yeah, so hopefully in 40 episodes It might be might be Eric's almost there I Will say Scott McBrian is also gonna be on the show regularly, but he is off this week So he's on PTO. We're gonna talk about conveyor today and specifically Where's mine, where's yours? I'm sorry. Oh, I can definitely get you on. Oh, there you go They are six dollars, I think on the cool stuff store for what it's worth So yeah, Eric you're here Hi, hello, how are you? Welcome back to the channel. Thanks for having me. I am Eric Nelson I'm a manager in an engineering manager in migration engineering. So I work on a project called crane, which is the topic of this conversation Downstream it's known as migration toolkit for containers. I've been an engineer at Red Hat for Six don't know five years at this point and prior to that I'm a computer engineer and actually I went to school at Boston University. So I know that's a pretty good program There you go.  Yeah, I should have done that Yeah, you could have been taught by the illustrious one But oh well go back for your masters or something. Yeah, I Yeah, I'm in Raleigh now. So but I do miss Boston quite a bit. It's gonna be I based migration There you go. Exactly So just let folks know we will still at least for the very short term have our usual points and everything else I like and that's sweet all that off to me. The sweet sweet internet points are here As far as their value that is up in the air Still sorry, I haven't heard anything otherwise But I know that something is coming so we'll keep the points going for now So we're talking about Crane migrating container workloads with crane Where so I already got a question fielded for you, Eric You but let's let's go ahead and talk about what crane as first before we dive into the question that I have Sure So crane is Should I show that the slide that kind of gives an overview Maybe before we even dive into crane, let's talk about conveyor conveyor. Yes. Thank you, Randy I know can everybody see my this this diagram here. Yeah, okay cool. So Conveyor is I think there have been some other projects from the conveyor org that have have been here So folks may have heard about it, but I'll just quickly give an overview. So conveyor is kind of an umbrella like collective of projects that All sort of have a shared objective to just move people to cloud native environments specifically kubernetes And and so that that can be like there's a lot of paths there There's a lot of problems that are involved. So there's a lot of projects that seek to solve a bunch of different problems So you can see here. There's re-hosting. So maybe you've got like one cluster in one place. You want to move it around? crane concerns itself with container workloads So like the traditional kubernetes migrations, maybe you want to go from one cluster to another and it's already been containerized Then there's a project called forklift which concerns itself with actually picking up vms and moving them to kubernetes clusters Which is pretty cool uses cube dirt So it's kind of forklift and crane or kind of sister projects in that respect Then you've got replatforms. So move to cube expects a source. That's not a kubernetes cluster So other platforms such as like docker swarm or cloud foundry This will pick stuff up kind of analyze it spit out kubernetes objects And then that'll help you get like to sort of convert your workloads to kubernetes Tackle I think was the one that was actually on here if I'm correct But this will help you actually refactor your applications I think we'll probably talk a little bit about this, but All applications are not just created cloud native out of the box. There's a lot that actually goes into architecting and designing and writing like Software so that it expects to be running inside of a cloud environment and that it can take advantage of all the benefits That are associated with that. So tackle helps you helps you actually look at your application and make some recommendations about how you can manipulate that And so that's turning into migration toolkit for applications Which is our supported downstream offering You'll hear me say like MTC or MTV or MTA. All of those are like our downstream references And then we've got our upstream projects and we're trying to move towards this like upstream first model Let's go easy on the acronyms Did the move to cube people not get the memo right on the new conventions because I love crane and forklift and tackle because they are in fact very descriptive of what these things do move to cube also descriptive, but that's not from the moving industry It's not for the building industry. Come on. Yeah, it's a well We know that like in software the hardest thing is like actually naming things. So yeah But actually move to cube is a is an IBM project. So it started with IBM So that's actually the background there and why it's not it doesn't it hasn't adopted one of those types of names Fair enough. And then finally there is Polaris So this is like sort of metrics like measuring the changes and impact of like how you've delivered things It's almost in order of like maturity Crane's been around since 2019 So I guess I can jump into crane any questions about like conveyor in general or what we're doing here Well, let me just throw a really high-level question out of there on this and so this project got kicked off really earlier this year, right? Crane or conveyor? I think it's been around Maybe let's say a year and a half Crane crane did predate conveyor as a as an organization But it's true that it it's been around for let's say under two years Yeah, so we have some of these independent projects that have sort of come under the the conveyor umbrella Is all being related in the sense of being able to Manage this big picture right in different ways whether it's replatforming Rehosting refactoring. It's there's there's a tool for everything, right? So here's the thing I kind of wonder as just sort of first high-level question is You know, we have this need right now to have this sort of tooling Do we anticipate that this is the sort of that that the conveyor project and that these Things under that conveyor umbrella will continue for for years to come because this is going to be an Ungoing need because I think a Polaris is sort of the end of the cycle But it takes you right back to the beginning of the cycle Yeah, for sure There is an expectation that this is going to live on And and it's not just migrations like migrations are sort of like a point-in-time event Once you're there you sort of but You know, we've learned that that's not really true Like like you pointed out Polaris is something that you're going to use to For day two operations monitoring things upgrades all that sort of stuff and then crane The genesis of like crane was to move people from open shift three to four because we didn't have a migration path there, but it quickly became apparent that I can get into this but you know In order to take advantage of hybrid cloud right like on-premises and like cloud environments where you have burstable resources and stuff like that Your applications have to be architected to be portable And and and not only that like there's a whole environment around them where they need to live Like they need to be decoupled from the environments that they live inside of And exist in kind of a golden source location. You could call this a get look get location a get repository that deploys them out so In an environment agnostic way so you can like point it at a cluster and kind of deploy it there So in that respect, it's an ongoing problem And and especially when you consider hybrid cloud where things are moving around You need things to be fluid and so a lot of these these projects seek to help you do that And maybe your source code 100 and portable apps is an ongoing concern So it's not it's not a single point in time migration So a lot of these projects underneath conveyor are seeking to help that and in addition to that conveyor is kind of a living breathing thing It's not just a set of Repositories. We also are running like meetups. There's conversations going on on mailing lists. So It's a community really is a more accurate way to describe that and And again, like that's anybody's interested in this is is this is a great place to kind of connect with other people Who are also interested in the problem and work on tools that will help you achieve that Great awesome So, uh, do we want to talk a little bit drill into The wonders of crane Sure, I'll I'll send my questions for a bit or when we have A bit of understanding of what what it does Okay, I already have a few questions So, um, I'll go back a little bit. Um The I mentioned that like crane Crane is the upstream name at the time. It was really only mtc, which is migration toolkit for containers Um, I'm just going to call it crane We'll call it crane v1 because there is a crane v2 that's on the horizon Um, and that's kind of the interesting part of like what I was hoping to discuss here um crane v1 Concerns itself with like mask cluster evacuations. So This is something that a cluster administrator and that's like an important point It is only available to cluster administrators To pick up everything that's in a in a cluster and move it over to a second clusters This is a total like a whole whole migration It does not concern itself with the control plane. So, um, it operates at the application workload layer um, and primarily, um Usually when you think about a kubernetes migration, uh, it's kind of three buckets You've got the kubernetes resources that make up the application. So you're like pods deployments Stuff like that your images that make up that back the containers So often these are images that are inside living inside of like a registry that you've either got inside of the cluster somewhere else maybe and then finally there's the The state which is kind of the the hard part Exactly that lives inside of the pvs. So crane Picks all that up and it moves it over in a specific manner. Um, I can get into like the architecture of how it's architected but Yeah, that's the objective and it will do it at like a sort of a mass scale. So We've had I mentioned in kind of the show description We've done like large scale migrations and I would call those like we're talking multiple clusters and many geos with thousands of namespaces crane operates at a namespace level. So that's the like unit of It's the atomic unit that we use in order to migrate things Often people will organize their applications it within namespaces on a namespace basis That's not always the case. So you could have like microservices that have the interdependencies spread across namespaces In which case you're you're going to want to evaluate that and make make sure that you bring them all over together as part of a plan But at a high level, um, it's effectively a it's an operator backed application that Is controller based and its api is composed of crd So it's using the kubernetes extensions as its core api So you can communicate with it as you would any other cube object And so it uses that sort of declarative reconciliation pattern that kubernetes is famous for To execute these migrations across clusters Um Nice. Yeah, so we already have a question in chat, which was the question I got before the show even started Go ahead. It's a far You sure? Okay. Yeah And jw matthew's just answered it in chat, uh, but air gap environments talk to me about that Can we use crane? Yes in okay, like Tell me more because yeah, we have a it's a it's a Primary concern for a lot of customers. Um, of course, they don't have these things just living out on the internet They're pretty important and they live within Disconnected environments So there there are a couple levels of disconnected that you can kind of think of air gap would be one which is like This is a hundred percent a hundred percent not accessible which is possible and then separately you've got Pseudo disconnected which is sort of it has limited external connectivity Maybe it can see the outside world through a proxy or something to that effect. Both of those are supported um And we've had a lot of people use mtc in that manner. So uh The way that that works is with with open shift four, um their open shift four uses olm which is kind of its Packet the operator repository, right and there are mechanisms that allow you to mirror the images into The environments so that you can actually deploy mtc In order to actually perform the migration we we of course do have the requirement that both of those clusters can see one another Or at a bare minimum Well, there's also this is getting into kind of how mtc is architected Underneath the covers mtc uses a project called valero Some folks may have heard about that. I've heard of it. Yeah It is a backup and recovery solution for kubernetes The world's kind of coalescing on it as like the the backup and recovery solution dr solution for native kubernetes So we're built on top of that So actually mtc what it is is a controller that orchestrates a migration using a series of backups and recoveries Communicating with valero, which kind of executes underneath the covers. This has changed over time, but that's kind of the crux of it For that reason we actually require an intermediary repository where We back things up to and then we recover things out of so We we recommend for a disconnected and air-gapped scenario to actually deploy you can run like ocs which is our container storage solution on An open shift forecluster and there you can set up your s3 repository that's self hosted so it's all on your let's call it your target cluster and You'd be able to use that as your intermediary repository So the controller would basically move all your data out of the source into that target repository And then it would recover from that repository into the target cluster Trying to think what else is relevant as far as disconnected Is the operator and the certified index or their operator index? I'm pulling up. Okay. That's what I figured Yep mtc. You'll find migration toolkit for containers there And then there's also a community operator called crane. Yep. Awesome So I do have a question Which is related to what so I kind of like the way you dissected the main Areas of interest when you are doing this kind of migrations You spoke about like the resources The communities resources which are basically like the the whole the animal fires you have to export in order to be able to push them somewhere else You have the data and then you have the images And so I know that we started this project a while ago like two years ago when we started addressing the migration from three to four And at that point in time get ups was not yet like a very hot topic So the whole thing was I had customers asking me while I was still, you know doing pre-sale stuff like so what's the proper way to export stuff like my resources and get rid of all the cluster specific details like creation date ids and stuff like that in order to be you know to to make them generic and then push them somewhere else um, so that that was sort of a manual process and I think that uh mtc At that point in time already was able to solve that kind of questions by introspecting the cluster and being able to To to migrate items without necessarily having the definitions at hand but uh, so my question now is now that we are also playing in the gilabs field where we have this this notion of now that I have my resources descriptors like all my animal finds stored somewhere I can push them across multiple clusters and I can also Keep that consistency like if I update something from the source cluster, then I can automatically Maybe you know update it And mirror it back to the remaining clusters So is it something that uh mtc is Is is working with is it something that you know, so how does it like because the way I see it is You have the operator that does this thing it introspects stuff and it moves resources somewhere else but Does it make sense to to sort of plug it a gilabs? I would say engine somewhere inside mtc to have that you know that Reconciliation loop Uh capability and all of those, you know new stuff that came up Uh recently I would say Yeah 100 um, I'm happy that you're kind of teeing up perfectly what crane 2 seeks to do. Um, okay, so um mtc one And I didn't know that yeah Yeah, that was uh, yeah I mean, it's it's interesting because you're pointing out like a real pain right like this is what people want to be doing um, and so we've spent a lot of time talking with Services folks and people in the field and customers trying to figure out like What are the things that you want to be doing and how is mtc was a little bit early? Believe it or not like it sort of missed uh like There were still a lot of like people sort of figuring this space out a little bit when we did a lot of the initial design But we still believe that there is space for mtc Um and crane 2 and I can get into crane 2's objectives Um, they're they're kind of they're they're targeting two different problem spaces the first one with mtc This is sort of its pro like some of its faults are sort of also core to its design It's a tool that does a like a mass cluster Migration so and it's it's a tool for cluster administrators So they can pick up a whole cluster pick up all the app workloads and move them over to a target cluster the um We what we found while uh interface like while seeing this in actually in production migrations Is that oftentimes cluster administrators have no idea what's actually inside of their namespaces? So imagine there's a large university right that has um all of their students running their word press blogs inside of namespaces Some of them are crashing like the whole cluster is kind of a disaster They don't really know what's in what's inside of those. They don't know if the like a crash loop pod is a problem um They don't know that like the interdependencies between namespaces the application owners do And the developers do there are the people who are deploying into these environments. So That was the first thing is that the fact that mtc is a cluster admin required tool in addition to the fact that The admins kind of don't know what's inside of those namespaces Means that there's kind of a gap that needs to be filled and that's where crane 2 comes in So the app owners are really the people who understand their applications and their interdependencies and everything about that Let's like shift that paradigm a little bit and have the developer the app owners themselves actually migrate their own applications So that distributes the load of and the burden of a full cluster evacuation It also allows you to do it in piece mail Which mtc can do depending on how you how you slice it up but um So that's one of the the first objectives of crane 2 is to actually allow for app owners themselves to migrate their applications the second thing is You're right like you kind of nailed a lot of the points that crane 2 is is thinking about There's I like to kind of describe hybrid cloud like crane 2 allows you to unlock hybrid cloud because There are a lot of these applications that are like snowflake applications in that they're very tightly coupled to the environments that they live in They've probably had like kind of duct tape like changes made to their resources over time that are specific to the environment We know of a lot of examples of that. Let's call them node selectors resource quotas like stuff that you pointed out So in order to actually access hybrid cloud the first you need to a portable application And what that means is you need to strip out a lot of that Like cluster specific information And then you need to version it you need to put it into get ops model It actually could be like kind of any real version the point is to version it in an in a cloud in a cluster agnostic way and then So really and then continuously deliver it to your various clusters And maybe you want to actually layer on some more cluster specifics as you do that So that's starting to sound kind of like a pipeline, right? And so what we're interested in is with crane 2 is is actually building that pipeline where you have that discovery aspect Where even app owners may not know what's inside of the namespaces So it goes out it exports all of the stuff that's inside of the namespace And then it applies some transforms to those objects that are by the way commonly Like a lot of people are going to have common interests Let's say they want to strip the status which is not actually relevant for a target cluster for example And also it's not it's not a live application. So it doesn't necessarily have status at that point in time So you want to strip that stuff out? Maybe you want to like there's another example with routers Like maybe you want to strip out the the host that it lives behind because of course that's cluster specific um, and then from there Once you've applied these transforms and we have a plugin model around this that's I think is pretty exciting that allows like services folks with shared problems to Build their own plugins that are reusable you can pull them off the shelf and apply them and then um So you'll get this kind of like stripped down Portable representation of your workload that can be version. So you check that into a git ops a git repository You flip on your git ops and then you can use this pipeline to deploy it out into whatever clusters that you're interested in deploying it in so Crane 2 is designed for that and in a lot of ways like it's a migration tool And it allows you to migrate from an application to cluster to cluster But really while you're doing that migration It's a great opportunity to modernize it too and and to to unlock that hybrid cloud by adopting that git ops model And so we want to help people get to there as well There's a lot of unsolved problems there state is one that's tricky And there's a lot of conversation that's happening that we're involved in in some of the upstream kind of like SIG groups data protection stuff like that where How do you like fluidly replicate your state across a cluster? How do you deal with the fact that these applications are live? So maybe you need to quiesce the applications move them into a read-only mode so that you're not corrupting your data while you're moving it around Stuff like that um are tricky tricky problems to solve but um, that's that's really what crane 2 is seeking to solve is is and how it differentiates itself from mtc whereas mtc will continue to live on as this like mass cluster evacuation tool and crane 2 seeks to be kind of that um tool that's designed for app owners to export transform and Modernize kind of their applications into a git ops flow so that they can be then redeployed in a cluster agnostic way Beautiful So yeah, thanks a lot for this like very comprehensive answer and i'm i'm happy to see that uh, you know it's sort of it's going to solve some of the questions that i saw within the field so That's that's cool to see that we are You know addressing those real problems So I I have another question if nobody wants to step in We have some questions in chat but go ahead Okay, so you mentioned one key aspect I believe is the understanding of the application and its ecosystem like its Uh dependencies and when we mean dependencies, it's uh, it can be things like The data it's relying on like databases So you can have the pod running in certain namespace But if you don't migrate a database that runs in another namespace or another microservice Or something like that the application is not going to be fully I would say Working so So I was doing uh for for a few years some enterprise architecture consulting Consentancy and and one of the key things that we did before, you know when we wanted to address these types of migration was Assessing the application and like properly Having an application topology where we Model the application and its dependencies like in enterprise modeling solutions So without going into that much of advanced stuff Do you guys think uh I haven't seen MTC recently, but is there a way to sort of Gather the dependencies of a specific application. So you said for instance The admin doesn't know the application. It doesn't know to what services it needs to talk et cetera, so I believe it would be very helpful When you look at an application definition within the those tools to to have some something that says Here's the application and here are all the services that it relies on like It can really point to some specific namespaces services, et cetera Do you have something like that that like gathers objects from the existing Cluster to to to give an overview or a topology of the application and its dependencies Or even if we can you know just describe it by text or something like that Is it something that you have that you plan to have? What's what's the the take on that? And it's definitely important. Um, and MTC doesn't do that. Um, and neither does crane And and it's kind of a gap in kubernetes in general kubernetes doesn't have a high level abstraction that kind of encapsulates like this is my application and all of it's interdependencies, which is A gap and I know that there are conversations. There's a lot of conversations that are going on around that In the six gaps. I think they're trying to address that hundred percent And that's what we would build our migration based on instead of like a the unit of migration and like the closest we can get to that It's kind of a namespace Out of the box the problem. Yeah, everybody kind of represents them in different ways Yeah, because sometimes you can have several components living in different namespaces for the same application 100 and and that's where you know, it becomes harder to get the whole application Which relies in like 10 namespaces or even five namespaces or whatever Yeah, our recommendation is um, we we kind of expect you expect that legwork to have been done Um, it is a problem that I think there's there's opportunity to solve and make that easier for sure I think that I've I've heard some like rumors of other tools that may seek to do that um, but it seems it seems like it would be an opportunity and a perfect problem to kind of be explored underneath the conveyor umbrella. Um, We also have mechanisms for actually onboarding new projects stuff like that. Um, but yeah MTC and crane don't concern themselves with that sort of analysis Cool. All right Good to know. All right, randy. Do you want to tackle these questions and chat? Do you want me to handle it? Please be my guest. Okay um What if I want to migrate my ocp cluster from one data center to another Just because I heard the word for larrow is just covered by these conveyor tools Yes, um, so if you have an ocp cluster in one data center, you've got another one. Um, That's that's like the bread and butter of what mtc does. Um, so you you Just to quickly describe kind of the the data model and how you would Do that You install, uh You have to install mtc on one of your clusters and actually it doesn't have to be the source or the destination It could just be a third cluster. In fact, like I've seen people run it on their laptop and like a One of the developer clusters um connect out But um, so you have to you install it somewhere and then we've got this, uh, crd data model that represents You know everything that we need. Uh, that's our api and so you can communicate that with um, so we have a ui That can be used and that's super helpful. It's gotten uh, very very useful. Um as we've moved forwards um, it's particularly useful when things go wrong and um as we've learned with migrations These almost never go smoothly. So it's very helpful. Um to have um And uh, so you register your uh, the clusters that are involved in your migrations So you'd register the two clusters, uh, you you give us like the coordinates to the cluster as well as your credentials So we can access it Then you're going to go set up your replication repository, which is this intermediary, um storage where we're going to export and import our data, uh to and from between the two clusters and then you're going to um draft a plan and what that means is, um You're going to pick the namespaces that in your source cluster You're going to pick the pvs and how you want to handle them Maybe you want to change your storage classes. You're converting from like cluster to saff or something like that and then, um There's there's a few more details, but at its core you kind of the mig plan represents This is what I want to move. Um, and and there's a discovery process that helps you Figure out what's available to you to move Next up, uh, and the way that we've we've built mtc. This is getting into A lot of people want to reduce your downtime as much as possible, of course, so Some of these applications can be quite heavy. We've done terabyte volumes before so What you're going to want to do is stage as much of that data as possible This is assuming a stateful migration, of course, stateless is a lot easier where you don't have to deal with this Um, so you can you can leave your source application up and running You'll stage as much of your data as possible onto the target without taking your application down So let's say you get like 99% of your data onto the target Um, and then you do your final cutover migration, which uh queueses the source application By default you can actually switch that off Uh, and then we'll grab that final delta of data that's been written between the last time that you staged your data and Currently, so we'll get that last piece. We'll move it over and by staging all of that data The time that it takes for your final cutover and your downtime should only happen when during the time that we need to capture that final bit Um I think that's it. Yeah, that's mtc in a nutshell So that's going to help that's how you can migrate from one data center to another Of course, there are these like caveats where both of them actually need to see each other They need to have access to your application repository. Um, but that's that's how that works So I do have another question go for you for yeah related to what you just talked about so Um, when we're speaking about the state state of the applications As you mentioned one of the problems is to to get that live data So it's easy to to migrate static data that doesn't you know change But uh, if you want to get the last updates to to the data while you are already, you know Planning or doing the migration It gets a little trickier and one of the things that happened in the in that space in the data Migration or synchronization is what we call cdc like change data capture Uh, where we have tools that are able to plug onto a source database Where you say so here's my e e-commerce database and here's my my target database and It does actually migrate and copy the data But as as soon as you have a change that occurs in the source data, it also replicates it into the destination Database and so I know that red hand has an extreme project about that called divisium Are there any plans to sort of you know try to Plug in to tools like that that could help address that issue of you know capturing data, but also Keeping that you know update easier or any more I would say industrial way With with with pain will it make sense to have something like that back been and How synchronize the data Actually, I haven't heard of that project. I'd be interested to look at it a little bit. Um We we don't have uh, we don't there's no plans on the roadmap currently to integrate with a tool that helps you do that Um, I actually don't feel but we haven't had a ton of interest in or problems that have been raised around actively like active applications that are writing during a migration or something like that. However, we do have um Concepts that will help you do that. Um, we have something called a hook uh that um So our our migration is broken down into a 19 area of steps And in reality, it's it's a state machine behind the scenes. So what you can actually do is write arbitrary logic into hooks during the migration So let's say like pre quiesce of the application. You'd be able to so we'll delegate out to the user It's hard to write like a fully generic uh migration tool that's going to satisfy everybody's needs Um, so as part of that we want to develop a hook mechanism so that you can hook into that process and do some custom work I've seen people use it for things like dns like changes during migration at a specific time That that might be an opportunity to use it. So maybe you want to hook in and do some intelligent Operations with like your you can take advantage of some data apis that you that are available to you to switch things on So that you can replicate it across clusters stuff like that So you can hook in and the way that it's implemented actually is out of the box It uses ansible, but actually it's just a we run an image And we'll provide it kind of a there's a there's like a Yeah, there's a there's a contract so you can rely on like you'll get some specific information about the migration that's going on And it's just using an entry point. So you it doesn't actually have to be ansible So you can provide us a playbook. It'll we can do ansible or um, you can provide us a uh Just an entry point and then it can be anything that you want so you can execute some arbitrary logic So that means maybe you're communicating with external systems or something like that to trigger like some type of replication during the migration There isn't that like ongoing data sync post migration But there's a lot of investigation that's going on we have folks in some of the like data the SIG groups that are looking at data movers and Volume replicators stuff like that that are that are lower level kubernetes concepts that I think seek to To achieve similar ideas where you're sort of like propagating data across and changes across multiple clusters in pvs and There there's also like I think some of the storage providers will actually provide that as well. I think like Um, we have a question. It's actually outside of yeah outside of kubernetes in general. It'll it's able to um Propagate data across so that like to cluster like storage leaders Storage layers will be synced and look similar So speaking of like all right storage layers and things like that If we're using something like open data foundations formerly known as open container storage or open shift container storage um How do we migrate how does crane migrate those pvcs? Is this still using velero at that point or is there something that we're leaning on in the ods phase? um We have a few different mechanisms There's i'm picturing a slide that we have that's floating around that's kind of like a pyramid There are a bunch of options that are available to you and like you're going to choose one depending on What makes the most sense for like the storage that you're using so There's two mech in two ways that you can actually move your data I'm struggling. I don't want to call it move because that's one of the ways is a move versus a copy so I think something like 70 of of people are using actually just nfs um So with nfs that's interesting because like if you've got a ton of data like a terabyte volume Copying that data is going to take a very long time obviously. So uh One of the better approaches there is like, oh, I've got shared storage and both of my clusters can see the shared storage Why don't we just pick up one of those volumes and plug it in into the destination? So that's called a move and that's going to be your fastest way to Get your data and make it accessible in the target if you're using something like nfs Of course, there are caveats there where that is the data. That's the golden source. So you're not cloning it Which means you need to be careful And there are there are implications for rollback and stuff like that when you're actually picking up your your data Of course backup all like healthy backup practices apply here. Um, so And then separately there's a layer below that where Depending on your storage provider. Let's say you're using amazon Uh, they're I'm blanking on the name of their storage. Yeah, ebs Whatever exactly they have a few so most of them have snapshot apis csi is one of the formats that Everybody's talking about and crane 2 is interested in so we can actually create a snapshot and recover from a snapshot on target side So that that operates at a different level than the file system And then one layer down from that is actually like a file system level copy So we had a couple ways of doing that um in the initial versions of of mtc we had We used a project called rustic indirectly, which is what valero actually uses. Yeah, so Um, it would copy data into the replication repository and out of it. Um, I understand that it was similar to an rsync mechanism I think it had, um Incremental backups Functionality we had some limitations with that we ran into some limitations And so we actually optimized that with some of our recent mtc releases where we offer a direct data migration So now we offer direct volume migration and direct image migration Which means you can skip that intermediary replication repository and we'll actually rsync the data using like an s tunnel mechanism Directly into your target cluster. So in theory that should cut your migration time in half for that And that's something that that we developed in house. So that's our solution And actually I have an interesting story about that too as of our our most recent release where there's This is moving towards the crane 2 world world We we kind of learned a lot and and and have codified that knowledge into a library called crane lib and so We're moving some of that logic out of mtc into reusable Libraries and then we're bringing that we're refactoring it and bringing it back into mtc So mtc now depends on crane lib. The interesting part about that is now crane 2's cli Is able to actually use the same mechanism and so we're developing all of that in in one core place There's also another project known as scribe. That's that's connected to red hat and that we're interested in Contributing to because it's also solving a lot of similar problems And of course, this is like kind of the open source way. Let's combine forces and start to do our development upstream So we've moved a lot of that that knowledge and logic into into crane lib And we're interested in contributing that into scribe as well Nice cool So lots of work Is what I hear lots of work is considered, you know being considered around state here Good good good good. Um And thank you. So I had a question that uh, if I got Far away back, you know, you were talking about crane 2 and some of the design goals and and things that you were looking at doing there and You know, if you think about crane 1 It's pretty much Sort of seen as a one-shot deal, right? And it's let's get you through that migration. I'm simplifying but with crane 2 it sounded like you're also guiding The application owner towards some practices that will be better in the future Did I understand that correctly? Absolutely. Um, that's the goal, right is uh, let's um while you're doing this mic it's it's almost like a um We try to avoid saying this but it's almost like an anti pattern to just pick up something and put it into the destination, right? Because you're You're carrying with you all of that cruft and and that legacy Just, you know weight that's associated with this like environment specific Application that's just not truly cloud native in the sense that it's like fluid and portable. So Um, we have the opinion that you know get ops is sort of the the solution towards that and and so that's one of the goals with crane 2 is to also help you adopt and help customers adopt the best practices because um, there's a lot of people who are not doing that. Um, and so when you when you're using those best practices, you're sort of it unlocks a lot of Things that are become available to you like hybrid cloud and all the benefits that are associated with it So that's one of the goals is to help people migrate to like not only migrate their application Which is their core interest but also get you like leave you in a better place than when you started Right. Yeah. Well, so, you know to make a metaphor of it.  I don't ever want to do that again And then you discover that there's other things along the way that you think well, okay If only somebody had told me That you know getting a storage unit, you know that I could that I could take the stuff and stage it there before Actually moving it to the home that would not be ready or you know until the the moving day You know, there's that's kind of what I'm hearing about crane 2 is that it does start guiding some practices Towards what we'll you know from from from the you know from experience, right? Is here's what has been a challenge in the past these are the hurdles that we have encountered Let's Just kind of set you up so that the next time you're having to do a migration You are going to have a much more painless one. You know, is that about right? Yes, that's it. That's I really I love that metaphor because it's actually really apt like When you're when you're doing it, you know, you you sort of like acquire all of this junk and then like And that's your source application But when you do that move it's an opportunity to kind of get rid of all of that make yourself a lot more Fluid and able to move again in the future without all of that burden So that's a that's a great. Um, that's a great analogy and then on top of that, um, the the It's almost like You you remove the necessity for migrations at all because you've moved yourself into an environment where you are portable enough where It's like I sort of it reminds me of when get started to become popular It was like for a long time branching was a very expensive operation It was not something that was to be taken lightly Get just said hey, why just branch early and often um, and it sort of changed a lot of those patterns like and and we're better for it and so, um It's almost like moving into this get ops world. It's just like I'm Uh redeploying into different environments and hybrid cloud. That's just a matter of course that that's a normal day to day thing And so you you remove the problem entirely Because you're fluid enough and you're designed to take advantage of of hybrid cloud Yeah, well, so I think that's the difference is the difference between being cloud native versus hybrid cloud native In a hybrid cloud native world The notion of migration starts to actually disappear a little bit because you've already built things in a way that You're not tackling it as a migration It simply just comes with the way you actually are detected the the app in the first place Yeah, you're you're basically just instantiating new apps Yeah um, so so I do have a question regarding that, uh, you know the notion of hybrid cloud approach so one of the the the things I liked with the with the two vert and open shield virtualization is the ability to have a hybrid application That is composed of containers and of virtual images, uh, you know within the same app so I was wondering What the approach would be or you know, what are the plans for that type of applications? So say I have a containerized app that talks to a vm that leaves outside of Of communities for for the moment, but I do want to migrate them so As of today, I understand that I have to use the two separate tools like I have to use mtc for the container part and I have to use mtv for the vm And at some point they're going to land. Maybe they can land in the same cluster So I have two questions. So the first one Would the correct way of doing it today like first migrating the vm from outside Kubernetes to the source cluster within the same like namespace of the application or something And then migrate the two together with mtc. So the question then is is mtc able to understand a CRD like virtual machine or is it just like CRDs that Kubernetes understands by default So that's the first part of the question and the second one is like if we think a little ahead Are there plans to have like an umbrella UI that allows you to combine both like I have Container coming from this and I have a vm coming from this and they do migrate as a single entity In a target namespace or something like that. Although they both come from two different tools Yeah, I think in that scenario, I actually have not had any I don't know of anybody who's actually tried to use mtc and mtv which is the container and the Tools in conjunction with one another crane and forklift I haven't seen it done and I haven't seen actually really any I haven't had interest come across my desk so to speak but I think There's no that doesn't mean that they can't be used in conjunction with one another mtc does understand crd's and actually it will Crawl and discover like cr's instances of the crd's and and actually migrate the crd's themselves Which is interesting because crd's are actually cluster scoped And it does handle some cluster scoped Resources when it recognizes that there's a dependency that's necessary Like the operator and it can go back to the operator to have the crd's. Yeah Operated workloads are actually an interesting problem that mtc in some cases has difficulty with because there's sort of an interesting This is a bit of a side tangent, but like chicken and migrate the Do you migrate the cr that the operator is actually operating on and delegate to the operator to reconstruct them the Application on the target side. That's actually probably the correct way to do it and then helm is another whole thing so I can get into that but to come back to your your question I would recommend in that scenario. I think crane 2 in conjunction with mtd would be an interesting option for you crane 2 is particularly Uh, I didn't I sort of touched on this a little bit But like we we built crane 2 to be a collection of tools and we like to call it like just kind of it's the classic unix philosophy full box 100 it's It's it's a bunch of like really small sharp tools that do one thing well that uh that can be composed to do Larger more interesting things and like as it turns out like they figured that out a while ago And it continues to be like a really good solution And so that's that's kind of the way that we're building that and we're intending on building An experience kind of around that where you can build these like pipelines that are executing those tools But like under the covers and we found this to be extremely valuable during some of our like more difficult production migrations We need those like those failure loops to be as tight as possible so that you're not waiting on timeouts You can't have this like monolithic migration. You need to be able to verify like the steps that are happening And and when you need to go backwards because something Breached your expectations. You need to be able to go backwards like like one little step not not back to the beginning of the migration So we've taken a lot of those Learn it like lessons learned and incorporated them into crane tool crane 2 So as a result because it's this collection of tools it's very Very flexible for its use so you can kind of pick and choose what's useful for you For example, like crane has a the crane cli has a command that says export If all you care about is just discovering and exporting everything that's inside of one namespace It will do that for you and it will output that then it has a transform command that will allow you to layer on those changes that jafar mentioned To strip things out really actually do arbitrary things because it runs it runs using a plugin mechanism And so you can use it you can choose not to use it And so you'd be able to use crane 2 and maybe the cli you can build solutions on top of it That you can then integrate with something like mtv Which is by the way also architected in exactly the same way that mtc is so that it has an api surface that is represented by a series of CRs and crd's So you can automate that you could build something that's calling into crane 2 and then maybe it's creating crs and and watching status of those crs To wait for mtv to migrate vm's and then maybe you go do something else. Maybe you move around to load balance or something like that so That's kind of our philosophy is that we're trying to build these like core building blocks And then we're going to offer like an opinionated solution that uses those We'll take sort of a Position on how we think you should use those but of course you can get into the toolbox and start building your own solutions easily as well Nice, that's cool. But that'd be to say that mtc2 would be the opinionated usage of all the tools in crane 2 Yeah, most likely And we're we're trying to to integrate with things like tecton and argo Um to build these pipelines that really like if you're familiar with tecton, it's going to look like a pipeline I slogged it for us and as you saw that said that yeah And and so maybe those those steps are running these crane commands and that you're you've got validations That are going to stop a pipeline in its tracks if if you've validated something and it's filled that validation so So I have another question. Oh my god so many questions. Yeah, so but that's that's a that's a lighter one Yeah, so so we'll mtd2 bill is just getting hired jafar. I mean, yeah, you realize you're going to be invoiced for this, right? Yeah, but this one this one is like is lighter. So will mtv2 be named vh1? I'll need to run that by uh I'll have to bring that one up. Um, that's not the first time we've people really like the mtv name Awesome. All right. That's what it beautiful. Thank you. Uh, let's get through some uh point sharing over here real quick I've realized we're we're kind of already diverging from our normal Schedule here to an extent. Uh, so this is the level up hour. You've been watching that for a little while. Let me know if you can't see my screen So about us so randy is here, but randy does not have A twitter so that's fine So I made him a shortcode just for uh, apparently it's not sharing the right screen. That's fun Uh, new share Share desktop 2 No, I was serious when I said that Wow zoom is just being fun today. Thanks zoom. All right. There's points. Um, enough inception so Randy has linked in Just read that ec slash triple r dash linked in they'll take you to randy's profile. Uh, scott mc bryan who I said is off today You can follow him on twitter Jafar is on twitter as well I am on twitter, uh, and we have our world famous discord where you can check in with us at any given moment in time Yeah, small peer pressure No big deal. Um, but If you are curious, uh show notes from last time I put those together. Uh, they are up there on the show notes page as well as Show notes from this episode are already done Uh, and you can find those in the level up our github repo, which I just dropped in chat But what you all really want to know is this sweet sweet internet points Who's got what and what's got who so if you want to participate in what we call sweet sweet internet points All you have to do is go to the form that's shown on on screen here and I'm going to drop links for everybody in chat Um, you too can get your sweet sweet internet points, which at this point Have no real value other than bragging rights but We hope to have some some real value to them in the near future Uh, we keep saying that and you know, maybe that'll happen. Uh, but the rend of is Clearly in the lead with and I'll hack them right behind them And then no affriction followed by debt conan kudo bacon fork and joe fuzz So if you want to see your name on this list Feel free to check the archive of shows and go back to get other codes to drop in Uh, to the form to get you more points and I think that's the last slide. Yes, it is so Any other questions from the audience any other questions from Jafar or randy or anybody at this point Erica, did you feel like you covered everything? Hopefully I didn't talk too much. Oh, this this was great and like that's like the I know this show is in good hands with jafar randy and scott because jafar can just ask questions that are related to everything um, so Yeah, uh, please check out the conveyor community itself length drop there and then When in doubt, uh, please like subscribe and share as always and uh Coming up next on the channel assuming there's no more questions or anything from the audience I'll give you a few seconds to chime in Coming up next on the channel today is ask an open shift admin with the one and only uh, andrew selivan But we also have a special guest my friend john spinks will be coming on We'll be talking about red hat insights for open shift. This is like having A a highly technical person sitting in your data center saying hey something's broken over here But you don't know that because this config mismatch is a very unusual kind of thing so It'll be interesting to see how that works in open shift I know we've talked about it on the channel as far as rel goes But in open shift clusters, that'll be a very interesting topic of discussion So Eric do you feel like you got everything off your chest? You needed to mtc mtv nta vh1 bet whatever yeah All that's gonna share one other thing. Oh, yeah, if you are interested in mtc, okay red hat does offer a course On Migrating so if you want to find that if this is all something that speaks to you and you need to know about this Please look up do 326. That's the course code Or our migrating applications course that we offer at red hat And you've got me googling now, so hang on. Let me find the link for that for everybody do 326 is the number Wow, that is actually used in a lot of different things Uh Okay, randy if you have a link to that. Yeah, I gotta include red hat with it. Otherwise, you might pull up Yeah, digital ocean. Who knows? Uh There we go red hat migration lab Ta-da Lincoln chat And there you go So, yeah, you can get uh you can get your training on For all the only other thing i'll mention too is that um conveyor is an organization is actually on the kubernetes slack instance So if you want to actually reach like all the engineers are there that are working on this you can talk to us every day We're in there. So that's um the kubernetes slack and then it's like just pound conveyor. You'll find us. Yeah Let me uh find the slack.kubernetes.io link real quick to drop in chat And you can find them on the conveyor channel that is conveyor with a k And they'll be happy to help you out I am also in that slack so feel free to ping me with anything as always you can reach me on twitter at kreshort Or if you have any questions or suggestions for future shows feel free to email me short at redhat.com And uh, yeah, thanks everybody We really appreciate y'all tuning in and thank you randy jafar And eric thank you chris. Thank you eric. Thank you. Thank you And see you soon. Yeah, we'll see you soon and uh check us out here at 11 a.m. Eastern 1500 utc for ask an open shift admin that is an office hour You don't have to just tune in for only the topic you can ask whatever questions you want during that office hour so feel free to jump on that and uh We'll answer all your questions for you. If not, we'll find somebody that can So thank y'all Thanks. Bye. Bye. Thank you