H dahod, mae gennym eu Sam Ilingworth. Rydw i'n meddwl am cymdeithasol ysgolwyddaeth yn yma ar fy ysgolwyddaeth ym Mhwyllgor Newydd. Rydw i'n meddwl am gael y Gweithdeithasol Gweithlu Gweithdeithasol. Mae hynny'n ei fawr yn ymgyrch yn ei fawr, ac ydych chi'n rhaid i gael gweithio i'r gweithio gwneud o'r ysgolwyddoedd a'r gweithdoedd ar y Mhwyllgor Newydd i'r gweithwyr yn ymdwy'r gweithwyr i'r cyfnodd o'r cyfnodd, a'r cyfnodd i'r cyfnodd, a'r diolch yn ymddangos. Yn ymdyn nhw'n gweithio'r cyfnodd, dr Jeff Hanson yw'r lefwyr yn y Ffwrdd Sgolwyr i'r cyfnodd yw'r cyfnodd. Jeff yn ymdyn nhw'n cyfnodd yw'r Cymru yn 2011, a'r cyfnodd i'r cyfnodd i'r Cymru yn 2013, ddiw i'r cyfnodd o'r cyfnodd ar y ddweud o'r cyfnodd o'r cyfnodd o'r cyfnodd. Mae'r cyfnodd yw'r cyfnodd dim yw sefyllfa a'i lleidgaredd o'r cyfnodd o'r cyfnodd, a ddim gweithio'r cyfnodd o'r cyfnodd yn ymddangos. Mae'r cyfnodd yn erbyn yn hyd y cynnig i'r gael a'u hynny. Bydd dr Jeff yn gallu gymrytro i'r ddweud o'r Cymru. Felly, Jeff. Dyma sut yw hynny am weithio'r cyfnodd. Mae'r cyfnodd yn ein hirymau Nouf, ym Gweithio'r Gweithio, oes i ni'n gweithio ychydig o'r cyfrifio gyda'r proses ymgyrch yn south-westod ar Australia. Felly, ddweud i'r ddilog hefyd, ac ydych yn gynnwys gweithio'i ymgyrch. A os ydych yn gweithio'r cyfrifio gyda'r ddweud ynghylch yn gweithio'r ymgyrch yma, ond yn ymgyrch yn llunio'r gael, ond ond mawr yn gweithio'r ddweud. this is a sort of an extreme example from a colleague at UNSW. But if you go down to the beach every day to walk your dog or whatever, you'll notice that the beach is almost always different. That's really what makes trying to understand coastal processes and coastal management quite difficult. This slide from colleagues at the US Geological Survey provides a good overview of the challenges that people that are in charge of managing the coast, really what they face. So basically the coastline changes over scales from seconds to centuries and from sort of individual sand grain size to hundreds or thousands of kilometres. But if you're a coastal manager, you're probably most interested in the areas, the scales in that sort of red maroon box. So from say individual beach a few hundred meters up to maybe an area of coast, a couple of hundred k's. And you're probably not interested in things happening on scales less than a single storm or more than a few decades. But the difficulty is that basically the changes that occur in that red box really reflect the integration of the movement of individual sand grains, right? Because if you go to the beach and the only way that the beach changes is based on the individual movements of sand grains. So that's a very important scale, but it's very difficult and impossible basically to really think on that scale. So just to point out really that managing the coast is very difficult because of this vast range of time and space scales. If you go down to the beach, I think the best way to think about what the beach looks like is it really reflects a number of things that are acting to make the beach look that way. And I'll sort of go through those really in no particular order, but a sand supply and size of the sand. So beaches need a consistent and sort of reliable source of sand and the size of that sand really determines how the beach looks. Sort of all else being equal for larger sand grains are going to make the beach sort of steeper, whereas smaller sand grains are going to make the sand flatter. What's the average energy input? Are you on a coast that experiences large waves and has big tides or on a site that has little to no wave energy and small tides? And similar to the sand size, basically, on average sites that have larger waves and bigger tides are going to have flatter beaches. So what's the sea level? Sea level is very important in determining what the very minimum where the shoreline is, but what's the present sea level and also what the past sea level has been, and that ranges over recent and more historic times. What's the underlying geology is very important. So in Perth, we tend to have a lot of what are called Perth beaches where there's sort of a thin veneer of sand, so about a meter of sand sitting on top of rock. And that's sort of what you saw on the title slide, or you go to some places and there's sand that would go down tens of meters. So what the underlying geology is really important is the beach essentially has to have a place to be. What are the previous conditions? What was there a storm yesterday or was there a cyclone a few years ago? And those previous conditions impact what the beach likely looks like right now, as well as the present conditions. So how big are the waves at this moment? Is it high tide? Is it low tide? So if you can kind of incorporate all of those, then you can get a reasonably good understanding of what the beach is doing. And I'll go through a few of those in more detail that really reflect my area of research expertise. But another and really important point that I like to make in presentations like this is that what you see when you go to the beach is just actually a small fraction of what we call the beach profile. So the beach profile sort of what incorporates the area from say behind the dunes out to what we call the wave base, which is the water depth at which the waves are able to actively sand around. And in Perthur, around the coast of WA, that goes out to say about 20 meters or so. And so when you go to the beach, you're really only looking at just that part that's above the waterline, which is quite a small part of this overall beach profile. But what the beach looks like and where the shoreline is really reflects the movement of sand over that entire beach profile, which in many cases is going to extend for more than a kilometer. Do you can't just consider what's happening above the water because often the processes below the water are just as if not more. So now I'm going to really kind of get into it and go through the drivers of beach changes. And I'm sure I focus on Perth, but really most of what I'm going to talk about is relevant to much of the southwest WA coast and a lot of other locations globally. So the main drivers of beach changes in Perth are really the variations of wave height and the direction at which they come from. So that's on the sort of seasonal to storm scale and changes in the sea level and that's over hours, two years. So we're looking at this contemporary times scale. So in Perthur or in southwest WA more generally, we sort of have three dominant sources of waves. And the biggest and most consistent is the southern ocean. So this is an image I took from a forecast output yesterday. You see four sort of red blobs to the south of Australia, and that's four separate storms marching across the southern ocean from left to right. And each one of those storms is sending out large waves indicated by the areas of red up into the Indian Ocean. And those waves are impacting the coast of WA and then also extending all the way up into Indonesia. Because there's no land in the southern ocean until you get to Antarctica, it's actually really the most active area of wave generation in the world. And the southwest WA is actually one of the most exposed areas to waves of anywhere in the world as well because of that. So the second source of waves that we have is from actually local winds. So the storms that are in the southern ocean are generating waves in the southern ocean, and those waves are emanating away from the storms that have been impacting the coast. But if we have quite strong winds out our coast, we're able to generate winds actually out the coast. And this is output of the wind that I took yesterday during that sort of storm that we had. So on the front side, so just as the storm is first arriving on the coast of WA, we tend to have quite strong northwest winds. And those winds from the northwest make waves coming from the northwest, and those are really important and probably the most damaging to our coast. And finally, in the summer, if you go down to the beach almost any afternoon, you'll notice that there tends to be quite a strong sea breeze, and that's almost always from the southwest. And those winds, just like the northwest winds, make waves that approach the coast from the southwest. And those waves tend to not be very big, but they are very, because of their consistent, are very important in redistributing the sand across the beach. And if you sort of integrate this all together, what you see basically is that we have a very cyclic nature of the wave height in this part of the world. So this is just a time series from the Cape Naturalist wave buoy over the last eight years or so, showing basically that on average, we have the largest waves in the wintertime and much smaller waves in the summertime. And then also notice that during some years, for example 2013, the waves tend to be much bigger than they are in other years, say 2015. And so because these wave conditions are constantly changing, we see corresponding continual changes in our beach. And this is just an animation I made for Trig Point over the last few years. And you can see basically the width of the beach is changing dramatically. And that's associated with the sand being taken from the beach by the waves, and that sand moves offshore. And by doing that, the waves break further offshore, and that's a means by which the beach sort of detects itself from those large waves. So the sand is being exchanged from the part of the beach above the water to the part below the water, and then moving back and forth. And that's a result of the constantly changing wave and water level conditions. So jumping into the water levels shortly, but first it's important to point out because of our regional geology here, that the coastline at Perth particularly is actually pretty protected from those really large southern ocean southwest swells that occur in the winter. And that's because, as you can see in this image here that we have a chain of islands going from Point Parin up in Sholes up to Rottness Island.  And as a result, by the time that they get to the coast, they're a fair bit smaller. So we're reasonably well protected from those big consistent southwest swells, but not so much from those northwest winds because they're able to sort of shoot through from the north of the Rottness and come along the coast. And this is a picture from that storm we had on the 25th of May that I'll talk about a little bit more from City Beach. So water levels are probably as important or in some cases maybe more important locally in determining where our beaches respond. And so in Perth, we have quite a unique sort of sea level climate, if you will. So we have our tides, which are about a half meter, and that's every 12 to 24 hours, but there are actually tidal variations that are as large as 18.6 years. We get a lot of storm surge, and that can be on the order of say half a meter as well, associated with the storms. And we had a bit of that yesterday, which I'll show. We have seasonal water level changes, and that is around the order of 20 to 25 centimeters. And those are mainly due to variations in the strength of the Lluan Current, which is the warm water current that comes along the coast of WA. And we also have inter-annual changes in sea level, mostly due to El Niño, El La Niño events, and those affect sea level by about 10 centimeters. And so if we lived in a place that had a tide range of say three to five meters or something, we wouldn't necessarily notice all of these other things, but because the tide range is only 50 centimeters, all of these other processes, the storm surge, the seasonal and the inter-annual water level variability, because they're all about the same order of magnitude, they really impact our coast. And those last two, the seasonal and the inter-annual variability are actually quite unique globally and sort of quite interesting processes. And finally, as we would all be aware, the sea level globally is rising in NWA that tends to be about three and a half millimeters a year. So all of these other processes are sort of superimposed on that longer term trend of sea level rise. So those northwest winds that we get, not only do they bring larger waves that impact the coast, they also bring quite strong storm surge. And the northwest winds cause the most storm surge because of the Coriolis force, which is the force that results as a result of the earth spinning. So the same reason that a cyclone rotates clockwise. So when we have winds from the north or the northwest, they actually result in water being transported to the left of that wind. And if you go to the left of that northwest wind, you run into the coast and they also sort of pile the water up against the coast down, particularly in Geograph Bay. So on the right there, you can see a screen grab that I got yesterday from the Port Geograph tide gauge at high tide around lunchtime. So you can see, if you can see my mouse there, you can see that the predicted tide was meant to be about a meter. And that's the black line on the graph. And then the blue line is actually what the observed tide was. And the difference between those two is what we call the residual. And in this case, that was just storm surge, which was about 50 centimeters. So we had a 50 centimeter increase in the water level yesterday compared to what we would expect from the tide, which is a big difference when the tide range is only about 50 centimeters. So we're effectively doubling the high tide. So the seasonal variations are mainly due to changes in the strength of the loon current. The loon current is this as a unique in that we normally have warm water currents on the east coast of continents, but the loon currents that comes flows, it's warm water current on the west coast of a continent. And for the same reason that the Coriolis force causes storm surges to be the strongest during northwest winds, because the loon current is coming from north to south, when the loon current is the strongest, which happens in the autumn in the winter, as more water is coming down the coast, it gets pushed to the left. And when it gets pushed to the left, it runs into the coast and that causes an increase in sea level. And the reason that it's the current stronger in the winter is mainly due to a reduction in the southerly winds that we have during much of the rest of the year. So here is a 90 day running average of the water level from the Port Geograph tide station. And what you can see is a very distinct seasonal signal, not unlike we saw on the waves, but we have the highest water levels in the winter and they're about 20 centimeters higher than the water levels in the summer. So this is effectively increasing the mean sea level at a location by 20 centimeters for a couple of months, which is actually quite unique globally, quite interesting. And then we have the lowest sea levels in the spring and the summer. And that's because we have really strong southerly winds that sort of hold that warm water up to the northwest. And if you would have looked closely, you also see quite big fluctuations year to year in that water level. And those are due to Launinia and El Nino events because those events affect how strong the loon current is. So during Launinia events, we tend to have a very strong loon current and that strengthening of the current causes higher sea levels. And so that can change sea levels by another 10 centimeters or so. And that's really quite a unique thing as well. But this really impacts our coast because if you increase the sea level by 30 centimeters from different seasons, for example, you really affect where the shoreline is and how far up the beach the way it's able to go. And this effect from Launinia and El Nino in Launinia is not just at the coast. This is really a regional over much of the eastern Indian Ocean. And this is just a plot from satellite of Altimetry showing basically that this increase in the red or the decrease in sea level during El Nino versus Launinia is really over the entire eastern Indian Ocean, but concentrated out the coast. So now getting more into the research, now we have an overview of the sort of processes, the sort of modern day processes. Can we use that knowledge to really try and develop a system that will predict what the beaches are going to do from these storms? Not unlike you would predict how strong the wind is going to be from a storm like the pure of meteorology does. To get at this, we have an Australian Research Council linkage project with partners University of New South Wales, Bureau of Meteorology, and locally the city of Mandara and the Department of Transport. And that's to really test whether or not we can come up with a system that will develop an early warning system for coastal erosion. And that includes two components which I'll run through quite quickly. And the first is basically a regional scale medium resolution first pass.  So we predict the areas that are going to have the biggest waves and therefore likely the most erosion. And the second is really at a few sites, can we apply very detailed models to try and predict essentially how much of the beach is going to change, how much sand is going to move where the shoreline is going to go. And the storm that we had on the 25th of May really provided us with our first data set for this project. And it was quite an impressive storm. I think the sort of consensus is about a one in a 10 year event and it was quite unique in its broad spatial extent. So the left figure there should is essentially output from a Bureau of Meteorology model basically where you have large waves and indicated by the dark green extending more or less all the way from Albany up to Xmouth. And that's quite uncommon. And on the right there is a sort of similar model that we've set up specifically for this project to predict the waves and this is during the height of the storm. And this essentially showed us that waves offshore were upwards of about 10 meters. And this also then is feeds into that regional first pass because from this we can look at the areas along the coast that have the largest waves. And the biggest waves from this storm came more from the west, actually. And so you can see that the areas to the north of Rodness Island there, north of the Swan River actually were getting a fair amount of wave energy more than they would if the waves were more from the southwest. Whereas the areas to the south were still reasonably protected. And then once you get south of Point Pair and everything was pretty well exposed until you get down into Geograph Bay. And we were really quite lucky with this particular event because we were able to get a couple of other data sets that we normally wouldn't get from a storm like this. And the first of that is that we operate a couple of drifting or a couple of wave boys, two of which are drifting. And those drifting wave boys happen to be sort of in the perfect spot to record the waves from this particular storm event. So the first is sitting about 500 kilometers to the southwest of Cape Llewyn. The second is sitting about 500 kilometers northwest of Geraldton. And they were floating more or less in those same locations when that storm came by and gave us a pretty impressive direct measurements of how big the waves were during that storm. And so this is sort of the time series from those two locations. So the red is that wave boy to the northwest of Geraldton. The black is the one to the southwest of Cape Llewyn. And the one off Cape Llewyn exceeded nine meters on the sort of 25th of May there. And the one to the northwest of Geraldton got up to about eight meters. Now waves that large aren't that uncommon in this part of the ocean, but what was quite impressive about this event is particularly for that location to the northwest of Geraldton is how big the waves were, but how long they persisted for. So the waves were more or less in excess of seven meters from more than 24 hours, which is actually quite uncommon. And then you can see for that location to the southwest of Cape Llewyn and the black line, the waves were sort of between six and seven meters for another several days. So essentially over almost over six meters for about a week. And the second unique data set that we have for this event is that with our partners, the parent naturalist partnership, which is the consortium of local councils from Rockingham down to Busselton. We for them fly the coast that section about 200 kilometers of close twice a year in a helicopter and take photographs essentially out of the helicopter to document the changes in the coast. We sort of did our normal bi-annual flight in early May as sort of scheduled. But then after this event, we were lucky in that the parent naturalist partnership was able to fund another flight for us to do that. And I'll sort of highlight some of the pre and post storm imagery now. And all of that imagery is available on a website that we maintain, as well as the wave data is available on another website that I showed a few slides ago. And the URLs are both on that presentation there. So just looking at a few different sites. So this is Busselton Jetty actually sort of before the storm. And this section of beach actually wasn't looking that great. And that's because we actually had quite a big storm in late April and early May. Some of you may remember. And all of that brown stuff on the beach is beach racks. So that sea grass that's sort of been ripped out of the water and washed up on the beach. So this is the sort of pre storm. And then a few weeks later, we can see the beach to the left of the jetty got pretty well eroded, as well as to the right on the far right. You can see that water is right up against that riprap that those rocks. And we also have a lot more rack on the beach. Moving further down the coast. And what was particularly interesting about this event is that water levels were quite high because of that storm surge. And the dunes down and once you get into Geograph Bay are quite low, about two meters. And so what we saw is quite a lot of evidence of overwash and inundation. So if you look particularly at the house with the swimming pool there and the ones the next to the right of it, you can see that they have nice grass in their backyards. And now if we go to the photo after the after the event, you can see quite a lot of sand has been pushed into the backyards of those houses. And that shows a significant evidence of overwash from that event. Just not too far down the straight sort of similar thing. We have some houses here that actually show some evidence of inundation prior to the storm. But then if we zoom through, you can actually see quite a lot more evidence of inundation. If you look at that vacant block there in the middle, you can actually see what looks like a fan as the sand ran over that dune and then sort of washed out and spread out as it got into that calm open area. And as sea levels increase, then these events like this are going to become increasingly common. And you can see in these last two photos, the water essentially was right almost at the back of some of these houses. And this was only a one in 10 years. Moving up the coast to Mandra, so the section from Mandra down to Bunbury is fronted by much, much higher dunes. And the dunes from that event were really significantly eroded. So just looking at this photo of Mandra before and after, we see all the dune vegetation cut back and quite a significant cliff there on the edge. And similarly in Cable, if you just sort of focus on the areas near that stairway in the center of the image, that's the before and after, the dunes are quite cut back. And then actually if you look at a photo of that stairway from after the event, you can see basically the footings for three of the posts holding that stairway are either exposed or totally ripped off. And you can see all that fresh vegetation on the beach indicating that that dune has been totally undercut. And the second thing that we did at our very focused site in Mandra, we did pre and post storm drone surveys. And for those of you that would have seen Nicolo's science exchange talk a few weeks ago, you've provided a great overview of how you can use drones to calculate or to produce very detailed digital elevation models. And in the coastal space since about 2015, drones have really taken off and replaced much more time consuming methods of serving in the beach. So we were able to get out just before the storm and then after it again. And essentially what happens is the drone flies up in the air, takes all of these images and you're able to stitch those images together using a technique called structure from motion to produce a very detailed digital elevation model of the beach. So this is the site in Mandra just to the south of Dawsfield Channel. So we're able to stitch those images together and this allows us to survey almost two kilometers of beach in about 40 minutes, which would have taken hours and hours and hours with sort of the older techniques we used to use. And from an event like this, we can do our pre storm survey or post storm survey and then take the difference of these elevation models and that will give us the elevation change. From this event, we essentially saw a pretty significant erosion of the dunes upwards of a meter and a half in some locations indicated by the red there in broad areas of excess of a meter of elevation change. And the dune erosion I'd like to end on really this is the last slide, but basically a lot of people, it can be quite distressing to see the dunes get eroded like this. But I'd like to think about the dunes as really being the beaches savings account. So when the times get tough that the beach can draw down the sand that's stored in those dunes to buffer against the storms and protect more. And that's part of the natural system and you know the dunes will hopefully come back, but when you get into the big problems is when you have development on those dunes or the dunes are otherwise sort of eroded. But you know what the dunes eroding like this are really providing their sort of natural buffering capacity and so long as they're able to perform this natural they this natural cycle, they really provide a natural buffer for the beach. So that's all I have. Hopefully that was enjoy one and provided some good information about what happens along our coast. That was great Jeff. Thanks so much for that interesting talk and got quite a few questions coming in. So if people can just add their questions to the Q&A at the bottom, then we can and we can pick those up as we go. So the first question we've got from is from Leslie and Leslie says some council trucks some councils trucks sand into popular beaches after erosion. Is there any value in that? Yes, there is so that's what we would call sand nourishment and so sand nourishment can be a good way to alleviate temporary erosion. And that's become quite a popular way to avoid having to build seawalls and other hard engineering structures. The disadvantage of it is, is that it's sort of a continual process. It's not something that you can usually do once and then forget about it. You have to kind of commit to continue to do that, but it is much more of a softer sort of more natural means of coastal protection. Okay, great. Thank you. So the question from Emma, Emma says, why is there such a large amount of rack and debris on Kotoslo beach? Is it a result of the storms or are there other ocean conditions that are influencing the debris? No, it's it's purely a result of the storms. I mean, there's been a couple pretty big storms this year already. And at the end of the summer, all of that sea grass and other aquatic vegetation is all is all grown up. And as those big waves come along, they have provided make strong currents and a strong current to rip all of that off the sea floor. So it is, it is quite a natural process. The problem is when you have structures that catch all of that, like jetties and like. Okay, loads of questions coming in now. So from Karen, Karen asks, is there any evidence that the storm events on coastal erosion are getting more severe with time? Well, the general consensus is that the storm intensity may increase. And this is speaking more globally rather than locally. But the frequency may decrease. So locally, generally speaking, in a warmer world, you would expect the storms to be pushed further to the poles. And so generally the expectation is that that WA will get less storms because more storms will be passing south of Australia. But the storms that do hit because they're over a warmer water may be more energetic. But it's still a very active area of research and debate within the community. Great. Eva's got a question that I'm desperate to know as well. Where do you source the sand that you put back on the beach? Do you get that from another beach?  Yeah, that's an excellent question. And it's a question that people have thought about for eternity. So it sort of depends where you are. A lot of places it's very common to actually dredge sediment from offshore areas and put that on the beach. That's not done in WA that I'm really aware of. Usually it's from quarries or other inland sources to most of my knowledge. But in some places, yes, they do take it from one beach and take it and put it on the other. And actually in Mandara, in a lot of places where there's natural or artificial structures that block the movement of sand, they often will pump it so they make a slurry and they pump it from one location to the other to get it around the jetties or various structures. Loads of really nice comments just saying what a great presentation it was as well, Jeff. Question here from an anonymous attendee. What is unique about the West Coast of Western Australia to allow the presence of a warm current rather than a cold one as usually happens? It just has to do with the sort of geography of the Indian Ocean basically. So you get quite a lot of warm water trapped up in the Gulf of Carpentaria. And that water, as it sort of spills out of that area into the ocean, it wants to flow to the south. And normally when you would have it, you normally just don't have that warm water up there, but because it's flowing to the south, and the Coriolis force forces it to the left, it runs into the continent of Australia. So it really has to do with sort of just the geography of the Indian Ocean basin. And Guns got a question. Is there a way of knowing if the sand washed out will return in part at least? That's more difficult to predict, but Perth or in the southwest coast of WA is actually quite lucky in that there's really a massive offshore reservoir of sand. And so while there's been isolated areas, pockets of long-term erosion, most of the coastline around Perth particular has actually been fairly stable. And that's largely because there's an offshore reservoir of sand, and that sand continually migrates on shore. OK, great. So Simon's got a question saying the shape of the coast shows geomorphology that suggests both south to north sand transport, and then big storms that also head north to south. So in your opinion, which is the geologically significant vector likely to lead to long-term preservation over thousands of years? Simon, we can talk about that potentially later, but you're about three orders of magnitude beyond my attention span with thousands of years. Fantastic. So Robin's got a question here saying rock groins constructed 30 to 40 years ago at Sorrento beach have contributed to unnatural sands movement. Should and can these be removed, do you think? Well, I mean, they certainly could be removed. The should is a more difficult question. So, I mean, the groins there are actually just sort of part. I mean, the whole Hillary's Harbor is a natural structure that's blocking sand transport. So the groins there are kind of more of the sort of bandaid to protect the much larger injury, which is the harbor. So I think you have to would have to consider that whole system rather holistically rather than think about the individual groins. And in a related question here from Meg, she says that to protect coastal assets close to June's being eroded. Is it recommended to use some engineering treatment e.g rock walls at the base of the June as well as planting on the June as well? Well, I mean, certainly if your objective is to not allow the shoreline to erode past a certain point, then rocks or sea walls or whatever are kind of your only option.   But if you make a decision that you don't want to allow the coast to erode any further, then the solution sort of is has to be those hard structures like that. Great. And then final question from Aggie, which is saying that there are citizen science means of collecting data, for example, through Flucopost or the Photomon app. Would you be interested in data gathering through these means and which areas of our coastline are the most dynamic and least studied where this would add value to your research? So that's a great question. And actually next, well, in the next few weeks, we're going to be launching a program called Coast Snap. And so that was this initially started by colleagues at Uni of New South Wales, but essentially at nine sites in the coast from Rockingham down to Busselton, including on the Busselton jetty were installing these photo brackets. And the idea is that you put your cell phone in the photo bracket so we can then you take a photo and then you upload it to one of our sites. And similar to as what we do with the drone to were able to, by knowing in known position of a few sites locations within that photo, actually get quantitative information on the shoreline position out of those photos. So yes, these are very valuable. And stay tuned in about two weeks. These are all going to be active at nine sites from Rockingham down to Busselton, and hopefully we're going to expand that. We'd like to do it at some sites on the south coast, because the coastal processes on the south coast are really been much less studied in an area that I would really like to expand into. So I'd just like to thank Jeff very much again for such a fascinating talk. So many questions, Jeff, but you didn't see it, but a really, really active chat going on as well with people coming up with some great suggestions for how to manoeuvre sand effectively between the different beaches. Also to the attendees for such a really, really rigorous series of questions and a great chat all the way through and everyone stay safe and look forward to seeing you at the next exchange series. Thanks and goodbye.


Welcome everybody. This is our panel about the state of the art for enabling performance sensitive workloads and what we need to do in the future. And this panel is about how different companies and different participants are working together in the Q&A ecosystem to enable the running of performance sensitive workloads in the Q&A ecosystem. And I would like to ask our panelists to explain who are they of course, which companies are they coming from and what is their motivation to participate in this work. Swati. Hi everyone. I'm Swati and I'm based in Ireland. I work as a principal software engineer in the ecosystem engineering group at Red Hat. My team and I have been focusing on enhancing Kubernetes and OpenShift to enhance and enable our customers and partners to run performance sensitive next generation workloads on Kubernetes. The primary use case that we've been looking at is how to make the Kubernetes scheduler topology aware. The performance sensitive critical workloads in the industries like telco, 5G, machine learning, artificial intelligence, high performance computing, they require resources such as CPUs memory devices to be allocated such that they have access to the same local memory. And essentially that leads to optimum performance. Topology manager, which is a Kube. The component was introduced for topology alignment of requested resources at a node level. But scheduler's lack of knowledge of the underlying topology can lead to suboptimal placement of workloads. And we are trying to solve this problem. Thank you, Sasha. Oh, I'm Alexander Konevsky. Some people call me Sasha. I'm cloud software architect. I work for Intel and I'm part of a team who are looking at resource management topics in Kubernetes and in CNCF infrastructure. As we are a hardware company, obviously, we have a good knowledge of how to optimize the workload to get the best out of available hardware. And our team is working on enabling different pieces in something in Kubernetes, something in runtime, something in add-on projects. And our goal is to give a possibility to utilize hardware 100% for all your needs, for all your demands. Thank you very much, Cliff. Hi, my name is Cliff Burdick. I work at NVIDIA as a DevTech engineer. Currently, I do optimization on GPU code and kind of working on the input and output of GPUs from a network perspective. I'm trying to optimize that and get the latency and throughput as high and low as possible. Previously, I worked at a satellite communications company where we built out something similar where it was a high throughput GPU and NIC solution, where we solved some of the problems that we're going to talk about today using an alternative method. Thank you very much, Alexey. Hi, I'm Alexey Pirivarov. I work in the Advanced Software Technology Lab of Huawei, Russia Research Institute, Moscow. I have been working on supporting performance critical application in orchestrators like Kubernetes. I'm working on it and on OpenStack in the past. I mostly focus on bare metal deployments. My scope is covered by deployment security hardening and other use cases for multi-tenant scenarios. Thank you very much. My name is Gerger Chatari, and I'm working in the Open Source Program Office of Nokia. As we are a telecom vendor, we are interested in running our workloads in the Kubernetes ecosystem. Of course, some of our workloads are performance sensitive. This is why we are interested in in this effort. We are supporting the effort with requirements and let's say technical consultancy. Let's jump to the technical details. I would like to ask you to explain a bit the solution space of this problem. Sure. Kubernetes has become the de facto standard for container orchestration. It's attracting performance sensitive workloads that are very demanding and need the speed and raw performance as if running directly on bare metal. In the diagram here, you can see an example of a Kubernetes cluster where we have master node where the control plane components like EPS server and scheduler are running. Then we have two worker nodes. Each node is running Cubelet, which is the node agent. It communicates with the control plane and makes sure that the containers are running in a pod. To create a pod, Cubelet needs a container on time, which is responsible for running containers. Kubernetes supports several container runtimes like Docker, container D, cryo, and any implementation of Kubernetes CRI, which stands for the container runtime interface. Then we have, of course, the hardware, the underlying hardware. Our aim to enable Kubernetes in Kubernetes is the support for resource management for next generation workloads on the underlying heterogeneous hardware. In order to achieve this goal, we need to tackle this problem at various layers. We need to solve various problems at cluster level, node level, runtime, and the hardware level. There are a few key questions that need to be addressed. How do we make sure that at a cluster level, the scheduler is able to make placement decisions not merely by taking the amount of requested and available resources on a node, but also taking into consideration the underlying topology of those resources? At a node level, how do we ensure that resources such as CPUs, memory, PCI devices are aligned for optimal performance, and then there are optimizations at runtime and the hardware level that we need to be able to leverage for low latency and high throughput applications. Then the other aspect of this is at some layers, we have the ability to create custom plugins to alter the default behavior. However, in the other layers, we cannot do that. Therefore, to solve some of these problems, we need to create plugins. Whereas for the others, we need to enhance the core of Kubernetes itself. Thank you very much. I think it's a very interesting and several component problem. It's an interesting question, how the members of this group are collaborating together and how this whole problem space is not solved in a Kubernetes Sieg or a CNCF tag. Alex, can you tell us some words about that? Yeah, sure. Kubernetes Siegs such as Sieg node is responsible for everything related to working nodes. Whereas other Siegs and Kubernetes community is responsible for everything related to Kube schedule. For example, CNCF tag calls mostly runtime such as ACCRI container runtime interface or OCI open container initiative specification. But before proposing something in those Siegs, the group of involved people gather together and tries to work out a common solution. Try to find common requirements. The communication occurs in Slack, sometimes by video call conference or in Google groups, when we are gathering together and discussing issues or making a brainstorm. But Sieg node meeting is not the best place for brainstorming, much better to meet them before to prepare slides and then go to Sieg node where we are sure we found a solution which satisfies our general requirements. Thank you very much. And now we are talking about the requirements of next generation performance sensitive workloads. But what are these requirements exactly, Sasha, can you explain that? Well, first of all, I need to say what those requirements are unique to each workloads. And everybody in the space who are using Kubernetes for workloads, we have probably different meaning of that. So if you have a compute workload with high demand on CPU time, most probably what you really need is exclusive core or set of course allocation and exclusive usage of a cache to reduce the disturbance from other processes. If you have memory intensive application, you need to have a good alignment not only on the CPU but also memory controllers. And when probably you need to optimize how it's allocated. If you have some devices, let's assume like the 5G network where you have strict requirements about latency, about the bandwidth, about the ability to process the packets in predictable time. When you need to have even more complex setup where you have like the CPU, the caches, input output lanes, the PCI devices like network cards and so on, all of them set as an exact pipeline which delivers optimal performance for your particular setup. And as an example, Cliff actually mentioned what he in his previous company worked on something similar and I would like to pass on a message to Cliff. Cliff, can you explain like what kind of complex setup you had in the past? Sure, thanks. So a couple years back in my previous company, we were working on an NFV style application where we had many 100 gig NICs, network interface cards in a node, many GPUs in a node, and a dual socket system. So it's a fairly common type of system when you're looking at NFV or accelerator applications and we had it containerized already. So the way that we were deploying it before was in Kubernetes, we would manually specify which resources we needed in the pod spec and the pod itself would go consume those resources. And this was very difficult to maintain because every time you wanted to deploy something, you had to specify exactly to the pod what it should claim when it started up. So it obviously wasn't scalable and it wasn't packing as many pods on the nodes as possible. So we looked at our node architecture, which is similar to the one shown here. At the very bottom, we have the CPU or in this case, it was one Numa node per CPU and it was a dual socket system. So you had two Numa nodes on there. And then in the purple and the kind of teal color, that's to indicate the hyper threaded cores. So you have a core and a sibling core that are paired together and affect each other in performance to some extent. Then above that, you have the PCI topology, which is where all of your devices are connected. So in this case, we had GPUs and NICs. And in this particular case, I'm showing a GPU and a NIC connected to a particular PCI switch. And then there's a tree of those going down to the actual CPUs themselves. And then at the very top there is the backplane for the GPUs. So in this case, I'm drawing it to one GPU to one GPU, but it could be an all to all. There's many different configurations that it can be. And then annotated in the text is the actual rough throughput of each of these links. And so you could see down at the bottom, you have about 30 gigabytes per second for the inner processor link. And then on the way up, you get 20 to 150 gigabytes per second. Now, if you were to schedule a workload naively and place it on any GPU and any NIC, there's a high probability that that traffic from the GPU and the NIC will go back through that inner processor link, which you want to avoid because it would be a big bottleneck if you compare aggregating all those 20 gigabyte per second links at the top. And really what we wanted was we had a NIC, which was feeding 100 gigabits of data into a GPU streaming, processing that data and then sending it back out. So you really didn't want the data going through that inner processor link, but you also didn't want it going through the second PCI switch in that tree either because that means that you have a bottleneck with the other two GPUs and NICs that are also going through that switch. So what we ended up doing, we looked at the Kubernetes landscape. And at the time, Kubernetes, the main way that we could solve this was Kubernetes had a way to plug in to the scheduler. And so all this meant was that if you wanted to run your own scheduler, Kubernetes, when you launched your pod, Kubernetes would hand off the entire scheduling process to your pod, to your scheduler rather. So we built a custom scheduler that we called NHD that handles all these scenarios. So it's hyperthreading aware. It's PCI switch aware. It handles huge pages, which are not shown here, but those would be the memory aspect of it. It handles NUMO awareness. So that's the part that Sasha touched on a little bit. So it's kind of handling our extreme case where we needed to pair up these GPUs and NICs. Now the big downside is that since it is a custom scheduler and it was back at that time, there were no easy ways to plug into the scheduler. It meant that we had to take on the entire scheduling process ourselves, which was a big pain because we didn't get any of the benefits that the scheduler already had, such as knowing how much disk space is available on the node. We would have to do all that stuff ourselves. And we did, to some extent, kind of reinvent the wheel. But since then that has changed quite a bit. And we'll talk about that later in some of the upcoming slides. This project is open source. If anyone wants to go look at the code or use it, the URL is right there. And feel free to ask questions at the end as well. Thank you very much. So now we saw how this program can be solved with the custom scheduler. But the question now is how can we solve it in the different layers of the Kubernetes ecosystem, where we have the runtime and hardware level, we have the node level, and we have the scheduler level, and all of these levels need to work together. And all of these layers have different kind of gaps that we need to close. Swati, can you tell us something about the gaps in the scheduler layer? Sure. So I touched this on my intro. So while making the scheduling decision, the Kubernetes scheduler looks at the amount of requested resources and determines the nodes that can fulfill that resource requirement. But it doesn't consider the topology manager policy on that node or whether or not those resources can fit on the same NUMA node. So essentially, scheduler lacks visibility into resources available on a per NUMA basis, which can lead to unpredictable application performance. At a node level, topology manager coordinates the topology of resource allocation of CPUs, memories, devices, and helps to extract the best performance out of the underlying hardware. However, in scenarios where topology manager is unable to align topology of requested resources based on the configured policy, the pod is rejected with topology affinity error. And if the pod is part of a deployment or replica set, it results in runaway pod creation, because the subsequent pod that is created ends up with the topology affinity error as well. So in order to optimize cluster wide performance of workloads, resource utilization and enhance the overall performance of system as a whole, the default scheduler should consider the resource availability along with underlying resource topology to increase the likelihood of a pod to land on a node where it can fit. So essentially what we are doing to solve this problem, we are introducing a few components and in addition to that we are making enhancement through some of the existing ones. So there are two pieces to this puzzle essentially. So one is a component that exposes resource information along with granularity of NUMA node. And then there's the scheduling piece where we have to enhance the scheduling process to take that information to consideration to make a proper scheduling decision. So as I mentioned, the first piece of the puzzle is node feature discovery, which is often referred to as NFT. It is a project that is part of Kubernetes CIG repo. NFT is a node fleecing agent which exposes hardware capability in the form of node labels, annotations, extended resources. We are adding a software component called NFT topology updater, as you can see in the diagram that collects information about the resources allocated to running pods along with associated topology information using pod resource API to determine the available resources with NUMA node granularity. And then we expose this information as CR instances per node. Let's talk a bit maybe about the pod resource API because I guess not everyone would be familiar with that. It's a Kubelet endpoint for pod resource assignment and we enhanced it to add support for exposing CPU IDs and device topology information. And the other thing that we added as part of this was additional endpoint to obtain information of allocatable resources. Then the second piece, as I mentioned, is the topology wear scheduler plugin. Again, Kubernetes CIG repo houses a repository for out of free scheduler plugins which are based on the scheduler framework. And we contributed node resource topology scheduler plugin there. This uses the CRs created by NFT to make a NUMA wear placement decision. Essentially what it does is it runs a simplified version of the topology manager alignment algorithm to determine if a node is suitable to assign the pod to a particular node. Then there's a glue between these two components, which is the node resource topology API. That is a CRD API, which essentially is used by these two different components, NFT and the scheduler plugin. An important thing to note here is that topology managers still runs its alignment logic at a node level for resource allocation and the scheduler plugin essentially ensures that the scheduling process takes place and the part is assigned to the right node. Another few kind of understood the challenges at the cluster and scheduling level. Alexi, could you please share with us how resource management looks like at a node level and what are the gaps that you see in at at kubelet and node level? Yes, sure. We need to list every component in kubelet which are important for high-performance critical workloads. Such components like topology manager, CPU manager, device manager and manager manager each has its own responsibility. The topology manager is responsible for aligning resources to a normal node. The CPU manager is responsible for exclusive CPU allocation. As you already said, device manager registers the set of device plugins. You can see it on the picture and provides resources, so we each have a name of extended resources to the cluster level. Also, if a device plugin provides a normal quality of device inappropriate resource, in this case, the device manager could help the topology manager to align the device on the same normal node where CPUs were exclusively allocated by CPU manager, a particular container. The latest component is memory manager. It also helps to guarantee normal alignment, but now alignment of memory. Before memory manager was introduced, we had to rely on Linux kernel or organize huge terabyte-first-month points in other world. We did it manually. Let's describe the port manager in details. It uses as a manager which implements in-provider centerpiece as the manager, device manager and memory manager that implements this interface. Bitmap of the possible normal allocation is provided by in-provider centerpiece. The port manager implements different policies. The most important policy for high-performance critical growth is single-numar node policy. This policy guarantees the target or resources from the same numar node or raise topology affinity error if it's not possible. Last year, the scope parameter was added into topology manager. As we know, component districts are resources per container and topology managers. Policers also apply its policies per container. For example, in case of single-numar policy, it could be a situation when two containers from one port are correctly placed in different numar nodes. It could affect application performance. For example, if we have those containers interaction through memory, like db application, we have to clarify until memory manager was introduced. It was not 100% true. The idea was to introduce behavior to apply policies per ports, not per container. For this purpose, the program manager scope option was introduced. The port value for this option means to apply it upon the measure ports, the per ports resources, when the container value of the option is the default behavior as it was before. In this case, any resources of all containers of a particular port will be in the same numar node. Of course, if single-numar node policy is labeled on the node, to have exclusively allocated the CPU, the static policy of CPU measures will be selected by Qubelet configuration or by Qubelet command line option. The hyper-creator that the CPUs and chips with correstos of last-only locations are out of the scope now. But recently, this year, the CPU manager policy option was introduced, which aims to allocate all the CPUs from the same physical cores, but many other use cases are not supported yet. Let's move to device plugins. As you can see, device plugins are working outside of the Qubelet. Nowadays, there are a lot of device plugins, most of your ISROV device plugins into or under the network deployment working group when detonability. Also, NVidia GPU device are plugins where a popular device plugin. Before the device manager, this problem was solved by an APAC integer resource. In that time, for example, not officially discovered, provided APAC integer resources. For example, for ISROV, APAC integer resources was replaced by extended resources. Now, it's a history. Briefly, the memory manager calculates topology hints for a container and graduated quarter of service port for conventional memory and huge pages of all sizes. The poetry hint represents a possible set of non-annuals that has enough capacity to satisfy containers memory demand for all memory types. Before memory manager, we had to rely on Linux or organize huge at a very fast point. It was a current situation. George, your next question. Yes, thank you very much. Sasha, can you tell us something about the lower layer, so how the runtime and hardware layer is organized? Well, thanks for asking. I would like to complicate the picture a bit more compared to what Swati and Alexei explained. Yes, we have right now in the upstream Kubernetes set of CPU memory managers, topology manager. All of this tried to solve the most common setup, like the most simplified architecture of the system. You have one socket, which equal to one NUMA node, which consists of some CPUs and some memory. And assumption also, what this socket has only one single iobus. Reality a bit more complex. Even with hardware, which is released in the past several years, we have scenarios what like cores might be different. We have different performance or power settings. The caches might be different on those, where a socket might have several NUMA nodes where memory controllers are working in different mode. You can have multiple PCI buses. So some cores are closer to PCI buses, some are further. You have different types of memory, where these different types actually means different performance or like bandwidth or latency of accessing this memory. And none of those hardware details is actually visible to the Kublet, because the Kublet and overall Kubernetes was built on the promise of being hardware agnostic. And the things what we have is trying to simplify the task for most common problem, for most common hardware, but it lacks all the flexibility for custom solutions. And this is where we need to look a bit deeper. So what we can do on other layers. Of course, there are some solutions which tries to hide that complexity by hypervisor. So for example, like VMware Tanzu, it tries to optimize the hardware placement of a virtual Kublet node transparently. But if we're talking about some other deployment, like either cloud or bare metal, we have not that many pieces where we can plug it. So we have between the actual hardware and Linux kernel, two boxes. One is CRI runtimes. And luckily, there are practically two projects, most active crier and container D. And when we have actual OSI runtimes, and this OSI runtimes, we know how to operate with a kernel, with actual hardware nodes what kernel provides. But as we have this several layers of abstractions like CRI, OSI, and so on, it means what like low levels can modify the parameters more actively, but we are not exposing all the knobs. Same same losses between the CRI and the Kublet. Kublet evolved from the old times when we had a Docker shim. So the Kublet CRI interface is partially declarative. So it describes what needs to be run. But when it's partially imperative. So all this detection or creation of OSI groups, some of OSI group setting is dictated by the Kublet directly to runtime. And while we had only run C, it was okay. But nowadays we have VM based runtimes like Cata containers, GViser, and so on. So as soon as we start to use those like micro VMs, some of the assumptions about like what actually Numa node, for example, means is not exactly the true. So we looked at different means how to extend that. Like obviously you can have a custom OSI runtime, but you are limited of what you can do there. And when we looked at the CRI runtimes, CRION container D. And about a year ago, one of the maintainers of container D introduced the project which is called NRI, node resource interface. At the time of introduction, this project was very simplistic. So it utilizes the idea of C and I plugin. So something what can be executed on the start of a container. And it can alter some of properties of newly created container. But if you really want to utilize all the flexibility what hardware provides, and if you really want to have a plugins what knows about details of your hardware, you need to have something more flexible. And right now our team plus a few our people from the community, we are working on improving this NRI interface to be flexible. So we can hook into all life cycle for containers. It should be easy to deploy those plugins. And we would like to make it interoperable between those two major container runtimes. So this tries to solve the flexibility to the level what you can have a custom installation with custom resource policies tailored for your hardware needs. And these are the extension points for all the controller for resources. Thank you very much. And let's let me ask back to Alex say for a minute. So are there any work in the built-in resource management at the moment? Yes, right. Right now we'll find out that there is external resource management. Sasha told about it. But the work in internal and built-in components are still in progress. So there is an enhancement proposal regarding last time on caches, but it's still in review. So you can contribute. You can participate in reviewing and in implementation. Enhancement proposal for changing CPU distribution among sockets and cores is also on review. Port resource API also evolves. Additional features are added to the port and are on the way and changes are coming up for new version of port resource interface. Yes, thank you very much. And I would like drop the ball back to you and ask you about like how the requirements from internal company projects and requirements from problem source projects are synchronized in your case because we are working all of us are working for a company and the company has a set of targets while the open source projects have a different target. So how is it working for you, for example, Alex? So yes, all of us are working on different products for different purposes and different companies. But we are trying to find a way to implement components in a common way and be useful for more and more use cases as possible. Sometimes it's not possible. As Alexander mentioned, we have a lot of use cases, a lot of hardware architectures and vendors. Also, we are trying to reach a maintainability of our products inside the company as a community core. So let's imagine we have rewrote the whole CPU manager policy, but we are still calling relevant functions in kiblet and still we are being invoked by our components. And once we are moving from one version to another version of Kubernetes, we have to adopt our code. And it's a huge maintainability burden since function prototypes and interface are changing continuously in upstream version. From my point of view, the way to keep the balance is developing sustainable interfaces. And these very different components might implement different companies or might implement different components as it was done for, for example, for device plugins or different trend times implementation. And all components will communicate with each other, maybe without a problem. So generally, I just wanted to jump in really fast on that question. Also, I think one of the important things here is that as long as the building blocks are there and out there and documented, it makes it easier for companies to come in that are looking for building their own solution around something like this and figuring out how they can or can't use what's out there to do it. So a good example of that was NFT that Swati mentioned, the node feature discovery. That was used heavily in our scheduler because we needed to leverage a whole bunch of the features that NFT had and we didn't want to build our own. But in addition to that, NFT allowed extensions that if you needed to add custom user data that maybe wasn't something that was built into NFT, but you had other pieces of topology awareness that you needed to add, you could. And then in turn, you could go back and upstream that if it seemed appropriate for other companies to use as well. So I think it just takes time and there's a lot of different requirements and use cases coming in. But as more and more come in, we're seeing all these pieces kind of come together in a way that is, I think, general enough where you can build pretty much anything off of this in the future. Thank you very much. And now, like we saw that there are lots of things happened already. There are lots of changes happened in these components and there are lots of changes still happening.  So what are the next steps, Sasha? Well, my simple answer to it, what's next means actually a lot of work. And as you mentioned, all of those components are still in development. There are a lot of things that can be done for the for scheduler part for enhancing the internal resource model inside Koblet. Obviously, a lot of work in runtime space, a lot of understanding how the future generations of the hardware will be affecting your workloads. And even with your workloads itself, you need to understand what like workloads itself evolve with. If a few years back, we're just running like with databases, today we have application plus site containers, like service measures and so on and so forth. All of this includes the implications of how resources needs to be managed. And from our perspective, even though like in the Koblet, we are trying to solve the most common problems and we are different plugin mechanism, we are giving you a flexibility.  And all of this information helps us to, first of all, to develop with things what we already planted, plan the new things for those components, define the API is better. And even by catching the box by trying what we already created, all of kind of with feedbacks for our work is really appreciated. And if you're going to contribute, it's even more appreciated. Yes, and looking to that, let me just list all the forums where you can connect to us, to the members of these groups. So first of all, there is SIG node, which is a Kubernetes SIG. And of course, you can access the SIG node discussions on the SIG node channel in Slack or in the regular meeting of the SIG node or in the Google Groups discussion forums of the SIG. Also, there is a topology over scheduling Slack channel in the Kubernetes Slack about this particular problem where you can discuss with us and try to help us. And also, there is on the CNCF level a tag for runtimes under the tag runtime channel of the of the CNCF Slack. So if anybody is interested in joining to this work, please join us because there is a lot of work to do and we need all the help that you can provide us. And with this, I would like to give you the opportunity to ask questions in the meeting platform. We are doing everything to be there and answer your questions. So thank you so much for joining our session. Thank you. Thank you. Thank you. Thank you very much.