Thank you all for joining. For those of you who are not familiar with US National Academies of Sciences, Engineering and Medicine, we're a private nonprofit institutions that provide independent objective analysis and advice to the US to solve complex problems and inform public policy decisions related to science, technology and medicine. Next slide please. Today is the first meeting of the committee on the evaluation of hydrodynamic modeling and implications for offshore wind development for the Nantucket Shoals region. This work is sponsored by the Bureau of Ocean Energy Management, commonly referred to as BOEM. The committee has been assembled to respond to the statement of task listed here, which you'll hear more about in a few minutes. The committee members were chosen for their expertise and experience and serve pro bono to carry out the study statement of task. Next slide please. To accomplish this work, the committee will hold four meetings for the purposes of public information gathering, deliberation, report writing in response to peer review. The results of the work will be a peer reviewed report representing the consensus view of the committee and that report is expected to be released to the public in the fall of 2023. There will be at least one additional public information gathering meeting in the next month, so please stay tuned as the study develops. All upcoming events are posted on our project website and are announced through the Ocean Studies Board listserv. So I'll add links to the chat in just a minute so that you have the links for both the project website and to join the listserv. And I think that's enough context to get us started. So I'd like to introduce the committee, you all to the committee chair will be facilitating the rest of this discussion. Dr. Eileen Hoffman will be chairing the committee. Eileen is a professor and eminent scholar in the Department of Ocean and Earth Sciences, and a member of the Center for Postal Physical Oceanography, both at Old Dominion University. Her research interests are in the areas of physical biological interactions and marine ecosystems, environmental control and transmission of marine diseases, and descriptive physical oceanography. Eileen has served on several committees at the National Academies and we're delighted to have her reading this committee. Eileen, the floor is yours. All right. Thank you Kelly. As Kelly said, I have the pleasure of chairing this committee. And it's also my pleasure to welcome you to this public session webinar. The purpose of this first public session is to make sure that the committee understands the context, intent and expectations of the work that we're going to carry out in the next few months. And to do this, we're going to start by hearing from our sponsor from Balm and also from the National Oceanic and Atmospheric Administration, NOAA, which is a federal agency that Balm is working closely with on the topic that is of interest to this committee. We're going to have two presentations from the sponsors. And after the two presentations, we'll have a question and answer session. If you have questions, please enter any comments or questions in the Q&A chat. We'll start with the Q&A session by taking questions from committee members. And as time allows, we'll also try to answer the questions that the audience interested into the Q&A. Whether we get to answer your questions today or not, a record of all the questions and comments will be shared with the committee for consideration. But before introducing our two speakers for this session, I'd first like to ask committee members to very briefly introduce themselves. And I'm going to call on the members in order here. Richard, would you please go first, please. Hi, I'm Richard Merrick, and I'm retired from NOAA. Formerly though, I was the Chief Scientist for NOAA Fisheries. And prior to that, I worked mostly in the Northeast Fisheries Science Center with both Rheem Amel and Fisheries Resources. Thank you. Thank you. Hi, I'm Councilor Gokumar. I'm a scientist at Integral Consulting based in Santa Cruz, California. Thank you. Jim. Hi, I'm Jim Chen. I'm Professor of Civil and Environmental Engineering and Marine Environmental Sciences at Northeastern University. Thank you, Josh. Hi, my name is Josh Kohut. I'm a professor of marine science at Rutgers, the State University of New Jersey. My interest is in physical oceanography as it relates to dynamic habitats in the coastal ocean. Aaron, please. Hi, I'm Aaron Meyer-Gutbrod. I'm an assistant professor at the School of the Earth, Ocean, and Environment at the University of South Carolina. Okay, thank you. And Doug, please. Yeah, thank you. Doug, now a check. I'm a professor at Duke University in the Division of Marine Science and Conservation at the Duke Marine Lab and also an electrical and computer engineering in the Pratt School of Engineering. And I work on bioacoustics and behavioral ecology of marine mammals. Okay, so that's our committee. More information about the committee if anyone's interested is on the committee website, which Kelly will send the link out to that for everyone. Okay, so now we'll move into presentations that we have, the two presentations we have scheduled for this session. The first presentation will be from Dr. Mary Boatman, who is the Environmental Studies Chief for the Office of Renewable Energy Programs for BOM. She's been actively involved in environmental studies programs since probably well over 20 years now. And she was recently on detail at the Office of Science and Technology Policy where she was looking at implementation of the National Ocean Policy. And she's now returned to BOM in her capacity as the Environmental Studies Chief for the Office of Renewable Energy Programs. Dr. Boatman is the sponsor contact from BOM for this committee. So Dr. Boatman, please go ahead. Thank you very much Eileen. I'm going to share my screen. First, can everybody hear me? Yes. Okay, great. And now can you see my screen? Yes, it's fine. Okay, great. Thank you all. Thank you very much. First and foremost, I want to thank all of the committee members for volunteering your time to this important question. I know that it's going to take some time and your time is very valuable and your advice is very valuable to us. Hang on one second. Okay. And I just want to, I have a couple of colleagues here with me. Thank you for that very kind introduction. I am a chemical oceanographer by training of a PhD from Texas A&M and I have worked for BOM for over 25 years with the Environmental Studies Program and the past since 2012 within the Renewable Energy Program, leading and coordinating our environmental studies, as well as leading some environmental assessments. And with me, I also have Dr. Yoko Furukawa, who has a PhD in Geosciences from Penn State, and she's worked at the Naval Research Laboratory as a marine geologist, and the Office of Naval Research Global as a Science Director. And she joined BOM relatively recently. She works in the Office of Environmental Programs. And then I also have with us Dr. Desiree Reeb. She's a multi-decadal experience in oil research with her particular expertise in the right whales, and she's been working for BOM for the past 10 years as a subject matter expert on marine mammals and sea turtles. And she's currently leading the BOM NOAA North Atlantic right whale and offshore wind strategy. And then I also have Dr. Thomas Kilpatrick, who is an oceanographer with the Environmental Studies Program, and he oversees our portfolio on physical and chemical oceanography. He has a PhD in oceanography from the University of Hawaii, and he was a postdoc at Scripps. Also where he worked on satellite oceanographic data and ocean models such as ROMS, just prior to his coming to BOM. So they're going to help, they're going to be here to help me answer some of the questions you will have once we get to that point. Just to give you our, a little bit about BOM, not everybody in the world knows who the Bureau of Ocean Energy Management is. And we're a bureau within the Department of the Interior. Our mission is to implement the Out of Continental Shelf Lands Act, which is about the resources, energy resources and mineral resources on the OCS, and providing a process for expeditious and orderly development. It's all subject to environmental safeguards. And in 2005, our responsibility had up until 2005 been focused on mostly oil and gas and marine minerals, and it got expanded to offshore renewable energy such as offshore wind in 2005. And because of the Energy Policy Act Congress told us to take it on. As you can see in the map, the OCS is about one, 2.7 billion acres, and it's all in the orange. So we have quite an area of jurisdiction for a relatively small bureau within the Department of the Interior. So that carries our statement of this issue, giving you some understanding of where we are today and why this has become an issue. BOM has leased areas off the coast of Rhode Island, Massachusetts for offshore wind development. A number of projects going on. The first lease that occurred was in 2013. We identified the area. We went through an extensive process with Task Force and working with the two respective states. We leased 0486, the light green and 0487, the tan color in 2013. And in 2015, we had another lease sale. We leased 0500, that blue one, as well as the dark green, 0501 was leased in 2015. And then in 2018, the other areas were leased. We currently have two projects in the area that have been approved, South Fork and Vineyard. They're both currently under construction that the cabling has been put in, and this summer they plan on installing the wind turbines. South Fork is in 0517, that kind of orange is colored one small area and then Vineyard wind being in 0501, which is that dark green area. So we expect turbines to be installed this summer. So we have four projects that are under review, and they have environmental impact statements that have been prepared or are being prepared. And this is Revolution Wind in 0486, which is that light green. The sunrise wind is 0487, is that again that light color kind of cream colored New England wind, which is 0534, which is the sort of brick red color, and South Coast wind 0521, which is the purple color. So these are under review right now. Again, this is why, and in the process of getting close to being approved. And this is why we have a sense of urgency with getting this information and understanding it. And then we also have two additional projects under review Bay State Wind, which is the blue, bright blue and beacon wind, which is that kind of khaki color. So we don't have any projects yet under the other last 0522 that we are directly reviewing at the moment. And as you can see, if you look very closely, you can see Nantucket Shoals is just to the east of this development area. And there's been concerned expressed about, if all of this development concern occurs, how will it affect the hydrodynamics. And there are of course two different ways the hydrodynamics can be affected. One is by the presence of the turbine structure in the water column and the flow of the current around the turbine structure. And the second is by the wind turbines, they extract kinetic energy from the wind as they're turning. Results in a wind wake downfield from it and that downfield wind wake has the possibility of affecting or altering surface currents and possibly stratification. So those are the two concerns that are being raised, particularly with this level of development in the area. Now, BOM has done some studies with respect to hydrodynamics in the area. In 2014, we funded a study with UMass Dartmouth or Dr. Chen. What was being addressed was, does the presence of turbines affect larval transport? And this is larval transport from Georgia's bank down to the mid-Atlantic. This particular modeling effort. And as you know, with all modeling efforts, it's partly in growing and learning structure. It considered oceanic flow around the structures. It did not consider that wind wake or effects on surface winds. They use the Northeast Coastal Ocean Forkast system with a nested finite volume coastal ocean model. They have the report. So if you want to read the details, they assess two extreme storm events. One was 1978 on North Easter and one was the hurricane in 1991. And that is how they ran the models for those two events and looked at how it affect the currents, how the turbines affected it. The parameters, they included 135 turbines in the Massachusetts area with a five kilometer spacing, and the tower diameter was five meters. And this will become important as I explain where we are today with all of these parameters. We did also then, after this particular study, fund a second one, another hydrodynamic modeling study in the area. We looked at particle tracking and agent based modeling was included. Again, it was to address larval transport from Georgia's bank to the mid-Atlantic. The, the scallop people are very concerned about it as well as other fisheries. The model included both the currents and some wind wake modeling within the wind farms. We looked at local and regional hydrodynamic data from 27 2018 and looked at again the impacts on circulation and how to affect sediment nutrient and larval transport. This time they did use, they used a one nautical mile by one nautical mile grid pattern, rather than five kilometers distance between turbines. We also used a hundred one thousand and sixty three turbines in the entire Massachusetts Rhode Island area. We had other subset scenarios of smaller just like just the vineyard wind development, for example. And they looked at 12 to 15 megawatt turbines with a hub height of 140 meters and a foundation diameter of 12 meters. We're also currently pursuing the same type of modeling for the mid-Atlantic from North Carolina to New York. We just awarded it in last August. The contract went to to we actually had two really good proposals and we were able to fund both of them. We did dhi waters doing one using a mic model rps group using a delft 3d model. So kind of an ensemble modeling looking at the same process but different modeling approaches. And we wanted to assess, you know, again, how the construction of multiple offshore wind turbines would affect the regional local and regional hydrodynamics. Same question as before. And as I say this is progressing, they're going to have three model segments, and they're going to be looking harder at wind, wake ocean circulation and particle tracking for their modeling types. And this is ongoing and we're hoping that your comments and input will be will feed into this. And I went back to, you know, I talked about this grid pattern within the Massachusetts Rhode Island area. This is a one nautical mile by one nautical mile grid pattern. The origins of this pattern are from actually the fishing in the area the fishermen are concerned about being able to fish or maneuver between the wind turbines. The agreement, a historical agreement between the gill net fishery and the trawl fishery, where gill net fisheries would lay their gill nets on lines that were one kilometer apart one medical mile part excuse me. And then the trawl fishing would be able to trawl between these two lines of gill nets. So it came the one nautical mile distancing came arose from that kind of gentlemen's agreement that was that was had the industry themselves the leaseholders and the other came up with this grid pattern boomed does not endorse this grid pattern. We do not recommend, you know, we don't say everybody has to use this grid pattern it was something the industry came and said they would like to use this grid pattern. The Coast Guard examined it for the ability for vessels to navigate between the turbines with it and agreed that it was made it navigable. And so we are working with this grid pattern both self work and vineyard wind that are going being installed are aligning with this grid pattern and where they are putting their turbines currently. So for modeling purposes, a one nautical mile grid pattern for spacing for the turbines. And then also, just to talk about where we've come with turbines, Cape Wind, which was where we were, you know, in 2010 2012, the tight size and type of facility. The turbine ratings was 3.6 megawatts with a hub height of 78.5 meters, wrote a diameter of 111 meters and tower diameter it's a 5.1 to 5.5, very similar to what was modeled in the first study, Europe. We're looking at five megawatt turbines 90 meter hub height, 126 meter rotor diameter with a six meter tower diameter. There's been some extensive modeling in the North Sea that has used this type of parameter. Again, five megawatts dictates you know how many turbines you're going to be having the area. A lot of Europe there's over 4000 wind turbines operating in Europe, wind turbine structures in the North Sea and around England, etc. Most of them, if not all of them are of this size or smaller they're in the 3.6 to five megawatt or even smaller, the early generation. So here, Block Island wind farm is a six megawatt. The Seabild pilot project is also six megawatt, and we're moving right into 11 to 15 megawatt size turbines. South Fork will be 11 megawatts vineyard I think will be 12 megawatts. Hub heights are going to be on the range of 130 to 160 meters. The rotor diameter is going to be around 200 to 240, and the tower diameter is 11 to 12 meters. So these are significantly larger, but then of course we'll be putting in fewer to achieve the same amount of power output. So I just wanted to express that when you're examining looking thinking about modeling and everything you need to think about what type of a facility they're modeling and where we are going in the United States. And then, as I said we're also going through some environmental reviews of particular interest is the south coast. We have also funded with Massachusetts Clean Energy Center. We funded the New England Aquarium to do marine mammals surveys, as well as to do some collections of plankton in the area. We started those surveys in 2011 and 2016-2017 we also did collections of phytoplankton, obviously as a food source in the area we can make these reports obviously available to you if we haven't already. So we're just here. And looking at, you know, we did look at important calendar species, and you can check out that report to understand, you know, what we have done so far with respect to the types of calendars and other species that are in the region, and the amounts abundance. And then also, then lastly, I wanted to talk about that we are participating with NOAA as a partner in this North Atlantic right wheel strategy and offshore wind strategy. And your efforts will feed into this strategy. And most importantly, one of the draft goals of it is to evaluate and address any uncertainties associated with the effects of offshore wind development, which includes these hydrodynamic effects. So it's very important what you're doing and feeding into how we proceed into the future with offshore wind. And lastly, what is Bohm's expectations from the committee and what we would really like to see. And first and foremost, we want to ensure that Bohm is using the best available science and appropriately interpreting and applying models to inform our impact assessments. And then we're also concerned and interested in what the scale of effects are relative to natural variability. And I'm a chemist, so I also want to understand the ability to measure the effects. And then also provide us with recommendations for model validation. As you all well know, you can create models and models will tell you here's the stories of what will occur, but if they aren't validated with real world observations, they can only answer so many questions and there's some questions they can't answer. So thank you very much and we will look forward to working with you and hearing what those recommendations will be. And with that, I'm going to stop sharing my screen. All right. Thank you very much. That was a very informative and interesting presentation. We'll hold questions until after our second presentation here. That will be from Dr. Sean Hayes, who is the Noah Fisheries Chief of Protected Species. Dr. Hayes has a PhD from the University of California, Santa Cruz, where he studied marine mammal communication, physiology and reproductive behavior for large cetaceans and penipets. And he is working with Dr. Boatman and other groups on issues related to offshore wind energy development. So Dr. Hayes, please go ahead. I think you're still on mute. There. Can you hear me now? Yes, now we can hear you. Thank you. And you see the screen. And your screen is fine. Yes. Okay, you can see the presentation. All right. I apologize for that. I had the unmuted button hidden under another screen, so thank you for your patience and taking the time to watch this. I'm going to focus primarily on the biology of right whale as a reference point to the issues that we're considering here for this presentation. And starting with that is kind of the irony that the harvesting of whales for oil in many ways initiated the industrial revolution with the provision of lighting for factories and lubrication for machines that drove the need for more energy and the evolution of our petroleum based economy. And it's the sad irony that has led to the massive climate change challenges we're facing today. And as part of the multi purpose effort and Mary just went through some of this is to combat climate change by transitioning our energy portfolio to renewable resources and increase our energy security the Biden administration has announced a new goal to produce 30 gigawatts of offshore wind by 2030. And then ultimately to 110 gigawatts by 2050. The 30 30 commitment is really targeted to development as we've seen in the southern New England and mid Atlantic regions. And this region is home to the North Atlantic right well where it carries a mixture carries out a mixture of foraging breeding and calf rearing activities for much of the year. Pictured here is the current demographic history for North Atlantic right whales. And after 20 years of slow recovery peaking at just over 480 animals in 2011 population is declined by 30% to only 340 animals known to be alive at the start of 2021. And as a rate issues females have lower survival rates than males and there's only 143 females alive, and of those less than 70 are known to be a reproductively active. The precision of our population model and the recite rate allows us to estimate total mortality beyond just what is seen on an annual basis and from 220 10 to 2040, just under 240 whales have died. This is a significant event of the population at its peak 11 years ago for a species where individuals should be able to live up to a century or more. Population size is a function of birth minus deaths and sadly birth rates are also down and well below the rate needed to compensate for current mortality rates. So why is this happening. Prior to 2010 whales were experiencing a favorable ecosystem and regional mesh management efforts were sufficient for recovery to occur. There was significant ecosystem changes in the Western Atlantic that triggered changes and where whales went to forage including more time spent in Canadian waters during the summer, and then tuck it shoals for much of the year. Because management strategies tend to be regionally focused whales were suddenly re exposed to old threats but in new areas were management efforts were not yet focused on protecting them. The ecosystem changes provided additional stress in the form of reduced food resources and traditional foraging areas and time energy spent seeking forage. For entanglement and vessel strike these issues are so pervasive in the species that we have yet to document natural mortality and an adult right whale. And the repeated stress of sub lethal entanglement and changing food resources is likely contributing to a reduction in calving. Unlike their Balanopteran cousins which are capable of rapidly moving through the water to pursue a variety of prey from plankton to fish with lunch feeding strategies right whales are slow moving ran filter feeders. They basically just open their mouth and plow continuously like a bulldozer, which is energetically very expensive and dependent on getting enough plankton to grow. In a sense, these animals are specialists and they have very little resilience to change. While there are some evidence of alternative food choices right whales feed almost exclusively on Kalanoid cobalt pods and they do so with a simple equation that the energy content of the plankton they consume be greater than the energy expected expended in searching for it. For plankton four variables affect this issue. Obviously abundance just how many plankton there are in a foraging area. Distribution, how far are populations or group aggregations of plankton spread across their the ocean habitat and then density local density and aggregation plankton need to be able to overcome any mixing and turbulence in the water column to become aggregated and dense enough layers to offset whale swimming costs. And then finally energy content plankton store energy and lipid bubble pictured here on the dorsal surface and as ectothermic species their metabolism changes with temperature and there's concern plankton energy content will be lower under warming ocean conditions. Citing one of the panel members work in 2010 the Northwest Atlantic experience massive regime shift forced by climate driven changes to the Gulf Stream warm slope waters entered the region reducing Kalanus abundance. Soon after this right whale shifted their late summer foraging grounds from the Gulf of Maine to Canadian waters. Overall calving rate has declined. As whales move to the Gulf of St. Lawrence, it has been shown that the females using that habitat are experiencing greater calving success in comparison to females using other habitats. But the redistribution in general has come with a high price and significant increases in mortality associated with bestial strike and fishing entanglement that's outpacing the rate of calving overall. Currently 30% of the population shows signs of fresh entanglement injury every single year. At that rate, basically every individual in the population is likely to come entangled, at least once if not multiple times within a decade. The consequences of this are two fold. In any given event, capital breeding female is likely to have to spend longer time accumulating resources to successfully calve. Secondly, it's been shown that size and maturity is effectively shrinking for those animals experiencing entanglement and their associated reproductive success is less. The net effect is that for those animals that aren't killed, which is typically before they reach the age of 40 now, their calving interval has slipped from sometimes less than three, four years to over eight years on average. And the approximate lifetime calving potential for a female has dropped from something greater than 15 calves to less than four animals. As part of a recovery efforts, NIMPS created a multidisciplinary and multi-agency team to develop a population evaluation tool that can be used for population viability analysis. The tool bends threats to right whales into four categories and those same categories apply to all of the foreseen issues of offshore wind. We will consider all of these threats through all phases of construction operation and even decommissioning. NIMPS will be evaluating these threats at the project level as well as the cumulative impacts across all projects, but for the purposes of this talk, we'll focus on the ecosystem changes circled in orange on this slide. What are the consequences to how wind will impact regional oceanography and plankton production? The concern is simple. Oceanography is driven by the transfer of energy from air to water, and we're going to alter that process in some unknown way. The results and implication for right well recovery is very uncertain as a result. Potential impacts of offshore wind are both atmospheric and subsurface. Starting with the first, we know kinetic energy is both absorbed and redistributed elsewhere, sometimes more than 100 kilometers away. We know that air stacks up in front of wind fields, creating high local high pressure zones on the ocean surface, and then pushes up and over creating low pressure areas in the middle of the wind field. This has local effects on lateral and vertical water column movements and subsequent impacts to stratification, including temperature, salinity and potentially nutrients. Subsurface change is anticipated in two forms. The oceanographic response to the atmospheric changes described on the previous slide and the physical effects of turbine structure in the water column. Most work to date is theoretical and modeled with little empirical results so far. That said, there are expected to be local effects that increase turbulence and associated potential for increased nutrient flux and chlorophyll production, which could contribute positively to increased plankton production. However, numerical modeling efforts for the North Sea predicted large scale changes in annual primary production with local changes throughout the North Sea of up to 10% or throughout the greater North Sea region when modeled for a range of wind field production. And this had associated fluctuations in stratification, increased carbon sedimentation and decreased dissolved oxygen. This brings us to Nantucket-Scholes and right whales. This is a very different geographical area and very different species complex than the North Sea, with an ecosystem that is currently in a state of flux likely driven in part, or if not a lot, by climate change. However, both the laws of physics and Murphy still apply in this region. Of course, our knowledge of right whales use in this region when lease areas occurred a decade ago were pretty limited, although we are fairly confident that the use was less than it is today. That has changed a great deal. And now this region has become a primary winter foraging habitat, and the only habitat that they are known to aggregate in for foraging for several months of the year. NIMPS has been working with colleagues from Duke University to update their habitat based sedation density models. The model incorporates sightings data from non-directed survey efforts and uses that to build a series of regionally calibrated models which estimate density of right whales on a monthly basis. Because right whale distribution has changed so drastically, Duke has restricted the time period of the model to just those most recent years starting with 2010. Pictured here on the left is the model version 11 output, including nine years of recent sightings data. The middle pane is version 12 of the model updated with just adding one more year of sightings data through 2020. In the right hand panel measures the anomaly of just adding one more year of data with red being positive increase in density and blues being negative or a decrease in density. Clicking and hopefully an animation will occur. Animals showing all 12 months. Note that the peak increase in habitat shown in red occurred on Nantucket shoals in all 12 months. Our raw sightings data since 2020 show this trend of increasing use in the Nantucket shoals region is still occurring since the model update with signals of more year round use occurring particularly in August through October. This is visual for the version 13 update of the monolith still pending. NIMS managers have provided the following questions for the panel to consider and given time I won't read all of them but they can be viewed during discussion. And I'll close with just acknowledging most if not all of us are conflicted about wanting more time to work through this process. Climate change and the most recent reports that we might exceed 1.5 C in less than a decade are not going to wait. And we must acknowledge that climate change is this greater threat to right bails and the rest of the ocean ecosystem as it is to humans. The agency understands that any effective solution to climate change at this point will have consequences both economic and ecological. Hopefully the insights provided by this with the insights provided by this panel, we can make the most informed decisions possible to meet our national energy security and carbon reduction goals while ensuring minimum impact to the marine environment. Thanks and I'll stop sharing as well. All right, thank you very much. Again, a very interesting thought provoking presentation. Okay, so at this point we have about maybe 15 to 20 minutes for questions. So I'd like to open this up to committee members to see. Ask if you have any questions and Richard. Yes, please go ahead. So this is both Mary and Sean. And going back to the statement of task. In the title of the project. The implication there is that you want us to focus on hydrodynamic modeling. But there's a hint in the second bullet, the based on a literature review and then the comment underneath that that you're interested in the potential changes to ecosystem dynamics. And then Sean's penultimate slide lists the whole series of nymphs questions about the effects of WTG's. So we're a little bit confused as to where you want us to draw the line on just focusing on the physical effects of hydrodynamics and the ecosystem effects. Can you give us some guidance on that. Mary, if you want to go first. So, I'm thinking about it, sort of from a NEPA standpoint you always start with the impact producing to me the impact producing factor which is the wind presence of the wind turbines affecting the, the environment, the hydrodynamic environment most of the hydrodynamic, and then I see that as the primary effect with the secondary being affecting the ecosystem and then a tertiary being affecting any of the going up the food chain, etc. So we really want to focus in on first understanding that we have a good handle on the hydrodynamic aspects and the modeling and and identifying what those impact reducing factors are. And where does it how far out does it extend etc that's really important, and then does it does it impact or how significantly or measurably does it impact the hydrodynamics, and then go from there to yes, if, if, you know, kind of a, if, if yes it's quite it's significant is that altering or affecting enough to affect the, say the distribution of larvae or plankton etc. And then, if, if, and then you go on to okay then that can lead to the final what I call the tertiary effects on those that live on it and I don't know if that answers your question but our real focus is on the first part, you know, and then, and then it can carry on to the second and then on to the third, and that's the way I look at it anyway. And I don't know if that helped or not. I'll let Sean respond as well perhaps he can enlighten us. Yeah, thanks Richard, I guess, I mean, you've been through this process you know that when it comes into management, you get into splitting hairs about the specifics of the questions so the managers really work to kind of provide, you know, all of the potential iterations of split hair questions that we envision needing to be able to address and in our sort of permitting and review process. And I think for the agency, it really comes down to the last seconds in the first clear bullet under the statement of task is, you know, specifically for assessing whether these facilities could substantially affect North Atlantic right well pray availability near Nantucket sholes. I think that's the bottom line that names really needs addressed. So that to me would suggest that there needs to be another sub bullet there. On the I'm right reading right off the statement of tasks. I know, but okay. It really still starts off sounding like you really want to focus on the first effect, which is understandable. Okay. Richard, do you have any other follow up or. Okay, I'm still I'm still at a loss of how we're going to draw this but the line there but we can talk about this within the group. Right. I think it's a very important point and when we need to put some time into. Okay.  Please go ahead. Yes, yeah, yeah. It's a question also for for Mary, Mary Sean and possibly Tom Patrick as well. I was wondering just just how this statement of tasks relates to the RFP that bone release last year. Like the tasks and the questions and the recommendations in the model seem very similar to what came out in that in the previous RFP last year. And, and, you know, if they're unrelated or related or what would one one is motivated by the other. I guess a follow on question would be is would be is, would we have access to any reports or, or ongoing, you know, publications produced by, by, by, by whoever is working on that project that came out of the RFP. Okay, this is Mary so I'll start with, of course, all this whole issue of hydrodynamics and such is related to that that RFP from last summer which I assume is referring to that third study that I talked about which is extending our understanding of hydrodynamics down to the mid Atlantic and also improving our modeling. But then there's also challenges as with any modeling is that how you interpret those models results and that's one of our specific ask within this assessing, you know, whether I really assessing the models and, and looking at how they can be interpreted as well as how they can be improved. And, and then in terms of whether there's going to be some interim things I'm going to turn it to, to Tom, Dr. Kilpatrick to answer more about that specific study and what's being accomplished with it in relation to what we're doing here. So Tom, if you could take it. So the two studies, those are looking at kind of the adjacent region like the next sort of block south. I don't believe they're the least the focus was not meant to be revisiting that Nantucket Shoals area. So it's a little bit different region to the south. And then I don't think that at least complete results will be available in time to inform this committee's work unfortunately looking like more like late summer they'll start the analysis that would be DHI and RPS for those two, for those two studies. And in terms of the science I would say, like for instance in that say we did not include biogeochemical modeling they don't have this MPZ type of model and either of those, which we're hoping to get for this committee that would be one specific point that we're hoping to include. So the science part would be a little bit broader than the scope of what we had for those two things in the solicitation from last year, but the region and kind of thinking for this panel's work would be focused on Nantucket Shoals rather than that broader Atlantic area. Thank you. Sure. Thank you. Any other follow up? Doug, I think you have your hand up as well. Yeah, thanks Eileen. I guess, just trying to keep things in the context of what's already been asked I think, especially in the timing and what what Tom just told us in terms of these other, these other studies. And I kind of wonder where, I guess it's really a NIMPS question in terms of granting IHAs and permits and that sort of thing is where the Fisheries Service is with respect to its position and how it's handling what is the NIMPS approach right now without results from this panel as well as from the modeling studies that are out there. And this also certainly involves the spatial scope that we were that we were just discussing if we're really only focused on Nantucket Shoals, how will you approach the broader region. And other regions too, like New York and the New Jersey coastlines. So, the basic question is how, what is NIMPS position right now and how are you approaching the ongoing review and permitting requests for offshore wind development. Thanks Doug. I'll start by, you know, sort of putting clearly stating I wear the science hat and NIMPS and not the policy hat. So caveat to anything else I say beyond that. So our policy folks are working through, you know, the projects as they come to them on a case by case basis. And, and so that's part of it. And then Tucket Shoals ones are the first ones that we're really having to figure out how to how to permit and process permits. And it, and it is, you know, in some respects in terms of right well density, they're kind of the biggest, they create the biggest challenge for us in terms of sheer density of animals relative to the field I think the threats are likely to vary from the wind field, the New York, New Jersey ones, probably are going to be potentially dealing with more vessel traffic issues as part of just cumulatively in the area, which isn't necessarily in field but just to give a hypothetical that, you know, threats are going to vary from each region. And also, you know, there may be less of an impact in the more southern ones from from foraging ecology issues in the moist, and it may just be different noises so we have to sort of conceptually work through each of the various threats and that that was a slide ahead earlier presentation of we've really broken down threats for right whales into, you know, forming categories and it's basically, you know, in tank fishing gear entanglement vessel strike climate and ecosystem effects and noise and and that's our PVA sort of works to that so from the science side, you know, in any given project will take the project as described and, and sort of evaluate each of those particular threats, you know independently, as well as cumulatively some of them obviously have, you know, direct mechanistic impacts you know vessel strike is obviously a lethal threat. Other things tend to be more sub lethal. And so, you know, we wait those things, but I don't want to go too far here out into the policy lane. No, I appreciate that, John. And I wasn't, I didn't mean to frame it as a policy thing.  What it is and that's that's really what I'm asking. Well, I mean, and obviously there's, there's projects that have already been approved. There's projects that, you know, the process would like to have it approved before this panel was done and then there's, you know, a queue of 20 or more projects waiting to be approved. And so, you know, some of the stuffs, you know, water under the bridge and other projects are, you know, hopefully we can take the results of this panel to help inform us and bring a really independent, you know, insight into what is doing this I would sleep a lot easier if this came back and said, you know, we think that right wells are going to, you know, right wells have their own challenges but we don't think this is going to necessarily add a lot to them and I would feel a lot better about that conclusion. If we heard that from this panel and that would definitely feed into how science, you know, the science of the these results will, we'll wait heavily into how we advise the managers as a hypothetical, you know, but if it comes in that there's serious concerns and that'll, that'll weigh in as well. Yeah. Okay, thanks. Thanks, Eileen. Yeah, thank you. Very interesting comment. Jim, you have your hand up please. Yes, Jim Chung here. So, so I look at the task here, number two actually quite specific, you know, try to evaluate the applicability of existing models. So I have a question, I heard Mary and Chung's presentation that there are a number of studies and looks like that they are not available for us to review. My question is that do you expect us to also run models of the panel to sustain the effects or we just do a literature review and give you provide kind of expert opinions on those models and also the potential effects on the ecosystem. And this is Mary, I'll take that. So I showed you three slides with models, the first two reports are available for you to look at and we have been using those in in our evaluations and but like with all models, they can be interpreted differently. There's also some modeling that's been done in Europe where it's really available and we've made available the peer review, peer review publications with respect to those. So it's, we don't expect you to run any kind of models we had this discussion when we were formulating it. It's really taking a look at the modeling that's been done that's being used and whether and how best appropriately to use it as well as what can we do to validate it from a, we can some follow on study there's always recommendations for a follow on study for how to validate it by actually making measurements, if it's possible, because potentially if the effects I read one paper where they said that down when wake the effects were so we're within the very with when your ability to measure it. So, just because you can model it and come up with a number does that number is that number, meaningful in in the context of the whole environment and the natural environment. So we're really looking for your expertise and understanding of models and understanding of the assumptions that go into models and then understanding about how these models can be interpreted to inform this question about whether we are really affecting and we're focusing on the models. And those two, the, the studies that we have that aren't available to you, they are totally focused on the mid Atlantic from New York to North Carolina. So they're not going to be helpful with the shows per se. However, what you recommend and what your thoughts are about the hydride dynamic modeling, and the interpretations and how it's interpreted and what it means on an ecosystem level can be taken away and used to improve and validate those models as well down the road, but you're don't need, I don't think you need to be looking at things that are outside of the Nantucket Shoals area that we are working currently working on. And that's my view, but we're always happy to share whatever we can with you. Thank you. That's helpful. Yes, thank you. Is your hand still up or do you have another question? Sorry, I forgot to lower my hand. Okay, thank you. All right. So Richard, do you have a question here. So Sean, what you said really helps you understand the statement of tasks much better. A priority, you've got to believe there's going to have some effect on the ecosystem. So for us really to comment, there's going to have to be some part of this study that actually looks at how we couple the hydrodynamic modeling with the ecosystem modeling for nutrient distribution or primary and secondary productivity. I'm not sure how far we can go, but do you think that's a reasonable part of this, this panel's deliberations? Yes. Like, I think that that captures it. Okay. And that's important because it actually says something about who should be on the panel or whether we're missing something. So thank you. Mary, you're okay with that too. Absolutely. Okay. Thanks. That's very helpful. Thank you. Doug, you had one more question here. Just quickly, Eileen. So just follow on Richard's there just now. Then that also, we also have to include the sort of aggregating effects or disturbance of those aggregating effects, which may or may not involve actual production, but the physical aggregation of prey. Would you say Richard? Yeah. And then the other thing about the European side of things, I see some things in the Q&A about the size of turbines and things like that. But as Sean raised that these are a lot of modeling studies, but there are some actual empirical measurements from those areas that are going to be critical for us to consider. So we are well aware of the need for input from that theater, including empirical measurements. So I just wanted to make that point, Eileen. Thanks. Okay. Thank you for doing that. One of the things that this committee will do is take the advice from the presentations we've heard here and from the discussion and reconsider or think, not reconsider, but to look at what expertise is missing and do our best to somehow address those missing links and the missing expertise, either through committee members or through information gathering, having experts provide presentations. Are there any other comments? Okay, we have only about two or three minutes left and I just wanted to ask the, to Mary and Sean, you've made comments here about what would happen with the results from the result, the report results that we're going to provide you. But I would like to understand that this report from this committee will in fact be used to guide future research to fill in missing pieces and just talk about here is, is that correct? I mean, there will be an implementation of this report, I would hope, somehow. This is Mary. I'm going to give the standard government answer, which is, yes, subject to availability of funding. Right. Yeah. All right. I guess that's a fair answer. All right, thank you. All right. Are there any other comments from the committee questions? All right, then what I think I'll do here is I'll thank Dr. Boatman and Dr. Hayes for their presentations and for taking the time from what is sure to be their very busy schedules to participate in today's committee meeting. We're sure to be coming back to them in the next few months with questions and requests for inputs. I want to thank the audience for listening and for the comments that you've provided in the Q&A, the committee will look at all of those and their questions will be considered and where needed we'll see about providing some responses. So I also encourage the audience to look at the information on the committee website. It will be updated as the committee makes progress and continues to hold meetings and look at public meetings such as this from all that information will be available on the committee website. So at this point, I'm going to close the open session and the committee meeting. And again, many thanks to the speakers, the participants, the audience, and then the committee members will meet again later in closed session. So thank you all and we'll look forward to continued interactions. Thank you.


Right good afternoon everyone and and welcome to this room for the afternoon Considering the topic of the talk we have as the first talk for this afternoon I think it's a good idea to start on time. So without further ado, please welcome Daniel for RTLA timer lat. Thank you Thank you. So I'm Daniel. I've worked for ready head for quite a while and I mostly work on the Upstream on kernel and things related to real-time runtime verification and scheduling and so on and so This is about a tool that is inside the kernel the set of kernel tools that is time time or not as a tool inside RTLA so Here we have been talking a lot about the real-time Linux Even if beyond presentation this morning So Linux has been used as a real-time operating system, right? It's a fact many people might say it's not a real-time operating system because of of It's a large operating system because of the general proposed nature, but Linux is able to deliver Reasonable good timing behavior And and many people are looking for Linux as a real-time operating system option Because the the stack of software is increasing people are using AI stack and video processing all these accelerators and And and these technologies have been developed using Linux as the default operating system So there's a huge advantage of continuing using Linux on these setups So people look at Linux as a viable option as a RTOS also because there are many people that understand it hopefully and And I say them also because Linux they choose the desire timing behaviors And and some of the key features for this is like the fully primitive mode Which is the preempt RT kernel the real-time scheduling locking and so on One of the problems however When we try to say that Linux is a real-time operating system is the way that we show these these properties Right because Linux has been tested using a black box Like method and when you start talking with people that run real-time systems on critical systems They might say it okay black boxes It's it gives me that the number but it doesn't gives me why that number happened And how I can debug it and how can I explain if we if that's a bad scenario or not? so The way that these black box tools work like cycletest is that they they are a timer in the future using an external clock reference and then when this timer happened the the High the highest priority thread is awakened and when this thread starts running it reads the time again and measures these latents That's the base idea. So the base can do we cannot say that this approach doesn't works, right? Because we have been developing the different RT doing things this way and the results are good. So yeah It it works. It's just that it gives no root cause analysis. It doesn't and doesn't And it doesn't do that that step to that next step, right and generally for us when we are working In the kernel when we see a latence what we do is we use we use tracing Features right we go there. I've been working this for quite a while I already know by by the top of my head. What are the events that I need to enable? I just say okay enable this this event and then give me the trace and I will try to analyze it and But but trace is not always accessible for for let's say the non-experts like my project manager Right like my manager and when they see a new bug zealot redhead say, oh, there is this latence and this number, right? Do you always need to get someone some expert involvement to try to to debug the problem and also after their 10 years? Doing this kind of repetitive ritual one gets annoyed, right? I was annoyed red. It's all do these do that and do that. Oh, yeah, again the same same The same ritual So who cares about this anyway? So other than the poor dude doing tracing like me other than me who cares? so one of the point is that there is a I would say even incredible for for if you look like 10 years ago If we we think the number of use the case that we have for real time nowadays and the request for red hat and for for the upstream It's impressive Many people are using real-time Linux like in telecommunications and now in the elisa project for safety critical So there is a large demand for for the real-time kernel and And as as the kernel gets merged there's also a fact that All the kernel developers will have to start testing for regressions On the real-time kernel because the real-time kernel will be part of the regular kernel so but not not not all of them are interested on on Learning the details of the kernel the real-time kernel and synchronization, right? If I am the video for Linux Maintainer I and there is a regression. I will talk to someone that is expert on real-time not necessarily The expert on another field in the kernel the kernel is huge And also that there is a lot of projects as I said that that That required is real-time like automotive and automation to name a few telecommunications and so on So what is timer lot? The RTLA Timer lot is a new approach for that So the RTLA is actually a test suite that is inside the kernel source that has a lot of tools inside Timer lot is one tool inside of RTLA So RTLA is a new approach that tries to to integrate all the bits and pieces that I use on the bugging that we use on the bugging So in in one side we have in the kernel side. We have a tracer that is the timer lot tracer that contains the It does the internal processing for for the timer and Adds a set of OSNOS trace points that we'll see later that does most of the processing in kernel Or some of the processing in kernel to reduce the overhead on tracing and try to write just Processed information to the trace buffer and it uses lockless synchronization to avoid like false positives and so on And and it also has an in kernel workload that simulates that a black box Tooling and if you are interested only details of the lock less synchronization and the noise reduction There is a paper published at the IEEE transactions on computers that is about another tracer This is a sibling tracer that is the OSNOS tracer, but the trace points are the same and the idea is the same So you can find a lot of information here and so That's the the kernel part and we can use it using the tracer fast interface Together here, I say I put LibitraceFS So I use the LibitraceFS to glue this kernel part with a user space side That that that gives us a tool that's timer lot that tries to bring a Tries to give to that tracer a benchmark like interface trying to make it easy to use It sets up the trace it collects the data press the data and gives us some output that I'll show later And it also allows us to have some auto analysis for bad values And I'll discuss this also later and gives the possibility of running the Workloading user space that's something that's that's hopefully you show up in this near future on kernel so Before starting There is one thing because I'm in kernel. I'm controlling the timer The timer not the it already breaks that latency metric into two metrics So it has a special IRQ handler that is activated and when it's activated I measure the The IRQ latency These IRQ we wake up the tread and then the tread latest will be printed later So instead of one we started breaking it up But instead of have one metric it shows already two metrics and it so Here is one example of the two output This is my system like five years ago. I'm running. It's a regular workstation. It has 24 CPUs I'm running one on I'm running kernel compiling on all the CPUs That's the this terminal here and here. I'm trying to use the tool So RTA timer not top will give a top like interface that will show me the latency that my system is facing now So here I have the the side the CPUs the amount how many cycles My currently IRQ latency the minimum the average latency the maximum latency for the IRQ That wakes up the tread and here's my current to the minimum the average latency in the maximum latency So it's already giving some some two-stage information here. This is the top Interface there is the histogram interface That gives us a histogram of the system. It does the same thing but instead of keep displaying it runs in silence and It collects the histograms and when I either hit control C or say run this for one hour at the end it brings a summary of the output and Here it is. So let me just pause here. Maybe it's better So here's the latency and how many occurrences I had on that latency. So as you can see we can see here This is the IRQ latency. It's shorter. It happens more close to this time This is the tread latency. It goes longer and it it spreads more So this is useful for for analysis or for regression analysis There's a lot of options for me to format here to make it easier to to be parsed by by scripts So this is just to show an idea of what is the test What happens is that when we are testing the system? We have we generally have a maximum acceptable latency or a threshold and if that threshold is it's generally in microseconds, right? So and when this threshold is a Latest occurrence it goes Beyond the threshold it's interesting for us to try to understand why it happens So why my system is not within my limits, right? So time or not allows us to set a variable On the command line that says okay if if a large latency happens, please stop everything Then I can say oh if the tread latency was higher than a hundred stop or if IRQ late for higher the than 100 stop Or there is a magic option that is the dash A that enables all the default Things that an expert would enable just an easy shot So dash A 100 it will enable if you tread if IRQ get the stack trace enable the tracing and so on I use it a lot. I just put my defaults there and hopefully people will like it suggestions are well So when the system stops time or not not only stops tracing and collected trace It also prints an auto analysis of the root cause for that for that spy Daniel from the past was faster than Daniel from now So I'm saying that's a 30 is top if there's a latency higher than 30 bro and here the tracing is stopped and And higher we can see there is okay an expert would say okay There's this IRQ disabled that's made the problem But you can see here that there is now a variable. That's IRQ latency delay IRQ latency the duration of my IRQ which was the task that was running and this test running was this one and this a suspicious call and so on and Don't worry. I will explain this filter that that's the subject of the talk It's explaining how to read this report. So and my latency was this one and and And it gives this Hopefully easy to read the output are as easy as we can get for now. So Next Auto analysis, how do I analyze the auto analysis output? How do I read it? So the auto analysis tries to decompose the latency into a set of variables Not one number not two numbers, but a series of numbers somehow independent that compose the latency and Each of these values can be analyzed them mostly independently And and when we see like on the IRQ latency or a tread latency They have different ways to be analyzed because an IRQ has a different priority or follow different rules than threads, right and And the auto and it's important to say the auto now is here Even though it's probably most people using print RT we use these two It works for our kernel for the no primitive as well. There is even one example. So it's generic I try to abstract the Concepts to to make it as generic as possible So the timer latitude some abstractions that we use from that have on RT theory I like the RT theory many people doesn't but I do and So we have there the execution time, which is the time to accomplish a task For example, if my RQ handler executes for 10 microseconds and I need it to accomplish my task This is the execution time of it Blocking happens when a lower priority task Post opponents the execution of the higher higher priority task. That's blocking The interference is the contrary is when a higher priority tread prems a lower priority tread. So blocking low priority Interference high priority When we think on tasks on Linux, we generally think on threads, right? But if we look in the internals of the kernel In turning the current it's easy to see that we have like four classes of Tasks if you want to see the prems like how many prems we can have inside the trace buffer We have these four abstractions here one thing that can prep the other So the rule is somehow like this we have NMI's that can prompt every every kind of task and Then cannot be prems it we have IRQ's that prems all but NMI's And we have software cues that can be prems it from IRQ's and NMI's but cannot be prems it by tasks and Treads that can prems to themselves but cannot prems to the others if we read backwards We can see threads can cause blocking to software cues software cues and threads can cause blocking to IRQ's And nothing can block NMI. Sorry Other than the older NMI's Sorry or bugs. Yeah bugs kids And and I don't like to hear bugs on NMI code Like NMI prems in NMI's never good So let's get one example here Let's get one example here. Now an IRQ latency examples. Let's focus on the first part first So I set my latency to 30. It stopped on the 50 on threads and yeah Oh, I also draw a timeline here to make it easier to visualize So the IRQ handler here the IRQ handler, it's it had 31 microseconds delay from the starting So this was the time where the the hardware interrupt was set to run And it took 31 microseconds until it started running And the IRQ latency was 32.17 microseconds boom And that that was the fastest time I could read the timer and report it The blocking thread that is the thread that could have postponed at it. It was this object tool and Here is the this part in It's a it's the IRQ handler itself. So just from from here is the IRQ handler from here is the task So we can see here that there is this function IRQ restore that re-enables the interrupts So it makes sense. There was a thread that disabled interrupts That's why my interrupt was postponed and then we can see here that it was inside the butterfly fast and a VFS right and Then I see here why why is this C group or our stat flushed lock at Call it the row spin lock, right? I can just do a git blame on kernel and try to figure out the patch In this case it's a legitimate use case because the person sending is is the prem 30 maintainer Sebastian, right? So there is a reason for it to be there What one can do is try to optimize the code inside there to reduce it or try to make this workload not run on that CPU that I care But this is a legit case and I could connect the code with the analysis And also we learned that these use case has the signature if there is another bug with the signature My PM can say this is mouse like that the other bug duplicate. Hey, let's work for them Well, that's not a point in time. That's interesting to have less work, but that's another discussion Another one other thing that he mentioned on his presentation about having the the is the X from idle latency, right? On and here you I have one example of exit from idle latency In the theory. There's these release a jitter term. That's basically this it's So I RQ delayed there the RQ can be also delayed from hardware because of factors in harder And I'll talk more about it later But there is this special case when when the system doesn't have like exit from idle tuned And there is also he mentioned the tune the profiles that Fred had that can help on that But here's a system that the the C states aren't configured the latency on the RQ handler He took like a 39 microseconds The system is running the swap or tread which is idle and the analysis are at say, okay This is exit from idle This is the problem here You need to check this and it also mentioned okay over the analysis I also saw that my largest delay was this amount Exed from a idolite latency Even if the cause that made me system stop wasn't this it would print at the end because that's a common limitation So and the tab where that has a option to that same sector that has an option to work around it But the real thing that one needs to do is is set up the C states So going further one example of tread latency Here it's harder to get tread tread delay examples on prem 13 because it's running well So I went to the regular kernel and that's why I took the regular kernel But I can say also well, I just wanted to show that it also works on the no no preemptive kernel so Here we have an example where the the IRQ handler delay. It was zero It was some nanosecond. So okay, that's good. It's not a problem of delay in the IRQ Let me see if I had an animation or not. Yes, I did so the timer not Eric you Did Eric you latency from time or not was one microsecond. It's okay. It's just the time to put up and The time or not I RQ it the execution time of it. It was like nine nine dot five microseconds and There is also one interference. That is why The blocking tread was running. It also suffered some sort of interference. That is older IRQs happened So the the timer I RQ another timer I RQ happened a Software IQ to software cues happened and we can see here Also, the migration tread was dispatched on that CPU and we can see the amount but they are not that relevant, right? It's zero seven five percent to one percent. It's not a big thing The big thing was a key worker tread blocking a key working tread blocked by 500 microsecond and here we can see that it was again I'm not against butter fs's, right? It's just that I use butter fs and I'm getting you the example there And I do like butter fs so butter fs people. I like you I use on all my system. That's why they are here It's just that here we are running on the no real-time kernel So we have the Premchon points inside the kernel and it's normal to have this Premchon disabled It's like if Premchon is disabled by the fall and there are points in which you check if there's something to be risked So here what one could do is okay inside of this code I could go there and try to put more Premchon points more converse cat converse cat converse cat to try to reduce the latency for the no Premchon kernel or or if you have like the The the Premchon kernel but not the the Premchon 30 Why would one like would like to run that that that case because they do not wants to have the overheads of the The real-time locking for example, it will boom to has the the low latency kernel It's just like that. So one could go there and optimize this use case. It's not Black and white. Let's say in the in the sense here that you need Premchon 30 for everything You can tune the no real-time kernel and the tool also works for that so Okay Real-time the RTLA has its roots on tracing, right? It the trade it has a tracer and it's done by people who likes tracing And so it's natural for it to have also interface for the tracing things So as it act as the RTLA timer lots the front end for the timer lot tracer By the way, all the documentation here for the tracer and for RTLA there are on the kernel documentation You can go there and read main page and everything so the The timer lot can enable the us and noise trace points and they report the amount of time that each task run on the kernel and And and it gives you a number and I'll explain the number. It's easy to read And so we have one trace point for each kind of task So there is the enemy the enemy noise the RQ noise software Q and trats Then we are connected with those blockings and interference and these values that they report they report a The amount of time in nanoseconds This value is already free from interference. For example, if I have a tread running blocking and an interrupt happens The amount that time that the this interrupt run is already discounted from the execution time of the blocking So I don't need to look when I see the blocking time I don't need to try to reinterpret it to all these I need to discounted this value because there's an IRQ and blah, blah This value is already readable. It's 10. It's 10. It discounted everything So it it avoids us having to look back and trace and look back and trace and get back and forth back and forth So here's one example of timer lot and the auto analysis and tracing out together. So Run the auto analysis stop with 30 on only on the CPU for and broom it stopped Here is we can have a look at the analysis the Daniel from the past will probably click on the something Daniel from four years ago. Yes. So in this this is the the the point where the inter or the pension was disabled and Go video Okay, you see there's this function there So I'm doing tail on the trace file that's automatically saved and here we have the the F trace output the tracing output by default, it's only enable I'll see this is the IRQ latency and This is the amount of noise that the blocking tread gave and it has the starting time and the duration How long the IRQ run? These are my late and so I can read it here as well If I wanted to go deeper and and and and trying to read the auto analysis from the trace because Because I think that we will see now Okay No, this is not public yet. So that's the problem of using chat in presentation It's Luis Claudio Is a colleague? so The RTLA timer not can also be used or it can also be used to enable other Tracing features so I can use all the tracing events on the kernel is starting by the timer lat interface and And here is the option that can enable future enable events run futures and triggers here is a video so I Wanted to say run the stop on 30 and the stop if it's cad It's cad work queue it's cad IRQ Vectors IRQ And run so the thread will stop And if I see here the Go ahead any of them past If I look at the trace file there It will show okay, then from past trying to explain. Oh, then we need to run faster It's here. Okay. Here's the trace output I just read that same file and here I can see I can see the trace points the I Can see like it's cad waking is cad wake up Local timer. These are the events that I enabled via the command line. So I can use them So time or not tries to avoid or tries to reduce the amount of time that you will go into that direction Going deeper in tracing, but it also allows you doesn't block you to go into there The one can always go deeper and deeper and deeper as much as one can or like There is even more things we can even go Further into the tracing because there's a lot of options that we can do in tracing So for example, we can use the tracing histograms to get all the interference that IRQ gave on my system and have all the interference that software Q had on my system to check then at the beginning because as As the timer out for a second they work by sampling It might be the case that there was an interrupt that was very long But it was just not at the same time that the timer was fired So that could have been a longer latency that I didn't catch but with the OS noise trace points and the histograms We can have a look on that So I will paste one histogram line they are here, right? So now you see why I posted the link So this page and I have this command line That's why I placed the reference there because I don't I cannot memorize it And I paste it on my site just because of this because I need to check it So I'm enabling timer lot and say create my histograms for the IRQs and display the IRQ name on DCPU how many of them took 10 microseconds and then it shows a histogram Same for IRQs same for software cues and now the timer lat is running. It has these histograms running in the background and When the traces stops it will automatically save those files and this works for any histograms If there is a histogram that was enabled time a lot will save the histogram file So here, let's see how many oh My system has NMI's and I wasn't expecting actually and my system has NMI's and I can see how many times they took So they are generally on the 9 to 10 microseconds So my latest can be at least a microsecond because an NMI can happen So then the IRQ noise I Can see all the IRQs that happen on the given CPU like your CPU CPU 1 2 Here you can see the pattern go into 12 17 and Going down you can see the pattern, but look this I wasn't expecting this either when I run The next one here. There is a timer IRQ that took 32 microseconds So my latency is bound to be at least 32 microseconds on my system Always because there is an IRQ that takes at least 32 microseconds And I know that there is an NMI that can take at least 10 microseconds So I can have a 40 microsecond latency on my system Even even if I didn't have one example here if it didn't stop So I can start doing this reasoning if we keep doing this reasoning We will reach on another tool that is the one I present on 2020 That shows the worst case latency that tries to compose these things It will be part of the timer lot is just not yet because it depends on some of the Optimization that you have to do on the preempt and an IRQ disable trace points and those trace points aren't enabled by default on distros Also, we can get man and also can get away to have those trace points enabled by default because if we reduce the DD overhead we can have these other tracer enabled and I can try to ask the it for merge There is one feature that is Hopefully will be part of the current in the future that is now the workloads in the kernel But people say that I would like to see it in kernel in user space and there is a feature Hopefully on the way for this. So instead of having the the user space the current kernel workload We have a user space workload that use interface provided by the timer lot And okay, I think I'm running out of time. I just need to run So here's one example of this option. It will be enabled with just a flag that time allowed that shoe And so instead of having two metrics now, I have three metrics The IRQ the tread and after pinging back in the user space So it goes to user space and if it goes straight to the current it showed the latency or the execution time of going to this space and returning This data will be larger also because now as we increase the window we can have more chance of having interference and that's why thinking on this abstraction is easier for us to communicate and We can also have histograms and this interface is exposed to user space I can use my own tool to use that interface and then I can read this number as the response time for my task to be Awakened started running run my computation and then go back and the autonomous We also hopefully work so I can see the response time or my of my task have these amount of interference and these amount of blocking and this and there are more things on the way that I some by the ways when trying to figure out the The delay this is right. Okay. The timer lot has these these options I can set a different period a different set of CPUs how long I want to my session to run I can set the priority saying like I want my thread to be scared deadline Not it's cat-fifo. All right. I would like to test if my I know real-time tasks used to see how much Interference I know real-time test could run There's a we can we will have full chose that this is under development. I say I Want to run on DCPU my collection The MAI latency that I mentioned and I I don't want to collect it I don't want to collect the trace the The numbers I just want to a report if it stops like I have a huge system and I would like to have a very fine-grained Period that would cause my air-to-lay tried to run too long and but I'm not interested I just want to see if it stops and print me a report This dump task is important when you have Delatance induced by another tread. So for example, I get only I'm presented So you have things running on another CPU causing pollution on the cache and my tread is running here And it's stopping and my system was idle. So what could have been it displays What was running on the other CPUs for me to get a hint of what is causing me the indirect delay and And by the way, there's another tool RTLA is a set of tools and there's one tool Which is the hardware noise that measures this case when the hardware has some kind of latency For example SMIs or bus locking that ends up creating latency on my On my tread latency because if my hardware has latencies it will also postpone my tread latency So here's an example of the output. I will pause here So my period it on each period like each second period it tried to run for 75 percent of the time And on each of these period the total of noise that my hardware noise received was this amount So I'm having this amount of CPU time net from the hardware noise each of these cycles in one cycle I have 18 it was the maximum and These 18 was composed for at least two because my max single noise was 13 microseconds I can see here also that there were hardware noise and NMI thousand noise This is the a way to the thing on the back here is the US noise tracer if I are accused disabled So now I'm saying, okay I would like to stop the trace If I see a hardware latency larger than 15 and I would like to see the and I would like to see the NMI trace points Right, I would like to check the trace So it stopped and I see the trace and Here I can see oh, there is a sample threshold for 15 microseconds and while it was running I'm sure that there was an NMI because this is interference is one. I'm sure that it is one because of the lockless Synchronization that I have on kernel that ensures me that this was one there between the time I read here and here Just one thing happened. It was this NMI So the NMI noise and the and if I wanted to remove then I need to check why is this perf event NMI handler running here before I'm going deeper on the analysis and So final remarks Time or not integrates the workload tracing out analysis all in a single tools and it tries to produce a summary with Root calls that can be hopefully read by my project manager without having to call me but The idea is also to speed up the the way like for example a person that is not from RT testing the Their subsystem in kernel. They saw that they have a regression Now, how do I report this to the real-time people for them to give me a hand boom just one one output and And the tool also allows the users of more advanced in tracing and we will more and more integrate those things with Steven's library like trace command And The RT is the RT is the home of photo tools for analysis We have the timer lot the US noise that measures interference only the hardware noise that I showed but it can only get better, right? I'm working on a tracer that shows the execution time of the tasks is using those trace points There is this tool that I mentioned that that shows the worst case analysis with a formal proof and It it it will be placed inside of RT LA I'm working with Apollo Bonzini and And the Francesco and student a Milan To integrate the timer lot and less noise with KVM so I can see if there was an a noise On my system instead of saying it was harder I would be able to say if it was either hardware it could be hardware, right? Or it could have been the host that prems to the vCPU and we will be able to show those things and So it can only get better Because the tool was think on that way and whatever the community needs it the more tools we can we are able to put there the better I've I've made a script Before writing the presentation. I wrote like a script of this will be my presentation so we have a how-to like of this presentation here and I Just placed online today and that's it questions We have time for a few questions So from what I understood you have the ability to identify the difference between the hardware latency and an NMI latency is this taken into account for the auto analysis or Will unexplained latencies be attributed always to NMI's No when when there is those three points Right that I see when an NMI happened or when an IRQ happened or when a thread happened If none of these trace points happened between two reds of the time It was nothing inside the operating system. So we can say that it wasn't the hardware So when you see there was no that trace point There was This trace point here Okay, then it says one When it says zero There was nothing happened on the operating system level. So it was something on the hardware And that's where we would have we could extend saying okay, if there is nothing here in the hardware Can I check with KVM if there was something then KVM can give us and we can extend for other things as well any more questions Currently not so thank you very much good. So the I will go home thinking that the output is good enough for people not to have questions Thank you, Daniel