Good morning. This is the Vermont House Human Services Committee, and it is Wednesday, April 28. And for the first part of our morning for about the next hour, hour and a half, whatever we are going to be focused on J. R. H. six, which is a resolution being treated as a bill that indicates that racism is a public health emergency. And this morning we have bore Yang. The human rights person for the state, and we have case long, who is a Vermonter who ran for the Chittenden County Senate, and you're a realtor, if I recall. Yes, that's great. And looking forward to the hearing from both of you. And so if we can start with bore and just so that folks who are on YouTube can follow us on our committee webpage is one a source list for the different resolution, the different where as is in the resolution. And there also is a case provided some testimony or provided some information to representative Rosenquist on comments on the bill that represented Rosenquist shared with us and so that is on there as well. Good morning for good morning. Good morning. Thank you for having me here today. I see a lot of familiar faces and I think a few new ones, and that's wonderful so I wanted to just kind of introduce myself because I think this is actually my first time in this committee, this session. So for those of you who don't know me, my name is bore and I am the executive director and legal counsel for the Vermont Human Rights Commission, which is Vermont State Agency that enforces the anti discrimination laws of the state in three primary areas, housing places of public accommodations and in state government employment and places of public accommodations is hospitals, roads, police prisons. So it means a lot. All places really that provides any sort of services and benefits to members of the general public. And to sort of provide my testimony today for some of us who are also not just auditory learners by visual learners I created a quick PowerPoint, which might give us a nice little break from listening to people and looking at faces all day I appreciate it. Especially during these times so I'm going to share my screen. Do you all see that. Okay, yes. Okay, so obviously we're here today to talk about racism and declaring it as a public health emergency or a crisis. The mission of the Human Rights Commission is to promote full civil and human rights in Vermont, and the commission again protects people from unlawful discrimination and those three areas that I talked about by enforcing the anti discrimination laws of the state. The ultimate goal of the Human Rights Commission is really to eradicate discrimination and all forms across all systems. But fighting racism and other forms of discrimination, one case at a time is a necessary, but really inefficient way to mitigate and eradicate discrimination. I want to demonstrate that by talking about both the federal and Vermont's fair housing laws, which are probably the most comprehensive and most protective anti discrimination laws that we have. And yet, we know that housing discrimination remains prevalent in the state. And one of the results from Vermont legal aid tells us that housing providers in Vermont, generally disfavor African American renters renters of foreign origin renters with children and renters with disabilities. In fact, in 44% of the tests housing providers demonstrated either preferential treatment, or the housing providers events on ambiguous discrimination. The origin discrimination occurred almost 48% of the time. And what they found was that discrimination was subtle. Housing providers display polite and courteous demeanor. They often shook people's hands and smiled and subject testers walked away and control testers walked away without sort of feeling any sense of animus or hate. Housing providers shared information with white tester of us origin about other available units when the testers price range. Within the testers price range, if the unit was no longer available, but might fail to share any information, or shared only limited information about units outside the testers price range to a subject tester. Housing providers were more likely to follow up with control testers than subject testers, even when the subject testers call to share that they were still interested in the units. Many of the subject testers believed that the units had been rented and no discrimination had occurred when in fact the units were still available and offered to control testers. Similarly, African American subject testers were less likely to be told about other available units and were asked questions about household composition and their employment, much more often compared to their white control testers. Why is housing discrimination so prevalent when we have such a huge body of law that prohibits discrimination from occurring both federally and at the state level. Because in Vermont people rent and sell through word of mouth and homeowners are still predominantly white and more affluent because affordable housing is scarce and there is a tough competition for housing. People are more likely to tolerate discrimination and less likely to report housing discrimination for fear of losing their housing, and because implicit and explicit biases persist. The example of this is looking at our history and the connection between systemic racism and housing individual racism and implicit biases. So we know that white people and black people had actually started to live in integrated housing in the 50s, except to that civilian public housing programs demolish some of those integrated housing to develop segregated housing. The federal government then subsidized suburban housing development on the condition that those homes be sold only to white people, and that the deeds of those homes, the prohibited the resale to black people. And there were many, many zoning laws, which made black parts of towns, industrial plants waste and toxic use, while the same was not true of white neighborhoods.  And today, we see through testing that real estate agents and rental housing providers still show fewer available homes and apartments to black people. The fear housing laws did not address past discrimination homes with restricted covenants continued to be sold to white families for generations thereafter. And those homes increased substantially in value and equity.  And statistically, that black income is approximately only 60% of white income. And what is more devastating is that black wealth is only five to 7% of white wealth. So what is, what does all of that mean. Well housing is connected to education, housing is connected to transportation, housing is connected to where we work, and housing is also connected to our health care and our health care systems and access to health care. And the reality is, is that if we're really going to eradicate discrimination. It's going to take a roadmap that kind of looks like this picture where all paths sort of converge and work together. At the end of the day, this is a really great quote from Dr. George's Benjamin, who's the executive director of the American Public Health Association.  We know that high school graduation is a determinative of health. We know that women who have a higher education, their children are much more likely to live beyond their first year of life, a range of things from social determinants of health that make an enormous difference. So what is in a public health emergency? What does that even mean when we talk about a public health emergency or public health crisis? A public health crisis or emergency is something that impedes individuals and communities from being healthy. We know that discrimination is deadly. Dr. April Joy Damien, the associate director of the Weissman Institute says that the turning point of a disease when an important change takes place, indicating either recovery or death. A crisis signals a critical inflection point that is a matter of life or death. It is abundantly clear in the US that racism is lethal to black Americans, to indigenous people, Latinos, Hispanics and other people of color. And Dr. Mary Bass said the professor of the practice of health and human rights at Harvard says there has never been a time, not a single year, where the United States population of African descent hasn't been sicker or died younger than white. Racism contributes to shorter life expectancy and poor overall health. Here's some important data that I thought would be important to kind of review today. People from ethnic and racial minority groups are at greater risk of getting COVID-19 and of dying from it. As of late July, black people who make up just 13% of the US population accounted for a quarter of COVID-19 deaths according to an article in the Harvard medicine. Several researchers at Harvard also said that healthcare workers of color were more likely to care for patients with suspected or confirmed COVID-19. They were more likely to report using inadequate protective care and nearly twice as likely as white colleagues to test positive for the virus. According to the CDC, there was a 16% difference in the mortality rates of blacks versus whites across all ages and causes of death. In real world terms, the disparities can mean that black Americans in some cases have more than a decade shorter life expectancy than whites. And in places such as Milwaukee, which recently declared racism a public health crisis, a black residents' life expectancy was on average 14 years shorter than a white residents living in the same city. Racial and ethnic minorities throughout the United States experience higher rates of illnesses and death across health conditions, such as diabetes, hypertension, asthma, heart disease, obesity, when compared to their white counterparts. So the impact of declaring racism a public health crisis or a public health emergency, it isn't just performer, it isn't just renaming it, but it actually has a real substantial value and meaning. And I thought that a really good example of this is looking at smoking and the results of declaring smoking a public health crisis. We know how many people smoke now by sex, race and gender and age. And before I talk about that, I actually just want to want us to take a second to kind of examine how fast we have moved from the way we think about smoking. In the last couple of decades, when smoking was really prevalent and everywhere, and even high school kids were smoking and it was all over the newspapers and movies and magazines, and in covers, and it happened really frequently. And it was very tolerable and it happened in all places of public accommodations. And just in the last 20 years or so, we have dramatically changed the way that we think about smoking. And it didn't happen because Americans started becoming researchers themselves and started being mindful of that. It happened because we declared smoking a public health crisis, and we committed resources to addressing smoking as a public health crisis. And the results of doing that was, we now know how many people smoke by sex, race, gender data. We have a lot of data around it, age. We know how many people have diseases and are dying from smoking. We know the health care costs that are associated with smoking. We can test how effective our strategies are in addressing it. Current smoking has declined from declined from 20% in 2005 to 14% in 2019. It's a very short period of time. And the proportion of smokers that have quit have even decreased significantly. Today, no one would deny that smoking causes cancer and that it is bad for your health. We all agree that it should not be glamorized or advertised to children. There is probably 100% consensus that as members of the public, we have the right to know when something impacts our health. The smokers rights were balanced against the public's right to clean air and restaurants, airplanes and public places. And the tobacco industry was taxed to cover the real health costs of smoking. We insisted that its rights to profits did not entitle it to ignore that burden. This is incredibly valuable to treat something as a public health crisis or a public health emergency. Who has declared racism of public health crisis? Well, the city of Burlington has. We know that more than 170 local and state leaders and public health entities have declared racism of public health crisis or emergency. I think the number is somewhere in the 90s in terms of municipalities and states. Recently, the CDC declared racism as serious public health threat and outlined specific steps that it's going to take to address it. And there's a very similar bill at the federal level, looking to formally identify systemic racism as a public health crisis in the United States. And it would mean committing resources to the Centers for Disease Control to prevent develop to prevent and develop health policies that specifically address racial disparities. The clearing racism of public health emergency really requires two fundamental beliefs that racism exists and that racism is prevalent. And really it is everyone's job to address it where they live and work and it really is time for all of us to invest in addressing that crisis that we have really created through years, probably 400 years worth of not just practice and policy but very specific laws that created racism and systemic racism and individual racism and implicit bias as well. So I'm happy to answer any questions that you have. Thank you for I'm wondering. I'm hoping that you can stay through both testimony and understand if you can. But we've got we've got some questions right now representative Rosenquist. Madam chair and or thank you for coming this morning and was wondering when you were talking about the housing issue and that government started building housing in certain areas in the 1950s. It certainly wasn't the purpose to segregate people it so happened that people got segregated by the housing. When you agree to that I don't think they did it on purpose. It was on purpose there's, there's very good evidence that it was on purpose because it's written down that remember this is a different time of our lives now and the government purposefully said, we will develop housing. In the suburbs, we will target white people to go live in the suburbs, and we will make sure that that those homes cannot be sold to black people. In fact, developers will not get money if they sell if they develop homes and black areas. And, by the way, once we sell it to white people, we need to make sure that they can never sell it to black people and so that the government sanctioned those conditions to make sure that those homes were never redone and this is why this is important because oftentimes we think that racism is an individual problem. But there was good evidence that white people and black people had already started to live in integrated housing and that the government came in and said oh no no. You can't do that. That's not okay that's not what we want. And so now we have these very specific historical decisions that were made by our government that I think now our government has an obligation to undo. And not just our government, but all of us and that's why I think it's valuable to look at housing and education, because when I, when I do trainings for educators, it's their job to address bias and discrimination, historical discrimination and present day discrimination. It's not where they work, but it is the job of doctors and healthcare providers to address discrimination in the healthcare system. In fact, they're the most qualified people to address it. And that's why when we declare something a public health crisis, it says the government is saying, we did something to create racism. We can do it. And now we're going to force our entities to address it, the people that have the most expertise to, to look at that. So it's in that sense it's really valuable because it says everybody has their hand in on doing racism. I hope that answered your question. I think you're muted representative. I think this problem of the space part didn't work, you know, Carl, Carl, we still can't hear you. Sorry, thank you madam chair. Okay. And thank you for the answer. You know, I, I wasn't to say I don't remember all the things you talked about happening but I mean obviously you're more of an expert on this than I am as far as the history of it. To me, we've come a long way from those days though we have very integrated communities today. It seems like in most areas as far as I know, you probably would disagree but it certainly appears that we have a lot more integration of living conditions. I just, I also had another question and that was you spoke so much about the smoking as a parallel to look at and a lot about statistics do we have statistics on the smoking level of different races and currently. Yes, we do. Yes, the CDs, because once smoking was declared a public health crisis, the CDC was able to devote resources to collecting that data, conducting investigations, analyzing and reporting that and really it is that reporting that in fact has educated all of us in the last couple of decades to go, whoa, smoking is disgusting, it is bad for our health and now we have so much information, it's costly. And these tobacco companies are making billions of dollars, and they have an obligation to tell us the truth and to address it. And I don't think that that would have happened if we hadn't declared smoking a public health crisis. What specifically was interested in what are the smoking levels between let's say the black indigenous and white communities, if you will. Yeah, that's a great question I don't have the specific statistics before before me, I would say that the answer to that question is not relevant. Not necessarily the comparison that I was making was not about different races and how they smoke, but that the there's a value in declaring racism a public health crisis, like there was in declaring smoking a public health crisis. Thank you. Yeah. Representative Rosenquist we can get the answer for you from the department. I'm freaking up madam chair. Yeah, I'm. I was saying we can, we can get the both what a board boring Yang was saying, which is the comparison she was making was what you know what's the importance of identifying something as a public health crisis and what that then results in. And at the same time you had another question and we can get the answer, we can get the data that you're looking for as it relates to national. Thank you madam chair, it won't be that necessary. Okay. Okay, thank you. Representative McFawn. Thank you madam chair. Thanks for coming back. I, I, I have a factual statement that I want to make. You talked about the housing and how the government segregated it back in whatever 60s or whenever you whatever period of time. I want to talk to you about a lived experience okay. Cambridge Massachusetts. And you can, if you want you can do the research. There are tenements that were put up in Cambridge, Massachusetts. And the people that went into those tenements were white Irish. Americans and people of color went into those sentiments. My recollection was, there was never any government interference to try to segregate the people that went into those tenements. I have to say most of those people were people of low income. And people that were probably not probably that were being discriminated against. I just want you to, when you, when you're making statements that the government did this. My lived experience in Cambridge, Massachusetts was that was not the case government did not get involved, the government was dying to get these people a place to live. And we lived together. We played baseball together. It was a park that was right next to the tenements together all day long. I'm talking about the kids. So I just wanted you to know that. And I don't know how that stacks up with the, the information that you got from Harvard, because this place that I'm talking about is a stone's throw from Harvard Square. The other thing that I wanted to bring up. I've gone through most of the, whereas things with the way you press on it, and you get the information. Most of it is national information. Well, I'm interested in nationally making sure we take care of things the way we should. No question about it. But I want to know what's going on Vermont. And that's the kind of statistics that makes a huge difference with me. I think in Vermont, we have tried real hard. And to make sure that this is a healthy place to live for everybody. One question that I would like. I don't know whether you can answer or not. But one of the statements that was made is people of color. I'm more susceptible to getting COVID-19. Now, I would just like to know why that is happening. I'm not going to debate whether it's happening. I want to know why. Do you know why? That's a great question. Thank you for asking that. Thank you for your statement too. So, what, when one of the slides, I mentioned that the Harvard medical school did some research and showed that people of color who were first line responders were more likely to be tasked with serving people who had COVID. And had less access or less opportunity to use gear that was protective. And they were most more likely to be infected by those patients as well. But the answer to your question about why these things exist is precisely the reason why we need to declare racism of public health crisis, because we don't have all of the answers. When we declare something in public health crisis, we in fact create the language that we need to collect the data that we need. Now, if you need the data before you can agree to collect the data, then that puts us in this conundrum, right, where we can then never access the information that we need. I just want to say also that your question about being interested in what it looks like in Vermont. We have a lot of great data about what it looks like in Vermont in terms of racism in our school systems. We have data that tells us that kids with disabilities, LGBTQ kids and kids of color are more likely to be disciplined than other children. And that is really important data. We have data that shows us that housing discrimination is very prevalent. I shared some of that data with you. What we don't have in Vermont is a lot of data around health and healthcare. And that's why we need to declare, but hold on. That's why we need to declare racism a public health crisis. If you believe that racism is real, and that it is prevalent in our schools, it is prevalent in our housing, it is prevalent in employment, then to suggest that, well, I need to know if it's prevalent in healthcare before I declare it doesn't make sense. Of course it is, but we need to declare that so that we can collect that data for the purposes of analysis. So I also am very invested in looking and seeing what that looks like in Vermont as well, not just nationally. Dr. Richard Rothstein wrote a book called The Color of Law, and it was a summary of his investigation into housing discrimination. And so I would just encourage those of you who might question whether the government has ever done anything intentionally to create housing segregation, to look at statistics and data, and to look at that. And again, he's very well-esteemed. He is not even a person of color. He has done that investigation for us. He has looked at how we have strategically as a government created law specifically to segregate people and that the disparities that exist today is connected to those very strategic decisions that were made from the government. So I'm not here to deny anyone's lived or personal experiences around housing and housing discrimination. I would also just say that we can't deny the statistics. We can't deny the facts. We can't deny things that are written down in history. And he has done a really great job of showing that history to us in a very clear, almost mathematical way. So I would encourage you to read that. Before I'm not, as I said before, I even opened my mouth when I first opened my mouth, that I'm not denying it nationally. That's not an issue with me. I'm looking at Vermont. I represent people in Barry Town. Representative McFawn, let me interrupt here for a second. We are in terms of your health questions. I'm sorry, what I did not get it answered. I'm going to direct you to the testimony of the commissioner of health. On this resolution on, I think it's April 8. And we can go and there is a PowerPoint. And in that PowerPoint has numbers and statistics that are from Vermont. I know that we have been, I know the statistics of the madam chair. I've read them. I asked, so you were asking about Vermont. And these are about Vermont. Yeah, but it doesn't tell me why. The question is why, why are people of color at a greater risk of contact in COVID-19. I don't know. Topper, again, I'm representing McFawn. Again, I'm going to suggest that you read look at that PowerPoint because there is a heading on that PowerPoint that says, what are some of the contributing factors that led to the disparities we see for BIPOC. And so it's right there. I will re look at it. And if it's Vermont related and not Harvard University, or some other place outside of here. And fine, if Harvard University came in here and did the study, then fine, I'm willing to look at it. And I'm so good. I encourage all of us to, and this, this conversation has. I want to thank our, our committee assistant, Julie Tucker, who puts things so well so that I, and has taught me so that I can now find it. But it is for anyone on committee is dated. It's under documents. You can look on our webpage under documents by author and smart Levine. And it's April 8. Thank you. Well, thank you again for coming in. Thank you. Thank you, Madam Chair and thank you for for coming in today. It is a pleasure to hear from you and to have your presentation. I think from your last remark there about talking about the importance of declaring this, I think we saw this with the Department of Health when they also declared racism as a public health crisis. During COVID, they were able to create a health equity team that is now investigating the inequities, understanding that we didn't have translated materials for our various communities, especially here in my wonderful city of when you ski. And making those changes because of that declaration so thank you for pointing to that. My first question is, I assume that you're in support of this resolution. I think that why do you feel it's important that we name racism as as the contributing factor as a public health crisis and emergency, rather than say, economic disadvantages or economic inequities. That's a great question. And again, because we know from the data that we do have both at the national and local level that people of color are dying and have different mortality rates than people who are white. So that doesn't necessarily mean that there aren't other protected categories of people that also probably need the same kind of levels of protection. But before us today is whether racism is the issue, and absolutely it is. And I just go back to, we need, we declare it so that we can collect the data, so that we can report it and educate ourselves. We cannot expect that we are educated first, before we declare something it. It's a cyclical problem here. If that's what we're requiring. And we declare it, because the people who are best fit to address racism as a public health crisis are people who is a public health field. If we don't do that, then it remains the burden of the Human Rights Commission, it remains the burden of other groups of people, community advocates, and so forth to address these issues. It is important that we are addressing racism in each of the field that we are the expert in. And that's why we have to do it. And this is not something that radical. We have a lot of agencies and entities and municipalities that are doing this, including the CDC that has done this. And so, um, yeah. Thank you. Can't hear you a representative small. I knew it would happen sometime. Thank you. No, thank you. And thank you for I was saying that we have also heard from personal stories of BIPOC for monitors and understanding the difficulties and navigating health systems or social determinants of health and being able to thrive here in the state. And so my last question is, are there any edits that you would make to this resolution. That's a great question. No. None at all. No, I sort of leave that up to you all I think it making the resolution is the most important rather than the specific details of it. There are times when you're it's beyond a resolution it's a very specific statutory change that I do often weigh in on commas and semicolons and all of that stuff. But in terms of a resolution, I think the impact of it is much greater than the specific words and where they are. So, Wonderful. Thank you so much. Thank you and representative would you will be the last question or comment for boring, because we also have case long here. Because what we do here in the legislature is we hear from people who have different points of view, and I want to give case. The amount of time that he needs and deserves to share his views. Theresa represent would. Thank you manager I think I'll be brief. One thing that I want to confirm I want to make sure I interpreted what you said correctly on board was that in in in giving the example that you gave for instance with smoking and its connection to help that all of those things that you listed. In terms of the information we now have as a society, we're gleaned after we made that declaration I mean we started collecting it sounds like what I'm learning from you is that that that we now have as much information as we do about the health problems with smoking, because there was a, you know, essentially a declaration saying that it is bad for your health and are you saying that. Once we make a declaration about racism, being a public health threat that that that causes that there's a causation in terms of us collecting more information us, you know, being. When you say us and I referenced the health care professionals but then you know the changes that happen in society. That we will have more information because it. I guess that's my question. Yes, and and really you have answered it because that's that's absolutely right. When the government makes this kind of declaration. It says we validate that this is racism is real racism is prevalent and that we need to assign the task of looking at it more closely to the people that who know how to look at it. And that's that's the value in making such a declaration. There were some lawsuits that came about in terms of smoking that sort of made us realize that it was bad, but once the government declared that it used resort money and time and committed really people to doing that data collection that investigation and that reporting, and that is what shifted the culture around it. Truly what those lawsuits probably would have changed the class action plaintiffs lives, but it wouldn't have been enough for all of us to change our thoughts and beliefs around smoking. Right like today I think we even if you know people who are smoking today, they would say this is this I don't want to do this anymore. They're all still looking to quit it. We have dramatically changed our culture around smoking. And so this is really about changing culture and climate around race to and racism. And so, and to do that, we have to have these public ad campaigns we have to have strategies. We have to have everybody on board doing that work where they live and where they work and this to me is the step really in making sure that that happens. Thank you for and then not to not to disagree with my esteemed colleague from very town but I to him and interested in having a declaration that references Vermont data and 15 out of the 20 data points are Vermont data. It's not it's not a majority of national data I just wanted that to sort of be on the record because people are listening. Thank you. And for thank you. Thank you very much. Appreciate that. And case welcome. As you can see, we have lots of questions, even after there are perhaps a beginning remarks. So I want to turn it over to you. Thank you very much. Can I just introduce him. Absolutely, absolutely. All right. So, camellia long is a fellow comrade at arms. 11 year veteran of the Army National Guard, having served two combat tours and in Afghanistan is the proud father of three members, a small business owner in Milton, Vermont, recently ran for the Senate from Chittenden County and currently is serving as a school board member in Milton School District. And so welcome comellia. Thank you for the introduction. Representative Rosenquist and thank you madam chair and the rest of committee for having me here today. I appreciate the invitation. It's my intent today to present my testimony, more so in the form of questions in breaking down some of the sections within the resolution. I know that I submitted a document that had some cliff notes which also includes some minor spelling errors which I apologize for. I do know the difference between your and your. So, um, so that being said, you know, I do have some questions, some things that are on that document are not going to be in this testimony that I have here. I've had some time to do some additional research. I wish I had had access to Boris presentation prior to this testimony, but I do appreciate the information that she provided. So that being said, I'll just I'll jump right in sections I broke into the where the section starts with whereas would start a new section. So section one, which would be the title of the joint resolution relating to racism as a public health emergency. I researched this online. I found a lot of information on the CDC website. However, I do feel that the words economic disparity or economic inequalities would be a good substitute for language in this particular title. I think it's important that we recognize that correlation does not always mean causation. And I think that's just something that we should take with us as we look at the title for this resolution. As I go down to sections two and three, it mentions health inequality, excuse me, health inequalities exist in the United States based on race, and that are caused by systemic racism. And this is, are we defining health inequalities and what do the statistics say the remainder of this resolution is riddled with statistics yet this portion offers none. Also the words systemic racism are used in this section and offer no clarity on how Vermont defines systemic racism. If systemic racism is important enough to write into a resolution that declares racism a public health emergency. I feel that we would be remiss and not defining it so that so as to anyone, specifically for volunteers reading the resolution understands what it is that the legislation is trying to accomplish and exactly what it is that we're addressing. So we move to section four and mentions black and Latino people in the United States have been nearly three times as likely to die. My question is, why are we using, why are we using national data in this section. Why are we not using data from our own state. If racism in Vermont is a public health emergency, then we should provide Vermonters with real data as it pertains to our state. Are black and Latino people more likely to die, or are they dying at three times the rate are the deaths or likelihood of death related to cove it or racist. And I think that's a question that we have to ask ourselves before introducing it into a resolution, and then presenting it to the public. Black residents comprised just over 1% of Vermont's population, they account for approximately 4.8 of the total confirmed covid cases. My question is, why is the data only referencing the black population. Why are we not using data from all non white Vermonters. Would this statistic present a less ominous excuse me would it present as less ominous, if we included all non white Vermonters, I feel leaving ethnic groups out of the statistic depicts an inaccurate narrative. If we look at the data in terms of household with the percent of population to the percent of confirmed cases be closer to a one to one ratio. And I think that's again that's something else that we should consider as we write it into the resolution. It's important that we present an accurate narrative to Vermonters so that we understand what it is that again what the legislative is what the legislature is trying to accomplish. Section six, Vermont residents experience barriers to equal enjoyment of good health based on race and ethnicity. So, is this in reference to all Vermonters, or only non white Vermonters, what are the barriers in Vermont that prevent equal enjoyment of good health based on race and ethnicity. Is there data to support the statement, is this due to racism, or can this be attributed to economic disparities and inequalities. And as I read this I understand that board presented some information that would answer some of this question however I think that it's, again, we would be remiss if we didn't revisit these questions and ask them before presenting it in this resolution. Seven talks about COVID incidence rate and non white Vermonters. I appreciate that this section includes data from other ethnic groups, however, what does the statistic look like. If we look at the incident rate incidence rate by household. What is the incidence rate versus death rate versus hospitalization. Also is the disparity due to racism, or would it be better characterized as an economic this or an economic disparity or inequality. And again, I feel it's important for us to provide an accurate narrative, as it pertains to Vermont. Section nine, 30% of non white Vermonters had household contact with a confirmed case of COVID as compared to only 20% of white Vermonters. The question is, does this statement taken into consideration the household size, can this disparity be attributed to cultural differences between white and non white Vermonters, and the propensity for multi generational non white Vermonters. And I believe that this is something that we need taken into consideration when presenting this data, so that we're not presenting a skewed, a skewed version of what is actually going on. Number 13 21% of black Vermonters own homes, while 72% of white Vermonters own their own homes. So, this section should include all non white Vermonters, not just black. And that's something that I see throughout this resolution and just overall so as we talk about racism. I believe it's important that we talk about racism and be inclusive. Right. So, black people are not the only people who experienced racism, and black people are not the only ethnic group that make up the diverse population of Vermont. So, again, we would be, we are remiss in leaving out other ethnic groups and when we only specific focus on one. On some level we're disrespecting the, the other groups that that we would consider a part of a demographic here in Vermont. So my question still on 13 is, how many non white Vermonters want to own homes, and is this parameter defined in a similar fashion as unemployment in that, if you were not looking for work then you were technically not employed. So, if non white Vermonters want to buy and can't is that due to racism, or is that, or can that be attributed to economic disparities and inequalities, based on the cost of living here in Vermont. I understand that that's a different conversation. But again, it's a question that we have to ask in specifically regarding, is it because they are unable, or they don't want to. As we get into 14 talks about the median household income of black Vermonters versus the median income of white Vermonters and again. I believe this would read better if we if it referenced Don white Vermonters versus white Vermonters. And even still, are we saying that the difference in the median pay is due to racism. And if not, then how is the statement relevant. I believe that this section would better support a resolution that states economic inequity as a public health emergency. Or I'd also hesitate to leave the section in this resolution. I personally feel it's disrespectful to discuss differences of income and leave out the pay gap as it as it pertains to women. And that's, I'll leave it at that. Number 16. One in two white Vermonters experience housing problems, which is defined as having homes that lack complete facility kitchen facilities or plumbing, having overcrowded homes or paying more than 30% of household income towards rent, mortgage payments and utilities. So this section does not contain a comparison to non or excuse me to white Vermonters, and I'm wondering why that is other sections within this resolution are clear about making a comparison between white and non white or white and black. However, in here, we're not seeing a comparison to what our white counterparts are the housing problems due to racism, or again, are they due to economic disparities or inequities. Number 17 black Vermonters are overrepresented among Vermonters experiencing homelessness in that they make up 6% of Vermonters experiencing homelessness while making up approximately 1% of Vermont's population. Again, this section does not contain a comparison to white Vermonters as a percentage as a percentage of population. And additionally, we are again, we're again only looking at the statistic for black Vermonters as opposed to non white Vermonters. And then lastly, number 18 racism constitutes a public health emergency. My question is, is racism truly a health, a public health emergency in Vermont, or can this be attributed to economic disparities and inequities. So those are my questions or opinions in regards to this resolution. I speak as a Vermonter as a citizen of Chittenden County, and I do not represent my, I do not speak for any particular group other than myself. So I, if you have questions, I'm happy to answer them at this time. Thank you. Thank you very much. And I thank you for the closeness with which you took a critical eye to this resolution. You started, I guess I have a question for you. You started your, your first comment joint resolution relating to racism your first sort of comment was, was I disagree. There's a question behind economic inequality or economic disparities. So I guess my question to, to you as an individual is whether or not the, the underlying philosophical or value or perspective that racism is a given that systemic racism exists and exists in Vermont. Do you, is that something that you disagree with and so that is that the import of a lot of your questions. So I look at this document I just I look at it from a data standpoint and presented the presentation of that data. I'm just with military, so I thoroughly enjoyed numbers and looking at data understanding the data and, you know, and presenting a narrative. I do believe racism is, is a problem. It is absolutely a problem and exists everywhere. And we can, we can go back 100 years we can go back 1000 years but it will always be there. In fact, did we go to say that this is going to be the root cause right so I feel as though we are looking to tackle an issue that will never go. I feel as though human, the human race will always find something that we can that we can fight over. So whether it's race whether it's gender. We can, we can legitimately go down a long list and I feel like these things will will constantly be issues. However, if we can address economic inequality, then we can help. I would say a much bigger group of individuals right and in that we can help lift each other up and through education. Yes, we can absolutely address racism to agree but really what we're looking at doing is we're trying to change the mindset of people who don't even fully understand why they have the mindset that they have. Did I, and did I answer your question. Um, no. No, no, you know, you absolutely did and I have to remember that sitting in this seat right now. I am the chair of the committee, you and I are not having a conversation over coffee, nor are we having an academic in my other life I teach, nor are we having an academic discussion and I need to listen is important for me to listen and try to understand what you are saying. And the point you are trying to make on which is, which I'm still trying to understand. Okay. And as other people ask questions and comments that may help my understanding as well. I believe we have representative Redmond and then representative would. Thanks Madam chair and thank you very much for being here appreciate your testimony. I wanted to ask a question about you made a point that there continues in the resolution to be an emphasis or or a mentioning of data around black for monitors and that you know you you have issue with the fact that other non white or white for monitors are not mentioned at you know in data points and I'm guessing I don't know the auspices of the resolution and where all the data came but I'm guessing that that black for monitors the data points on black for monitors were pulled out because they were particularly dramatic relative to the, the population in the state and I'm, I guess I'm wondering why that's probably you know why that's problematic, because I think it is important to shine a light on people who, you know, seem to be suffering challenges or difficulties greater than another group, for example. That's kind of one question why that's an issue. And then, and then I have one other quick question I want to ask as well so. Sorry, is it my turn. No I was a, I should let you answer but my committee knows that I have to interrupt. Mary Beth represent Redmond I'm the comment that I'm going to make as a non researcher is that oftentimes data is not identified and that is something that actually Dr Levine talked about a lot on in his weekly things. Sometimes we do data is not reported because the numbers are too small for them to be statistically relevant or specific data is not reported, because in a place like Vermont. We're not giving out information that is too personal, because we would be able to identify. Now, now case, please answer. Okay. I guess I find it. I don't know that offensive is the word that I want to use but as we sit and we talk about, you know, equality, and we talk about again racism racism isn't just a black white issue, and there are so many other issues that would also need to be addressed that could could be contributing factors to where we are now but if we're going to talk about race, we can't just talk about black people and white people. We have to talk about, and unfortunately when we talk about racism that has to do with where it seems more often than not we're talking about white people against non white people and we know that racism exists within the non white population it's even it's frustrating to even have to say it this way so I believe that as we look at racism, white versus non white that we need to include that data, so that we present an accurate picture. If the data, if the data says that we're 10 times likely that that is great that is great data that we can drive forward on that, and then we can make decisions and use that information to drive the decisions that we make, as we move forward but if we're making decisions on data that is inconclusive, or that is skewed or isn't presented in a way that's accurate then that that slight change here that that little that that half of the that that half a degree turn here translates to something much bigger as we move<PHONE_NUMBER> meters down the road. Right, so I think that, again, I'm, I like data, and I think that it's important that we include all of those groups so that they don't feel excluded so that, well, you know, so that it's not. Oh, well this is just a black white issue when you know there are other ethnic groups that are experiencing difficulties and experiencing racism. I does that answer your question. I appreciate that I appreciate that perspective that that helped us.  I'm, it kind of goes to Madame chairs point which is, you know, do you feel that underneath those disparities that there are roots and systems in place that don't offer equal, you know, opportunity to everyone. I mean it's, it sounds like that's problematic for you so I'm just curious. You know what your thought I mean I, I guess I, I feel that there are there are systemic things that that provide opportunities for some and take away opportunities for others and I'm, I'm guessing that that is a that would be a new for you so I'm just wanting to really understand kind of what you think is below those disparities and inequities. I could understand. I could understand. Someone's viewpoint. If they were going to say that racism is the foundation for all of these things that are taking place, but then I go back to. Well, then what comes before racism and is that the thing that we should address. So how far back do we go, how far back do we place blame or assign responsibility before we get here right so I guess that that would be where I stand on that in regards to systemic racism again as I mentioned I think it's important that we define what we mean by systemic racism because honestly, I don't feel that everyone has the same understanding of what it is. My understanding is that these are systems or organizations in place that perpetuate racism or that that perpetuate racism so I guess I would challenge my question is, are we saying that there's racism within the health care system. Is there systemic racism there. And if so, then, why is it not being addressed.  Or is it they can't afford to get the health care that they need, or are we talking about the people who are in decision making positions.  So what is it are they systems and policies, or are they people that we haven't in power. We have to address those in two different ways right so we really have to we have to find where the problem is we have to fixate on it, we have to target it, locate it find out exactly what it is we have to, we have to go in, we have to eradicate it exploit the data, analyze the data and then move on and just and start the process again to help us identify, you know, these other systems and processes within a within an organization or within a system. So, if that is actually the case. So I guess when I think systemic racism I think of the redlining and, you know, black laws and the separate but equal and I think about all of those things and I believe that we've come a long way to get rid of a lot of those things and while there is some type of undertone. I don't believe that it's blatant and I think the thing that we're chasing. I wonder if we would be better to chase economic disparities and inequalities. And that's just my question. And I guess the question that I post to you and that's, it's the one that I would post anyone my personal experience as a person of color is very different than I would say others within the state of Vermont. Thank you. Did I answer your question I tend to round. Okay. That was very helpful. Thank you. Okay. You fit right right in. Representative would. Thank you madam chair. And thank you Camelia for for you've given me a lot to think about in this resolution and listening to your responses to represent Redmond and represent you because my sort of question in my in my head is still around this this concept of economic disparity. Not versus racism but sort of is versus racism in terms of this resolution and it, it feels to me like a chicken and a thing it feels like a bit of a circular arguments. I appreciate your comment about, you know, causation at the beginning of your comments and so I guess this is really not a question after all that I'm still struggling with the with the feels like to me a circular argument around economic disparity and race and I agree with you that that that also applies to other populations and maybe next year we're going to have a resolution that references, you know, women. In terms of their economic disparities and access to health care and all those kinds of things. And I'm trying to figure out what disadvantage. What disadvantage is there to naming race as an issue for proper public health. That's a good question. I wasn't prepared for that question. I'm going to add it if you want and come back to it. Sure. So I don't. I can't say one way or the other. I feel, I feel words are important. And I, and I, as a citizen, when I hear racism is a public health emergency. As someone who has experienced racism. I don't see it as a public health emergency. I see cobit as a public health emergency. Right. I see pandemics and epidemics as public health emergencies and I understand that we're defining. We're finding different ways to define what a public health emergency is my my concern is that we're reaching and going to racism as a public health emergency. You know, going to what or had mentioned earlier for presentation, having to name racism as a public health emergency will allow us to then do the research and collect the data. And I understand that I don't believe it's the right way to go. And with the analogy between, I'm sorry, the comparison between racism and as a public health emergency and smoking as a public health emergency. I don't know that we needed to declare it as a public health emergency to find out that smoking is bad and does bad things. I don't know that that's the case. However, I am not a medical professional and I'm not here speaking as one I'm just speaking as a citizen. I feel as though if I were to go down to Hanifords and have a similar conversation with a citizen, their response would be very similar. So I think it's important that we choose our words carefully. And the words that we do choose that we offer definitions if there's any ambiguity like systemic racism. And what constitutes a public health emergency. But did they answer your question. No, I really appreciate you sharing your perspectives. It really is helpful to understand the, the, you know, variety of thinking around around this issue and around this resolution and so I really, really appreciate you being here. Thank you. Thank you. Representative Rosenquist. Thank you Madam chair. My concern as I've expressed before is that term racism is a connotative word that carries a lot of extra baggage and unfortunately divides us. I know a famous movie star once said when asked about the, how can we reduce racism. He said, stop talking about it. And that was Morgan Freeman. And I, in many ways, that's one of my concerns here is we're focusing on something called racism, which I think many of us have don't really know. And so Camulia said, maybe it's something's going to be with us forever. And instead of concentrating on the outcomes, and some of them may be outcomes of racism, but some of the outcomes that are talked about in this resolution are not just racial, ethnic, cultural, and I would say circumstantial disparities. What I mean by circumstantial is new immigrants to this country, many times will live in congregate settings. My family when they came here from Sweden. My, my mom and dad they had my father's mother living with them. They had one of his uncles. One of his brothers and one of his sisters living with them. And then they actually brought over another brother from the Philippines and his entire family and they lived with us. This was just about the time I was born. So I don't remember too much about it, but my one uncle who had served in trenches in France, he lived with us for the rest of his life. And I remember that very well. So I just think that sometimes, and coming back to Camulia's use of the term causality. Okay, in other words, we really do need to understand the cause is may not be racist. And there are a lot of other contributing factors to some of these issues and one of the greatest ones. I mean, we know that right now, Winooski is sort of a melting pot of our state and a lot of cultural diversity. There's no two ways about it. I, I owned an apartment house in, in Winooski for 10 years and experience a lot of those things. And the, and it was quite beautiful. I'm walking down the street and in Winooski is quite an experience. Sometimes the different dress and that people wear and different things like that. So anyway, I just wanted to make that point that, as you know, that's one of my biggest problems on this, using the term racism, instead of dwelling on the outcomes and dwelling with them as a public health emergency or whatever divides us. And I think takes us away from the real solutions to the problem. Thank you. Thank you. And, you know, in that instant, not to piggyback, but it just makes me think again, how, how deep of a problem is that within Vermont and would we accomplish more if we address the economic inequalities and lifting Vermonters up. Would we accomplish more, you know, through, you know, you know, education training jobs, and if we're lifting, we're lifting each other up. It's not based on races, just lifting people up we're going to affect a much larger demographic and I think it's something that we need to consider. It doesn't have to just be race that can't always be race and, and I remember Morgan Freeman, Freeman saying that, and, you know, they're on on some level I want to say that but then I also realize that it's not the right answer because it's, it never really gets addressed, especially for those, you know, that that are genuinely having a deep struggle with experiencing racism. So on some level, yes, we do need to talk about it, but it doesn't have to be the root cause of all things bad. Right, we need to address that we need to, we need to, if it is, and that is the case, then let's label it as such. Again, let's make sure to just make sure that we're using accurate data and that we're using our words and just using the appropriate words to address what it is that we want to say. I hope that helps. Sorry, I know you had a question for case case. Thank you very much. We have one short question from Dane because then we are wrapping up with a final comment. Thank you Madam chair, and thank you case for being here in your testimony and raising your questions. I guess one point, kind of going back to Teresa's point of chicken or egg kind of thing board mentioned that is, you know, 60% black income is 60% of white income. Black wealth is five to 7% of white wealth. So is addressing racism in a way addressing economic inequality. If you only address economic inequality and not racism, then how do you change those disparities? Are we talking about disparities and race or disparities in finance and or in economics. So it's so if we address racism. Yes, I believe that we can address some of these other disparities that we're talking about. However, if we address the actual disparities themselves, I feel like we may get a much we will get more movement in the direction of again pulling people up and in that addressing some of the other maybe peripheral related items. Again, just making sure like the data is extremely important, and we have to understand why we're doing what we're doing and we don't want to make a knee jerk reaction in jumping to something and then calling it racism if that's not in fact what's going on because as I look through the resolution the data. But what it's not presenting me is with a uniform picture of what is going on in Vermont, which, which make, which makes me very suspicious about the information that's in here, and that's not to say that that the resolution or the data in itself is in fact suspicious, but it gives me that feeling, because it goes from national to state to black to white and sometimes we talk about other ethnicities and sometimes we don't sometimes we make a comparison with with white and sometimes we don't so it's. I would feel better about the resolution if the if the information was presented in a more uniform way. Oh, you're on the other side of the screen now sorry okay. Thank you, thank you for your question representative. And case thank you thank you for really being part of part of this discussion and representative Rosenquist. I really thank you for introducing. Mr long to us all. Thank you very much. I believe that I'd like to as offer some additional comments from boring. And then committee, we are. We will be taking a break short break after so boring you will have your final comment. What an honor. Thank you. I just want to say that we cannot group everyone into the same group, as if all of our experiences are the same. It is fundamentally unfair to suggest that 400 years of oppression against black people does not make their experience any different from all poor people in general. It's not a shortage of laws that protect women or protect people who are economically disadvantaged. In fact, if you take a look at a lot of our bills those are the ones that often get picked up. We know as a matter of fact that black and brown children are more likely to experience economic disparities be stopped by the police be disciplined in schools.   Being black in America is fundamentally different than being any other person. And I am an Asian American, an immigrant, a refugee, a person with a disability. I also have identified myself as being someone who has been economically disadvantaged for probably the majority of my life. But it does not mean that I understand or know what it's like to be black, it is very different. And all of the disparities that we know to be true impact black and brown children at a very alarming and disparate rate than it does any other This joint resolution does not dictate what the work will look like or what the work will be or the data that needs to be collected, or the strategies that needs to be utilized to address disparities. This is not a statute. It's a hope. It's guidance. And expecting that we should have a uniform picture of what the data looks like in Vermont before we've even declared it a problem to collect that data is problematic. I think and I don't think I'm incorrect in saying this that even the Department of Health in Vermont only recently started collecting this information. Okay, so to suggest that we don't have the data to support a resolution is just disingenuous, because we need the resolution to for the government to force the agencies and the departments to collect that data and to provide us that information. So, I would encourage you to pass the resolution. I am not offended that by any of the language that appears in the resolution. It is not dictation of what the work ought to look like and will look like that work will be done by the people who are experts in this area. And I would encourage you to pass it quickly. Thank you. Thank you for and thank you case and bore if I can just say, our committee doesn't do anything quickly. So I appreciate you would like us because one of the hallmarks of our committee is in fact bringing in opposing or alternate ways of looking at the at what is before us. And I want to say today in this morning, I think is a really good indication of the way that we need to do our work is to hear all of that. I would say I think I speak for my, you know, and how it will look I don't know, but to pass this resolution, ultimately, but listening to all of the other perspectives with that. And so I really respect and thank both of you. Case and bore for really engaging with us for so long this morning and represented Rosenquist I am trying as chair to finish and wrap us up. Do you really need to make a last comment. So thank you very much, both of you. And this ends this morning session on the resolution, we will be taking a five minute break. We'll be taking a 10 minute break and then as a committee we will come back to a whole different subject, which also has varied some different.


So, hello everyone and thank you for being here with us today at KubeCon 2023. Our talk is about hardening Kubeflow security for enterprises. I'm Diana Tanasova, a software engineer at VMWare's open source program office and I'm currently contributing to the Kubeflow project. Today I have the pleasure to co-present together with Julius Bonkhochert, who is freelancer, a DHL employee and who is contributing to the Kubeflow ecosystem for over two years and he is also a founder of the Kubeflow Security Working Group. So, that's enough for us or do you want to say something? Okay, maybe here. And how many of you are planning to use Kubeflow? Okay, we need a learning part. Everybody got their hands up, amazing. So, yeah. So, here is the agenda, so we will still do an overview of what Kubeflow project is and we will try to share the answers of who is using Kubeflow and why. Then we will do a short introduction of the security working group, we will share its main goals and initiatives. Then to make sure that we are on the same page, we will do an overview of the Kubeflow architecture and then we will share, we will discuss the authentication flow of Kubernetes of Kubeflow. Then follows the interesting part, we will have a discussion of different various Kubeflow security issues and we will end up with the conclusion. We will wrap it up. So, what is Kubeflow? Kubeflow is an open source machine learning cooperation platform based on Kubernetes. It enables data scientists and machine learning engineers to build, scale, deploy, orchestrate their machine learning workflows. It tries to standardize and automate the iterative nature of machine learning workflow. We can see Kubeflow as an orchestrator of commonly used machine learning platforms like Kubeflow pipelines which are reusable, modular, can be shared across the teams. You can do hyperparameter tuning thanks to the cat component. You can serve your model right in the Kubeflow thanks to the case of component. Users can spin up their favorite IDEs. They can easily track their lineage of their pipelines and models and et cetera. The Kubeflow is comprised of different components and it also has integration with different machine learning frameworks like TensorFlow, PyTorch and very important is that it's abstract away the complexity of Kubernetes. So, to summarize, Kubeflow helps teams to streamline their work. It improves the collaboration. It has multi-tenancy support and as a whole, it speeds up the time for developing their models. The project is developed by numerous companies and now the project is doing its next step towards joining CNCF landscape as an incubating project. We hope that this will happen soon. I'd like to announce that Kubeflow version 1.7 has been just released. So, if you still haven't tried, please do. So, who is using Kubeflow and why? Companies from various industries are using Kubeflow like communication, finance, medical, insurance and especially the regulated sector. But also some hyperscalers are using Kubeflow like IBM, Google, VMware, DHL, Ricto and others. Actually, Googles are using Kubeflow as their default machine learning platform on GCP under the name Vertex AI. But why Kubeflow is a platform of choice for so many users? Simple, there is no similar open source machine learning orchestration alternative available. And another reason for adopting Kubeflow especially for regulated sector but not only is that it's vendor neutral, it's scalable, standardized and we could say fairly secure. So, we have the similar reasons for adopting Kubeflow on enterprises as running Kubernetes on enterprises. So, now let's say a few words about the newly formed security working group which technical leader is Julius. Its primary goal is to define a clear policies and procedures on how vulnerabilities should be reported and should be publicly disclosed. Another goal is to enforce the use of security-based practices like automated every API call, like use the least privileged airbag, do periodic CV scanning and now still in progress is the integration of software build of materials or S-bombs. Of course, these groups provide place for discussions during our bi-weekly meetings and in Slack. And of course, the main focus of this group is to tackle different security-related issues. In the following slides, we will discuss some of the issues. So, now we would like to share some of the insights we got from our first CV scanning starting from 1.7 Q4 version. So, this table contains information about the number of images used by a specific group and the number of CVs divided by severity. Luckily, most of these CVs here originates from our external dependencies and the majority of them can be addressed by just upgrading to a newer version. Actually, Canonical already worked towards minimizing these numbers and they have even achieved zero critical and zero high severity CVs. So, we're expecting their work to be contributed upstream. So, it's important to take care of your dependency. We automatically inherit all their boxing CVs. And now still work in progress. The community is doing its next step towards securing software supply chain by integrating software bill of materials or S-bombs. S-bombs provides an exhaustive list of all the software components within the project with their versions and their licenses in standardized format. So, both sides and once this file S-bombs file is released, both sides, the user's community and the contributor's community could benefit as it provides transparency. Both sides know what are the ingredients within the project, in this specific version of the project. And it increases the accountability. Based on these CV files, you could do a CV scanning and we could say that it enables us to do accurate identification and remediation of security vulnerabilities. And I forgot to say that ensures that the project is licensed compliant. Now, Julius. Yes, thank you very much. Is this working? Yes. Let's start with the architecture here and please, please follow closely because this is important for the remaining part of the presentation. So, we start with the architecture here. In general, you can see the authentication and authorization part within Kubeflow consisting of an Inverse Gate for a DEX as a ROTC provider, the internal ROTCR service and in general a service mesh. Actually, the service mesh as well as certificate manager is connected to most components within Kubeflow. I'm not going to cover it everywhere. But giving this authentication and authorization part, oh yes, amazing. Let's first take a look from the user side here because usually you have quite a lot of users, in this case user X which owns a project or namespace or profile, however you want to call it. Why do we do this interchangeably? Because we have these profile controllers here which actually pick up a Kubeflow profile custom resource and transforms it into a Kubernetes namespace, adds some default service account, role binding, secrets and so on. So, yeah, this is done for all of the users automatically. For example, I run some clusters which have several hundred users. Then, giving this, we have the central dashboard here in way which is connected to most of the Kubeflow UI components. So, this is the main entry point for the user. As for example, volumes UI which is quite simple. It just allows you to manage your own PPCs in your namespace because as I said before, this is really just a Kubernetes namespace with some additions. So, you can do all the normal Kubernetes stuff as well there in your Kubeflow namespace. But then comes the probably most used Kubeflow component which is Kubeflow pipelines, which adds some, quite some resources to the cluster in Kubeflow namespace and allows the user to run these reusable, modular pipelines we talked earlier about in their own namespaces. This is heavily used, developed by Google. This is their main product also sold as a vertex AI on the GCP platform. And then we have Kube, as I said before, for hyper parameter tuning here in green. And also quite heavily used, the workbench is formally called Notebox. As the name already says, you can start your JupyterLab, which is through your code, RStudio, later on maybe also LabelStudio and MLflow within these as in a self-service manner. So, you can just point and click, choose how many GPUs memory you want and start your online IDEs. And last but not least, there's K-Serve here in pink, which is used for model serving. I think there was a talk a few hours ago here exactly in this room from Bloomberg about K-Serve and how they use it. Yeah, it's using K-Native under the hood, so that means you get serverless inferencing, you get scale to zero and some nice additional stuff. And all of this, as you can see here, runs in the username space. The actual workbook is always running in the username spaces, so you can add your quotas and whatever onto it. Then we have the Kubeflow name space, which covers most of these components and is the system and authorization for these here. So, this is the big picture of Kubeflow, but it's not complete because, for example, I'm working with any scale on Ray integration, maybe you know Ray, the training platform for JetGPT, as well as MLflow integration, labor studio integration and so on. So, there's a lot to add to Kubeflow.  Now, try to remember, but I'm always going to show it again. So, for us in education, I'm giving back to Janna again. I'm not sure that everybody hears you. Yes. Do you hear him? Okay. Okay, so now, on this slide, we would like to show you the authentication flow of Kubernetes of Kubeflow. So, it starts with a user or a machine who needs access to Kubeflow. So, the user hits the ingress gateway, which redirects the request and asks another service, OIDC Auth Service to validate the request. Currently, we do support two kinds of authentication, session-based intended for humans, and what's new in 1.7 is that we do support service account-based authentication. So, OIDC Service validates the request. The request is valid if it contains headers with a valid token or a valid session. If they are missing or they are not valid, it redirects the user towards to begin a new cycle of OIDC authentication to get the session. So, once the request is valid and it has a valid session cooking service account token, the OIDC service also adds another new header called user ID header. So, and then the request is actually redirected to the correct Kubeflow component. This new header, user ID header, is later used by the Kubeflow components for authorization. So, before 1.7, the OIDC service supported only the interactive session intended for humans. So, we were supposed to simulate the web browsers for machines. And now with the newer 1.7 version, we had a programmatic way to authenticate the machines using the service account tokens. And now I will give the stage to... Actually, this was a quite heavily requested feature by the users. Okay. Actually, this was a quite heavily requested feature by the users because quite a lot of companies are using GitHub workflows or GitLab pipelines to trigger Kubeflow pipelines in turn. And they definitely need a long term proper machine to machine authentication. So, this now in 1.7, it's in. And let's continue with what we've achieved so far because I'm not only talking about the bad security stuff. I also want to talk about what we've done so far. And it's actually quite a lot because, yeah, we had some problems over the last few years. And this started with misconfiguration of Istio sidecar services and so on, which allowed you to easily fake the user ID header and impersonate to a lot of services which you have seen before in the architectural overview. And you could just fake to be another user, hijack their namespaces and so on. Of course, completely unacceptable in an enterprise environment. And, yeah, even the user management self was completely unprotected a year ago. So, yeah, you could just add yourself to any other user's namespaces, hijack it, take it over and so on. Yeah, complete disaster. Okay. But I spent some time on it with some other persons as well. And we fixed these issues upstream. Everything is contributed. We spent some time to harden Istio with security best practices, add additional sidecars and so on. And in addition to Istio authorization policies, I later on also added network policies as a second layer of defense. And actually, it turned out to be quite valuable and catch some additional Istio misconfiguration. So please, it's under the control folder. If you use Kubeflow, please consider using the network policies as well. So this is regarding Istio network policies, quite some changes, quite some improvements. And another area where we made significant progress over the last two years here are rootless containers. And as probably all of you know, it's still an unnecessary security risk. You should not run root containers nowhere, never if possible. Although some people still do it. So this is obviously even with Kubernetes user namespaces, which is something you might know about improving the situation about root and so on. But even despite such efforts in the Kubernetes Foundation, it's still a big problem. And it's forbidden by most company policies in Stain Enterprise environments. You don't want to have your users to run root containers. It's quite simple. But what's the default situation within Kubeflow? If you go to the Kubeflow website, download the manifest, install in your cluster and so on, then by default, almost all of your containers run as root, especially in the user controlled namespaces, which makes it easier to exploit your cluster, escalate privileges and so on. So this is something I wanted to graduate almost two years ago. One of the first things I tackled. So you made it possible to run 99% rootless. Of course, you're asking, where's this 1% gone? It's gone within the Istio C9 demon set because if you want to run Istio rootless, this is the only supported option at the moment. So you want rootless init containers, no admin rights there, you want to prevent root containers everywhere. You have to use Istio C9. This is where the 1% is going and you will get an exception for this probably at your company. And then it's not enough to just do this for all default containers, which are there in the Kubeflow namespace, in the Istio minus system namespace and so on. But you also have to enforce it for new containers that you probably do not know at installation time because your users can bring their own containers in their Kubeflow namespaces, their own containers for pipeline training, their own containers for inferencing, and so on. So you have to enforce this on the cluster level, on the Kubernetes level by using pod security policies. And nowadays, I think with Kubernetes 1.25, you can use pod security standards, which is the successor of pod security policies as well. So this solves the rootless container issue, but there's one limitation because some people do not use GitHub pipelines or GitHub workflows to build their containers or something else, but they actually want to build containers in their own Kubeflow namespace. And this is slightly limited because, of course, no legacy in Secure Docker, but even with Podman or Carnico, it's still not yet possible to build OCI containers rootless. This is the only limitation so far, except for this significant security gain. Please use this at your company. I'm linking the GitHub pull request there, and I'm hoping to upstream all of this in the near future. So the second thing, rootless containers mostly solve. Yeah, then let's go to the exploit part. Let's start with a simple example. This is just a simple RBAC issue. Most of you should be somewhat familiar with RBAC stuff. I hope you can email properly. If not, please raise your hand. Okay. Then, yeah, let's start again. We have a namespace called Alice, because I think this is common in the security sector that you call them Alice and Bob. So I took this here as well. So we have Alice and Bob here. And we have the Kubeflow namespace. And in the Kubeflow namespace, those two controllers, controllers run, which you've seen before in the architecture overview. As first of all, the profile controller, which takes these Kubeflow profile custom resources and transforms them into namespaces at some stuff, wall bindings, authorization policies, and so on. But then there's a second one. Not only this one, the normal profile controller, also the pipeline profile controller from Google, they were developed independently. So this one here as well, add some additional stuff just for Kubeflow pipeline to the namespace. As you can see here on this side. So now that we have these controllers, what's the problem with them? Obviously, as you would have expected, they both just run as cluster admin. As you can see here, we should never be the case, never provide cluster admin roles to any of your service accounts. It's just don't do it. Because then one exploit in the Kubeflow namespace and you can easily become cluster admin. That's the problem. It's just about hardening in this case. So what can we do? It should, of course, be a reduced cluster role. And if possible, talk to the Google developers to merge all of this into one profile controller, to reduce complexity and therefore attack surface. So this is a quite simple example, rather on the side, not that big impact yet, but we can get to a more sophisticated one that includes namespace sharing. Because if you have tried Kubeflow so far by yourself, you will have probably noticed that you can share your namespace with collaborators. We start again with Alice and Bob, in this case, malicious Alice, who gets added to Bob's namespace as a collaborator. So what does this mean? She also will have access to the service accounts here in Bob's namespace. And this is problematic on several levels, because first of all, it does work. Yes, you can access Bob's Jupyter Labs, Bob's pipelines and so on. Everything is working on the surface, but it's completely broken on the Kubernetes level, at least from a security perspective. Because first of all, they are configured in such a way that even if you only have access to the default viewer, you can escalate to default editor in several ways. Even with the Kubernetes 1.25 improvements, where the secrets are not created by default for service accounts, it's still possible to escalate. So this is the first thing, even in your own namespace, you can escalate your privileges to another service account with higher privileges. Should not be the case. And the other way more important problem is, just imagine Alice's malicious at a big company. So she just steals Bob's default editor service account token, leaves the company and impersonate Bob with the viewer token. She can just log into the Kubernetes cluster. I actually did this myself several times. So should not be the case. If you leave the company, you should not have access to any Kubernetes clusters anymore, just because you stole the service account token. And what can we do against this? For example, we can just disable sharing all together. This is what is done at some companies because I'm doing consulting, I know about it. And then there is as well a second solution, which is might might be called a proper solution to some degree, which means actually deleting, regenerating all of the secrets tokens and so on. None of this is implemented yet upstream. So please take care of your companies. This is a problem if you leave the default namespace sharing enable. Then we have sharing and staring, as I always say in life, be careful. So now there's another issue about multi-tenancy, especially the artifact storage within Hubfloor, which is implemented using Minio, as SD server. And here you can see it's used mostly by Hubfloor pipelines to some degree also by KServe. And we start again. We have Alice, we have Bob, and the namespaces. So let's add the Hubfloor pipeline components, which you have seen on the architecture slide before, here in blue. And as well as additional backend stuff as the pipeline API server and Minio. They used, especially Minio is accessed by most Hubfloor pipeline components. So what's the problem? It's completely shared. These secrets, Minio Secret 1 and Minio Secret 2, they're actually the same. And not even that, they're not just the same. They're also the admin secret for Minio by default. So any user here in your Hubfloor installation can destroy your Minio instance, can hijack other users, read all artifacts from all other users. There's just no default multi-tenancy. You just share everything and can destroy everything. So a year ago, this is why I spent some time on a POC to isolate the artifacts per user and in the end really have separate folders with proper IAM permissions, which you can set in Minio, such that the users cannot access each other's artifacts anymore. And this is actually in place at some companies. So I'm not sure maybe it's already an implementation and not only a POC, but in the long term, I want to get even better because we still have credentials there. Just imagine now we have randomized, separate different credentials per namespace, not the admin secret anymore. It's still annoying. You still have to pick them up as a user. And in the long term, I actually want to get rid of those credentials all together. I want to use the namespace origin here for authorization. But this is not yet implemented for the POC with separate credentials. There is a PR. You can click on it. Please use it at your company. Try to contribute back if possible. And last but not least, the main developer of Kubeflow Pipelines, which is Google, is allergic to AGPL. What does this mean? They just don't allow it. And the problem is, as you may know, Minio was switched around three years ago from AGPL 2 to, no, from Apache 2 to AGPL. So we have a three-year-old Minio image with quite a lot of CVEs, I would guess. Yeah, and we have to find a successor. I'm working on that, but it's not yet upstream. So CVEs in the image and the admin credentials shared for all users. Quite important. This is something, I think, which is necessary to fix if you want to deploy Kubeflow. And then, orthogonal to the artifact storage, there's also metadata storage for lineage lineage tracking of your artifacts, especially the metadata for pipeline one artifacts. It's also heavily used by TensorFlow. So the Google guys decided to also use it within Kubeflow. And we start again, we have Alice and Bob with the namespaces and a lot of components for this ML metadata stuff. I'm not going into detail about this, but the main problem is, again, we don't have multi-tenancy support. It's, again, similar to the artifacts, it's shared for all users. Alice can access Bob's metadata. So there's a solution for it currently. You can just disable this component of Kubeflow because it's quite modular. This is what I do at most companies. But the problem is, the newer version of Kubeflow pipelines might require the machine learning metadata part, and then it becomes complicated. We will have to spend some time on isolating it per user for KFP version two. I talked to the Google guys. They are focusing on it for the second half of the year. Let's hope they deliver, but let's see. And of course, we are looking for volunteers here, not just for this issue, but for any issue. I'm really offering mentorship. I did my third mentorship already. If you want to contribute, just contact me. I'm going to help for free just to improve the project here. And then to add some variety, we also have a frontend issue for you. As you can see here, just imagine, this is Bob's namespace. He had just the bottom of the UI.  And the TensorFlow training produced some output artifacts, as you can see here, or at the bottom. This is the S3 path, including a namespace parameter. Now, the problem is that if Alice somehow manages to spy Bob's S3 artifact path, then the UI allows Alice to actually read the content. How does this work? You just remove the namespace parameter here, enter this in your web browser, and the UI will not check permissions. Someone forget to add the permission check. This is still the case in a default installation from Google. And there's also some additional technical depth that, for example, yes, we skip the API server, we access Minio directly, or this artifact proxy, which you've seen before on the slides. It's also rather obsolete. So there is some stuff to be done on the, especially in Kubeflow pipelines regarding the artifact storage as well as the metadata storage. So then there's also an additional denial of service attack. And I know some companies who are really affected by this, you can block the usage of the cluster for, at least the database and therefore KFP for most users. I'm not going to explain it here. Please, if you want to help us, fix this, it's just SKL stuff, join us, and we will find a solution for this. Yeah, so as a conclusion, we achieved quite a lot over the last two years, like authentication for most API calls, lower privilege RBAC. We also found that security working group, including the just recently released image scanning numbers, software bill of materials, the new machine to machine authentication, distio improvements, network policies, and of course, we'll just contain us. But yeah, on the other hand, we still have some open issues, which I've just presented here from the profile controllers and namespace sharing, which are mostly RBAC related to the multi-tenancy support within Kubeflow pipelines, the artifact storage and metadata, yeah, does not support it yet properly. And we have provided some solutions to some of those issues here. Yeah, this is what I would like to summarize. And of course, please join us. I said several times before, we're looking for volunteers. You can join the community, there's a community calendar, you can join us on Slack, quite responsive, there's a security channel on Slack, and we have the security working group meeting minutes, so you can even see what we've done so far in videos. Yeah, this is our contact information. Please do not refrain to contact us. I will try to answer you as fast as possible, and we would really like you to rate our talks such that we can improve on your feedback. So I know this is quite a lot. It's not an entry-level presentation, but maybe some people already have some knowledge regarding... Okay, maybe some people already have some knowledge regarding Kubeflow, want to get a more in-detail explanation of some of the issues I've shown here. Any questions so far? Yes, please. Is there any work going on isolating or like managing security of the notebooks? So let's say if I am in a notebook, I can authenticate in the notebook via my credentials. And within the team, any member can have my credentials. So that is one of the security issues we are facing in our company. Yes, if you share your namespace with other users, then of course they will have access to the same stuff. I wouldn't call this an exploit. So if you really want to separate this, maybe create separate namespaces per user and one big shared one. But it's not really an exploit if you explicitly share the namespace and then expect some stuff to not be shared. I mean, I understand regarding the exploits of default editor, viewer to editor. This is clear, at least that most of the companies, I know they just have per username spaces and if they then work as a department within a project, then usually these departments get their own shared namespace where all of the department users are inside and they know they are sharing stuff. Okay, thank you. Yes, put your hand over the microphone. Thank you very much. Cool. Hi. So I was wondering, did you ever consider integrating with other open source tools like gatekeeper, OPA policies? So for instance, the way KFP pipelines work, the V1 version is that you create like a workflow and then it gets turned into a pipeline execution. So one way I solved multi-tenancy is that I build an OPA policy that prevents running other people's pipelines. Can you get a bit closer to the mic? Oh yeah, sure. So the way I solved one of the multi-tenancy issues is that in the previous versions, you would be able to run other people's pipelines. So I built a policy that prevents you running other people's pipelines only in the namespaces that you whitelisted it. So yeah, I was wondering if you considered integrating with OPA policies and gatekeeper. No, not yet so far because I think if I understood your question correctly, then you are talking about namespecary. Like pipeline execution. So you can, in the previous versions, you were able to execute other people's pipelines. Ah, yes. Okay, now I know what you're talking about. Yes, this is too much to cover here, but I can tell you a lot about it. You had these, I think even in Kubeflow 107, you still have just the shared pipelines displayed in the user interface. Yes. And also on the Minio level, they are stored and you can access them even there and you can probably also access them in the database. Yes, that's still possible, but in Kubeflow 108, I think this is solved. We now have a proper UI which separates between shared pipelines and private pipelines. I can also, if you come later to me, I can tell you about the exact pull request. This is actually solved. The only thing that is missing is the KVP SDK support, but now we have really in the UI, we have a separation between shared or global public and private pipelines. And I even had my own implementation some time ago, but nowadays it's really fixed upstream. Okay, cool. So I'll look forward for the one point. Yes. If you want to help regarding the KVP SDK stuff, feel free to volunteer. Sure. I will. Thanks. Hi. Thank you for doing all the work. It's very useful. So thanks and also for the presentation, of course. My question is regarding the CVEs. You showed a quite scary table of vulnerabilities and you were talking about how they were getting solved, which is very nice, of course, but I was wondering, is this also going to be part of the release schedule that in the future CVEs won't be introduced into new versions? So we can use it directly? Yes, this is exactly one of the reasons why the security working group has been founded. And one of, actually it was mostly done by Diana here, who did the security scanning. Yes. And we are especially working on fixing all the CVEs. I talked to Canonical, they have their booth here. They fixed most of them and want to upstream it. And as said before, it's mostly outdated base images. So I think we can really get the high and critical ones to almost zero. Yeah, that's very good. The problem is actually it's scattered about over so many repositories. You have Kubeflow manifest, which pulls from Kubeflow pipelines, which pulls from Kubeflow slash Kubeflow case of and so on. And it's quite difficult to you have to manage with all of the separate working groups. You have to create separate pull requests per working group to update their base images. And then most of the CVEs should be gone. But it will take some time. Yeah, of course, I understand. But thanks anyways for the work. By the way, from which I would like to know from which companies you are who are using Kubeflow. If you could tell us, are you using it privately? Oh, sure. No, we're not using it yet. I'm from a local bank here in the Netherlands, the Volksbank. We're not using it yet, but we were looking forward to being looking at implementing it. But this was a very important talk, of course, for us because well, you can imagine why. Yes, I can imagine why you have especially the regulated sectors as I said before. Yes, it's doable. I know some quite quite a lot big companies, not just the ones listed here, but even way more companies, insurances and so on. And you cannot imagine how many people are using Kubeflow, but they're not contributing back. Some companies are really not allowing their employees to contribute back. So maybe this is something you should also check first, whether you are allowed to contribute to the project if you want to have fixing those issues. I think that wouldn't be a problem. Having enough time for it would be the major problem, I think, because most people have. Okay, do we have any further questions here maybe? No? Okay. And I guess we're also over time a bit. So if you want to talk about anything in detail, just come up front. I will be here for 10 or 15 minutes in addition. And thank you very much for your patience. I hope this talk was enlightening for you. Or entertaining.