Okay. All right. So it is 702 right now. So we're calling to order the meeting for Ward 8 NPA day being over the 26th at 7.33 p.m. If there's no objection, the meeting will get started. I will start with as always introductions, starting with a residence and followed by the available who's not from Ward 8. And then we will go after that to Sarah Morgan from Burlitt to Denver and the code presentation, followed by public safety update from Councilor Dardi, followed by Councilor King again and Councilor Dardi, Councilor King for Ward 8 and Councilor Dardi for the Eastern District updates and then the school board by Gary Golden, Commissioner Gary Golden. We'll go in that order. So I'll start with myself. My name is Romeo Von Herman. I am a Ward 8. Well, now I live in Ward 6, but Ward 8 State Committee member as well. I'm a Ward 8 State Committee member. I'm also somebody who works with the Green Man Transit, Stations Services Supervisor. I am also a commissioner at the Church Street Marketplace. I live in University Terrace and I'm glad to be here and be part of this important meeting. We'll go to my left and then we'll just go around. So I'm Bill Church. I live on Bradley Street. I'm close to go. I'm going to go north end, but I'm going to be in public aid. I'm Tim Dordy, and I live on London Square with Sam. I'm Sarah. I am there in the opposite city of Vinnie and I live in Newark. I'm also there. Thank you everybody. Do we have anybody online? No, I just got out of the room and Sarah. Okay. Thank you for that. Is there any Ward 8 member who would like to extend that invitation to speak at this time. So I've been, I've been thinking about the function and the practical application of Ward 8 NPAs to help the city of Erlington be the place that ought to be or is. And after me, what the role of the NPAs has been lately. I think that maybe a discussion around that at some point. I see useful on how this could be a. My understanding of the history of the NPA was that after when Bernie was mayor, and they won by a very small margin. He wanted to have these NPAs. It was both a grassroots democracy effort, but also a listening device right that he could identify issues that were coming up in smaller community groups that maybe the city counselors were not privy to. And I think that's a good thing. And as I've talked about in other meetings, I think that politics in Burlington has polarized enough that the NPAs can kind of really offer a different perspective on how we can just come together and be respectful and meet each other. And, and good things can come from that. If you put a bunch of well intentioned people were willing to do some work in a room something good will come from that. And that's kind of what been my vision of what this NPA could be. But I think that through redistricting, and through more of the political forces that war date has been kind of depleted of that. There's so many college students who are transients that the perspective of their isn't really the medium ledger here. And that if this is going to be the student ward, then maybe that should somehow be turned over to that, you know, the Asian be more their voice, so that these accomplishes what I think the NPA is which is something other than the political process here in Burlington. Anyway, if anybody comments on that this is great. Bill, anybody who wants to thank you. I mean, I would reject the notion that board aid is just a student board. I mean, and I say that as someone that moved to Burlington or school and then stuck around after and then actively ran for council I was while I was still a student and I think that is a way and we've seen it work over the last couple of years for our longer term residents or folks that aren't student age, or traditional student age, and students and young folk to be involved in the process together. That can happen and that does work and I think ultimately like we can see, like we will be better neighborhood in better community, if we all are willing to meet each other halfway on that and I think it will take takes a lot of work and I know that there are a lot of people out there about younger folks in our neighborhoods and the fact that a lot of individuals just come here to go to UVM and then end up leaving and I think they end up leaving for a lot of different reasons whether that's, they can afford to stay here there's no jobs, the list goes on. But I don't. I know when I am talking to folks around Ward 8, whether it's knocking doors or meeting neighbors, it is a wide range of folks that are older that have families that are in their late 20s, early 30s that are really establishing themselves here and that have picked Ward 8 to start growing and setting roots in Burlington and then yeah I mean you have your 18 year olds that are running around the ward and just like living on Mule Street. Ward 8 is not the only ward in the city also that has students I mean their central district has a large number of students as well and I think that they are able to coexist and work together as a community and so I know that it can be really frustrating and I do agree that we should be able to have an NPA where young folks can come. But that also our long term neighbors feel comfortable in too and so I think we all just need to be willing to do that work to work together, but my only comment on that. Thank you, Councilor King. Okay, and moving on. The next subject matter is the Burlington neighborhood code. We do have Sarah Morgan and Charles Charles. Thank you for calling to be made here. No result. Awesome. Thank you so much for having us here tonight. It's great to be here over in morning we're here talking tonight about the neighborhood code. It's the intent of the neighborhood code is to build on the strength of existing city neighborhoods and I'm kind of sure several of you have heard us discuss this before so it might sound a little kind of sounds like you just heard about this yesterday morning. So, but we'll dive a little bit more into this and how many of the questions I think. So, Burlington neighborhood code build on the strength of these existing neighborhoods, evaluate zoning tools that regulate new homes in areas today, and also identify opportunities for new neighborhood scale housing citywide. New neighborhood scale housing in this context refers to certain housing types such as deep boxes for boxes in cottage ports that are not currently able to be built in a lot of Burlington's residential areas today due to district and zoning regulations and this project is really about how he's having catching fit alongside existing buildings within these residential areas. So we've called it a number of zoning changes over the last few years related to housing that many of which I'm sure you're all familiar with, and this these changes with the neighborhood code will help us to lend our plan for how we use cities limited land area in the future. And you're all very well aware that most of the zoning changes related to housing, or most of the zoning changes recently have been related to these areas in gold on the left. These are areas that are planned for this most intense future for. However, not a lot of not many of these areas can or will accommodate future housing, and not all of them are intended for residential development. But an area in particular that has been focused on a lot for intensive high, like higher level residential multi family growth is that's area in downtown. And the area that focuses on the area that the neighborhood focuses a lot on is this area in blue on the right, that makes up about 40% of the city's land area. The neighborhood code is really about how we can allow for neighborhood scale housing types in these. The city's existing residential neighborhoods and how they, that these can allow for unique and interesting and will small scale multi family housing side by side with coinciding with single family homes that are in these neighborhoods. As residents, we have very strong connections with our existing neighborhood and the existing power of our neighborhoods. And this proposal recognizes that these inputs can and should be places for residents to that will meet and only the needs of residents of Burlington across across their way spoon and needs the challenges for our chronic and emerging housing crisis and also has tangible makes tangible efforts to address the existing, the existing climate concerns. So this, the neighborhood code also will increase housing opportunities throughout, then throughout Burlington 30s neighborhood scale solutions by requiring less than for home, utilizing existing infrastructure that's already in place in these neighborhoods, more housing, increasing the housing type choice and having potential socio economic integration in existing neighborhoods. A lot of more opportunities for multi generational housing and aging in place, as well as expanding tax base and sharing that tax for more hospitals. This also helps confine with not home after 23, which introduced new statewide zoning requirements for duplexes traffic system for unit buildings. Now, these are the types of housing that I mean when I talk about the scale. And these offer new opportunities for homes in existing neighborhoods and can make more housing options available across the individuals. I'm already, for example, like we talked about here tonight already. I student young adult, young professional young families, older adult all have very different needs, and with these different housing types, these can meet those needs across these changing needs across that lifespan aligned with residents to continue the age and place. And one of the reasons that ARP is really interested in these types of housing solutions, and I would be unable to build a strong partnership with them through this, much like accessory dwelling units, these housing types offer another tool for older adults to be able to remain an agent base in neighborhoods, and all and have options for more accessible and potentially lower maintenance housing types. And then you might see examples of these housing types across the city and different neighborhoods, especially in order to be able to build a strong partnership with residents. So the three of these are not legal to build in the city's residential areas for an example of this would be the regulation of duplexes. These are the one of the lower forms of density that add to add to more housing types. And despite policies that we have in place that in theory, allow for duplexes in practice they aren't able to build on the majority of density residential neighborhoods today. Because there's a minimum lot size requirement and density limits that work together through within the residential low density zones that limit that require nearly nearly a quarter to acre to build a duplex. And while the median lot size in these areas is seven thousand seven thousand square feet so a very difficult to build a building flex in these low density residential neighborhoods, and you can also see in these maps on the map on the left, the areas in green show where you are able to build a duplex and all residential neighborhood, residential neighborhoods in Burlington, and you can see that the places that it is possible to build this low scale density. So either in the poor downtown in this like high density residential area or in the more suburban neighborhoods that are less served by transit and are less walkable and less accessible generally. So Burlington has been regulating these standards since first adopting zoning back in 1947 with the zoning standards adjusted nearly every decade just resulted in those that don't necessarily reflect what's on the ground today. And for example some current low density residential zones are in a way miss zone, and might not not what's on the ground frame of that fabric of that neighborhood might not really necessarily match up. So this is a good example of what it is currently zoned for. A good example of this is the density standard that's in place tonight so dancing women's regulate how many units can you go based on the size of the property. So on the left here you see four different areas located in the old north end within a block of each other and the two on the from the street. They feel very much of the same scale with the two on the right are not conforming to the current existing density standards for that neighborhood, and that's measuring a very arbitrary way that's not really easily perceived from the street of dwelling units or a third. So it's the lot size and the number of units within that structure. So this is an example of how the current zoning regulations and limit future housing opportunities, for example, lots are often too small to utilize a sustained and unit development savings that will allow to develop a sort of condo or rental community so this is a, an example of a cottage court that could be built in in the New York and if we adjusted our standards to allow for middle housing types. So this is a cottage court or multiple detached structures, located on a lot that have a shared green space and it's shared open space. And another example of a housing of housing types that are not able to be built would be this example across from the Hanover on North Avenue, which is a complex energy flex. However, again, not able to have multiple freestanding lots on a single local freestanding buildings on what. So essentially what we're looking to do is how we evaluate how we can allow for these various housing types across several of our across all our most preserving districts today. So what would enable for up to four unit city line for providing for flexible for flexibility and older women's and especially along these major corridors, as you can see, going from these three load and three high density median density load density to four zones. That would also refrain it include major corridors. And the standards that we're looking at are regulating building mass instead of density, which includes looking at max building both friends height and the number of stories guiding the overall building mass, allowing for smaller lot sizes, a smaller setback and higher cover, which would import, which would inform the building location and overall development intensity, enable smaller loss as well as flexible subdivision, which would allow for greater utilization of existing loss and the creation of the desired housing types and the enable creation of college sports and outcomes, which allows for more variety of housing types to land cost and as well as potential ownership models. We've been looking a lot at different neighborhoods and have been considering a lot of these housing types in a way that really doesn't reflect that existing neighborhood pattern and context. And I encourage everyone to get involved in this process, but we're, we're currently working through a joint committee process, which is pending commission as well as members from the city council, the city council ordinance committee. And we have just wrapped up I think our third meeting on Tuesday night and we'll be meeting again next Thursday, as well as November 28 and the 19. We're also staying coffee chats with a rp with two more are found to coming up number 13 and 14. And this is our last meeting for the month. And thank you for having us. And also I want to invite you all to our virtual community meeting on November 13. It's really an opportunity for us to share with the public what we have discussed within the city council ordinance with the city council ordinance committee and then commission and share out what what this proposal looks like looks like after this process. Thank you. Thank you very much. We'll take questions anybody that have questions we have. My seven third and we should be done with questions and answers. Hopefully. You back your first slide. So a couple years ago the same room in paid $75,000 neighborhood plan to understand how neighborhoods could be preserved and strengthened within the city of Erlington that you look at that document at all that you develop. Sorry. I forget what it is. Adam Ruth had the money for it. I, my wife would know I can send you that. Yeah, obviously you did. So this term neighborhood. Can you just find that. I don't mean to be a trick question. I mean, to that point, I think there's different definition. My definition of the neighborhood is a fairly street and a viable. Physical area is now that contains one or more. Communities that self identifies in some way. People, people. This is really a housing. I would not. You know, I think that people living house. So within a body of land that you're going to build houses for people. There are roads, people come and go. There are services. You're trying to attain a certain diversity. You're trying to support schools by having children in the neighborhood. I think that neighbors that can take care of each other, but they do a more day. There's, there's parks. It's more than just the buildings where people live in. I'm not opposed to housing plants. I just don't, I think I'm the calling it a neighborhood I think is branding that I would, I would question. And the thing about war days is that because it's been designated high density. In the past, anything went as long as you put it in more structures. You know, in the middle of the park.  Damn the drug houses.  That isn't considered when you're trying to do neighborhood design, you're just trying to put more housing. And so I've seen and heard a lot of us through the years. We need house, but is it going to be only to make the higher density neighborhoods more desk? Because that's what it used to be. No, this is a great, this is actually a great question and I think something that every city and let's, you know, let's be honest, you know, every city in the country is is or has done an iteration of this kind of work. And the cities that have been most successful have been the cities that to your point, have tried to focus it, not just on housing. That said, we have a lot of projects going on in our office right now and simultaneous to this project and you're doing a project called the city wide transportation option study, which is all about making it easier for people to move around the city without having a car. The parks department is always living so we get right now. So our vision is fewer cars, and that's fine. I'm not good. Well, let me let me just finish my what I was saying is the parks department is constantly doing planning to understand the capacity of city staff like many businesses. Lower. What's the impact of density on the neighborhood. Those and cons. From a tax base, a fantastic shoulder from a livable livable scenario. So, I'm just asking that some of those things be considered to those of us who live in high density neighborhoods. Yeah, and I assure you that this work is not going on. Well, you're talking in the presentation so you updated the presentation as being presented to reflect concerns left mind. I think I heard a certain language. The warning is a high density neighborhood. Right. And also to that point, I think four day, for the most part, be affected by too many of these. We don't have a part. We don't have a party. We have been more than maybe two students. It's a little bit. Nobody seems to realize that we have a trash problem to track. Maybe the plan, maybe your idea to build more housing will address some of those issues because otherwise we're just going to pack more people into the small thing. Well, like I was saying, I think a lot of the recommendations about this study really want actually affects 40 too much because board is already either within some of the form code or residential high density district already and I think most of the projects coming out of the study are going to affect those properties that are currently some residential low density and residential medium. Last question. At the end of this one of the big delimiters in in society is who owns a house. And the fact that private equity has taken over majority of a lot of the housing people no longer own their own homes like my parents did. They didn't have the equity from that house at the end of life. Is this going to help with home ownership. How's it going to be financed with the financial mechanism for these is it going to be only property developers. Well, I think you're alone programs a lot of that question I just want to make sure Romeo that you know where you are getting close to 730. As I understand this now I've had the benefit of seeing this issue several times. I know how it ends. You know, as I understand it, you're asking a lot of questions bill I think about what's final product and all the way. They're what they're just sorry. What are wonderful folks in the city are describing is a process right this is a process by which you are trying to identify places that are appropriate or we're not appropriate or to be feeling different claims and housing that right now, because of the way our zoning structure you're set up, may not be. Right so they're describing a process they're not saying, or it isn't going to have, it's going to happen or it's going to have more of why this point, they're describing a process and it's a process that is. If you if you listened to that piece of it, the end is heavily engaged is heavily oriented toward public engagement so that people have a voice, right and this is going to the NBA is just back to your last point about what's the purpose of the process like this is only one part of this process and I feel like this is designed specifically to avoid. Hopefully at the end, anyone feeling like the zoning changes that do result from this process. I've been rammed down anyone's or done by some needless bureaucracy, and I just sort of want to call that out. And I have questions you might just be not asking them at the right, but you're asking them what the results are going to be before they've done the work, but that's why you would do anything. What's the plan, where are we going. I mean that's a pretty question. Well, how are we going to get there. I don't know the best first question I mean the first question of where we're going presumes that they know where they're going they're doing work to figure out where the best angle is, and that what is engaging the public. So with your, your question on process I would like to see how this evolves and to be safe. Clearly, this is what we've learned this is how we modify it. Sometimes and and this is prejudice on my part, it's crammed down our throats, we get developers, they're going to make money. There's nothing in it for the neighborhood. You know, they're, they're, they're going to take resources that could be used other ways. And you have the opportunity here to say, these are tangible things that we have changed in because of what we've listened to otherwise you're just song identity.  I'm out of code. I have a single family home in Bradley Street that's out of code. I do want to just quickly show that this on our website. We have a very comprehensive amount of information on here, as well as a 35 page document showing how we got to rationalizing why we are needing this middle missing middle house tonight, as well as a, all of our documents related to the joint committee process. We have a lot of presentation materials, you can go by date, viewing the slides, as well as recent no most links to sit at third where you can watch the reported meetings and follow along with the conversation and pretty comprehensive Q&A document, as well as just a lot of different diagrams showing what these different recommendations are looking like. So I highly recommend checking that out. Thank you Sarah. So we're going to move on to the next subject which is Public Safety Update from the Council of Dirty. Any additional questions, certainly if I could contact you. Yeah, I was just going to add, yes, if anybody in Bill happy to talk more simple. I want to see you guys see. Yeah, I'm really good. Thanks to take a call email. Have you come into our office and we'll chat that goes for anybody so please reach out. I think Sarah's managing this project with Megan Fuddle the planning director. Sarah's information is on the website. Thank you. Thank you guys. Thank you both. Thank you for your patience whenever you're ready. So thank you so Romeo asked me to do a short update on sort of public safety issues. I think because I'm on the public safety committee, or maybe just because you just left here. I don't know what it is. But I am on the public safety committee I serve on the public safety committee with counselors grant and city council President Paul. Not this most recent city council meeting but the meeting previous to that. It's a resolution on declaring the drug crisis, both a public health emergency, and a public safety emergency. In the process, you know, I'll give you an overview of the resolution. And what it does, I want to make clear to whoever, you know, that this resolution is a step, not an end in of itself the resolution doesn't itself make policy. It is a step and I'll give you at least my perspective on why I think it's a good step and an important step. The resolution has three components, roughly components for components, I guess I'd say one is is that it puts a permanent spot on every city council agenda for a space permanently on every city council meeting agenda in which we're going to talk about public safety and public health. So it's we're forcing ourselves to address this at every meeting. The second key set that the resolution contains is a number of requests to Montpelier. And these requests include the speedy timely release of opioid settlement money, which, which as I understand is in the works but the time that the resolution was passed was was being held up. There's many millions of dollars, a good portion of which we will have access to, and desperately need and and calls for urgency on that but it also asks for other things to. It asks for more judicial resources to address what we understand to be the backlog and criminal cases. It asks for more prosecutorial resources. And, you know, you'll calls upon Montpelier to help Burlington address what I believe is not just a Burlington problem which is, but is really a regional, if not statewide problem. A lot of which manifests itself in our state's biggest city. And I hope the message behind that and that we will use the resolution as a sort of springboard to continue advocating for this is to Montpelier, which is, you know, the state has a stake in the success of Burlington and its biggest city. The state should not want Burlington to fail or suffer or be driven to the brink because of public safety problems. The state has a role, a very serious role in playing by here in addressing this most pressing issue. The third piece of the resolution reaffirms the city's long standing commitment to addressing substance abuse disorder, which is, which is related to although not entirely overlapping with the public safety issue but we all see that there's an intimate connection to those two issues reaffirming the city's long standing policy commitment to addressing substance use disorder as a public crisis and advancing harm reduction policy approaches. It doesn't make policy it doesn't, you know, the resolution doesn't, for example, you know, set up a safe injection site or set up a needle exchange, but it clarifies that we have this policy approach and that we're sticking with it. And I think importantly, and this was, this was something that felt very strongly in states, perhaps more clearly than has been stated at least in recent times that harm reduction as directed toward substance abuse issues is not standalone solution and cannot operate without also addressing public safety and crime and crime and issues related to criminal justice system. So it says both of those things as a matter of public policy. And then the fourth thing that it does and this is important, again, I think, you know, going to Bill's question about it is engagement. It sets up a series of public forums in which we are inviting various stakeholders, both professional and community stakeholders to public forums to address these issues so the state's attorney's office, the United States attorney's office, the Burlington Business Association and a variety of folks who are going to we're going to be asking them specific questions about public safety, public health issues downtown. A number of people have expressed to me some frustration and said, you know, the last thing we need is more meetings we want you to do something about it. When I go downtown I see access to why this is a crisis, do something about it, don't have more meetings and that is a sentiment that I identify with, and also feel, I will say this. It's clear what precisely is to be done to quote unquote fix our public safety crisis downtown. And I do think we need more information to assess what not only are the sort of call them big picture issues of housing health. Not again justice, but I would say more short and medium term issues, more short medium term solutions. So for example, one of the things that we have heard, but don't have direct information on at least I don't have direct information on is this backlog in the system in the we've heard that this contributes to some of the public safety issues that we're seeing, but we'd like to hear more from the actual professionals will work in that arena. How big is the backlog. What do you think the solutions are to the backlog. What are your proposals for ideas. Can we get some of our retired spirit court judges to come back to work is that something we should be dishing out earlier if you know two more judges sitting three days a week would clear this 1000 case backlog worker. Maybe that's what we should be banging the table and asking for. Do we have enough prosecutors. We all know that we don't have enough police officers, but we also all know or we should go and I'll tell you that the chief is working incredibly hard to rebuild the police department and I think doing an excellent job of it, but nobody should be under any pressure that we are going to wave our magic wand and get to a full capacity right size police department in the next six months because it's not going to happen. It's been a while to get us to a right size police department again, and we can't wait for that to happen. We have to think of, you know, other ideas so is it is it more cameras to be invested a lot in downtown cameras where are the cameras where we put them, we loan them out. Can we explore our no trust pass rules. Can we invest more money and outside law enforcement resources is that an option. I mean, in order to sort of figure out the short medium term solutions, I feel at least if I think Councilor King agrees with me and a number of people in the city council agreement made. There are some people that there are people institutions that we really need to get some information from directly and then I think the public should be part of that process too, so that we all have a better understanding. But what are the elements that make up what we see every day down right. Why do we see people drinking downtown without consequences in places where it is illegal to not why do we see people shooting heroin or fentanyl downtown on a daily basis when it is illegal to do that. Why do we see people harassing people on the street when it is and when it is illegal to do that. Why are our downtown businesses being pushed to the break by retail fact when it is already illegal to walk into a store and take things without paying for them. What are we seeing like I did, you know, two days ago, just open brazen hand to hand drug deals when it is illegal to sell control substances already. We all, we all can see the problem, but why is, why is this happening now when it when it wasn't happening three or four years ago what has changed what are the differences. If you ask five people you get sort of five flavors of answers on that I for one take these public forums are going to be really important and I hope that we're going to work hard to structure them well, so that we all come out of them with a better understanding of okay, there are four or five sort of issues here. They're not not going to have perfect solutions but they may have short and medium term solutions that would be similar rates of the index that we're seeing incredibly negative that we're seeing in our community. So that's the resolution. And I am, I don't know, the counselor came whether you want to comment on it but I am hurting by the direction that we are, we are moving. It is not going to be perfect. I think that was really good overview. So, my question is, when, when do we start again, yes, we don't have dates. I think right now we're looking at the 12th, the 13th and the 14th of December. But the public safety committee is is working on that we're getting dates to people who want to trigger the right people there, right organization. Those are the days that I think we're steering forward now. And another question, because these organizations will be invited to these forums, would hard questions be put before them to answer such as the amount of needles that are provided to folks that are already having open use of needles out there. I mean, they have to be an exchange program, but in truth, they're giving anything but to drive this out.  Yeah, so I'm hoping that, you know, these questions will be asked directly to the stakeholder. Hey, this is what's happening. He said the question is that we have all the time. The truth is we do not have the mechanics of this planned out yet. We're working on it and the goal is to have questions like that put to the right people who can actually answer the questions so that everybody can hear the answers and, you know, right like, what is the criminal case back load right now. Like, you know, how long do you think the present case will take to get through that. What impact is that having on public safety in your professional opinion. I'm not a prosecutor, I'm not a law enforcement officer.  I don't have questions like leader catch community diseases, but, you know, who is, who is responsible for the downstream negative effects on us. Do we think that we are encouraging people to come to the downtown in these contravenous drugs by providing them all the facilities to do so. Is that an unreasonable question to ask. I don't think it is. I don't think it is. So the plan is my vision for this is that we are going to have these facilitated in a way that those questions are asked and people get to hear the answers in my hope that out of that is going to blow some meaningful resolutions from the city council that will address the short and medium term solutions that I'm forward. We're going to get seen a little bit more clarity. You know, it's a weird situation because of course, as I pointed out before, everything we're talking about it's not like one, the city council really has the power to criminalize conduct sort of unilaterally and they can criminalize certain violations of the city or the citizens. We talked a little bit about this with respect to changing the review or for example, but you know, all this behavior that really is acting that people have come to us playing about the sea and my children see on the way to school. You see, when our businesses are being driven to the brink of it's all already legal. It's not like we are, you know, people say oh we want to criminalize this you want to criminalize that it's already illegal to walk into a store and take things without paying for it. So why is, why is it happening. Why is it being enforced. Well, or is it being enforced, and is the enforcement. Is the enforcement not translating into outputs that the rest of us can see when you. This is a three foot stool. There's compassion. And I think Burlington is probably the most passionate city I know, we all want each other to be successful to live safe lives, not be disturbed by never being. There's the policy which you referred to its housing is mental health services. It's a drug system. It's a lot of things that we can address with policy. Then there's a legal issue. And it seems to be the best one we don't want to use. Why isn't it okay that somebody get arrested by smoking near a school, but they're not going to stream up somebody who's using drugs. It makes no sense to me. Why is it that there's not a law enforced that anybody using drugs in front of a student at any age. So we get arrested and removed from the situation. So I, you know, I think that people are asking those questions. And well, how's policy going to address that. Well, I mean people are asking those questions, but I don't know that we've had, I don't believe that we've had an actual sort of an actual sort of structure in which questions like that can be directed by the people who actually know the answer. So I asked questions like that all the time.  And the police will sometimes tell you that but then they'll say, well, you know, we're not arresting people because for example, you know, under those circumstances because, you know, we now have 67 officers we used to have about 100 or decades we have about 100 officers now we have 67. And that's one of the things we've done away with is our street friends, you know, we had to deal with because we don't have no bodies and street crimes used to handle that. Okay, well that's one answer. Another answer is that you hear people say is well we're not the police aren't arresting people because they don't expect people to be prosecuted. Then you have other people say, oh, no, the prosecutors are prosecuting people but they're held up because of, you know, these extraordinary backlogs that we see in our court system. And that means for police services that they're not, you know, exercising legal options. It makes no sense. I don't know what you mean by that but well they have an excuse, you don't have enough officer. It's been several years since it was defunded. We have to wait several more years. I mean the problem is service you've got to, you named it. You know, this is a crisis.  That's an excuse. Well, you know, again, you know, I don't think we're going, I don't think I don't think the city has really set up a process by which the right questions can be asked in a public way to the people who have the answers.  Why not enforce law sovereignty? Okay, you know, that's kind of basic. Well, it's kind of basic. It's also kind of over something, right? You know, so you can, but you can ask the state's attorney, why aren't you enforcing the laws? You can ask the police chief, why aren't you enforcing the laws? I think you're going to get, you know, a more nuanced answer. And I think I'm hopeful that dialogue will lead us to point us toward to at least some short, medium term sort of policy changes that we can make that again, and I'm going to fix this problem with a wave of a wand, but that are going to hopefully make things on some of these fronts a little bit better. All right, so thank you, counselor. You're welcome. We're going to move on to the next subject, which is, you know, counselors updates. Safety, is there any. I spoke so much. I would say the big things are I just had my transportation energy and utilities meeting and we started our conversation about the bike share program and the bird bikes and we'll have our extensive update on that. Next month and I plan to raise concerns about how the bikes can be left everywhere. I have HR committee tomorrow and we will be hopefully passing new policies into the city's personnel policy manual. I'll do a post on from porch forum after we do that. And I'm happy to answer any questions about any of our last council meetings, I feel like that will be easier than talking for. I don't know if you have any big updates. I mean our last two council meetings will walkers, they both topped out over seven hours each so. No pizza, no beer, no, no, we don't even take that from breaks. Crazy. We can, but yes, and I know that but. So yeah, we've done the last couple of meetings but jam packed so happy to answer specific questions. You think that's a good sign you've been to us a lot. You know, I have wondered that, you know, you, you started off the meeting by asking a question of the role of the NPA is, you know, one of the things I've thought about is whether or not the length of these meetings the volume of them does hinder more than being actively involved and serve as a barrier toward people wanting to run with me on the city council, because it really is. You know, it makes it and I hope this doesn't come out Sunday is in the plane. The reality is it makes it less you. It limits, it limits the categories of folks who can really do it. And I think when we talk about diversity and the city council like I do think that there should, I think there's a benefit to having people on the city council who work for a living. I think there's a benefit to having people on the city council who have kids in the public schools I don't think those are the only criteria by any stretch of the imagination. I think there's a benefit to it and it, you know, you know, it's hard right like the city council should not be made up of sort of independently wealthy people or retired people or students or professional activists. So what level of satisfaction. Like on a scale of one to 10. So to the last point, like, I think that Kim is correct that and we definitely talk a lot about this independently because we both work full time jobs, and I work two jobs and then also do council and I was thinking about it earlier today and I think by the end of the week, I'll probably clock 35 to 40 hours on council just this week. If you're working a, you're working service industry jobs like meeting a schedule like that is impossible, but I find it difficult and I have a very flexible full time job and so I think we do need to do better and figure out ways to make that work, especially for folks that have families, like it, it does get really hard. It does inhibit the ability for people to engage but I also think having a seven hour long meeting it there's pros and cons to it but at least to a level it shows that we're having dialogue and talking about the issues and thinking about the issues and so sort of it level of satisfaction. I mean I think that there is a lot of good that comes out of being a counselor and being able to connect with neighbors, but each day is different some days are great some days are not great. But I think for the most part we have a lot of great moments. But I do think that realistically speaking, you know, it is, you know, it is, it is a significant, it is a significant commitment in terms of the hours that I don't know is this entire. It is just sitting and beating listening to bunch of this or beating the people in your, in your area and listening to that, and maybe not being able to help them one on one but letting them have a voice where they tell you about the trash and getting things back where you really want to see you see your constituents I really think, except me some may not agree with your politics but they want to see you succeed because you come back to them, but when you just connected from them, or you have no better than they do. I think that that support you can get from the people will really benefit from your efforts.  You do.  You know, I, you know, I'm now trying to track down one of my constituents asked about how to get a speed bump with it or prospect rate. Yeah, I'm learning about the speed on the process. And then I, although we have frequently gotten distracted from this because of other things have been intermittently working on and on a draft ordinance with bill war that would we think hopefully we're going to get it done. I think it's going to like help the trash issue and words one and words eight like I find that stuff really, really satisfied. I also think it's really, really, I found it really really interesting to learn about the city and how the city works. And, you know, you know, everybody's got an opinion everybody thinks the answers are so simple but if it were easy baby, you know, the easy stuff is getting done. But there's a remarkable talent and political talent in this city. It's incredible. People, people like my smart people, a lot of people care, a lot of people care. Yeah, a lot of people. Yeah, kind of. Yeah. Yeah. So I found it to be a high level of satisfaction and I have to go to two or, you know, and you know, I find the MBA meetings valuable. Like I think this is a valuable process. Yeah, I don't know whether it's the most efficient process, but I don't know that efficiency is always the pain. Well, there's a opportunity now more to find out what a better MBA. Well, your end is a lot different than the word and wanted. Yeah. Every NBA. I find it's a little bit hard to be talked to by the University of Moscow, which isn't going to be. But they redistributed to our men's office. I wanted to quickly.  Yes. Yes. Yeah. You want to run the end game. I'm definitely not qualified to do that. Definitely are. I'm not just happy. I'm a, I'm sorry, I'm a student. This is my second time here. At an MBA general and at this one. And yeah, I'm just happy like listen, I'm just being informed. So for now, I'll definitely ask. Thank you. It's very golden. I need to talk to you. Thank you. So very actually help me and email that. Join us. I think by. He'll be the person next wants to give us as much updates as possible. And why don't that I just want to make a couple of two points based on the participation of council meeting. I think folks without cars. Also have difficulties with that name.  And also looking at the agendas. You know, they get longer when you look at the bucket. I mean, they're. 300 or something page. Because Burlington, yes, it has a city name, but it's more like a town than a city. But at the same time has so much stuff to be done in one city meeting. Three hundred forty pages plus. So that also in your folks to go through. As the reason why those meetings. That was longer as time goes on. Yeah, first time that I attended it was. I think I think maybe in the past, where I haven't attended those, but as I attend the longer I'm realizing. Whether I have a car or not, I have to stay no longer if I want to get a full experience in person. So I just want to have like those two, because Gary's not with us today so we can. German to us. So we have a little bit of time. I have no other comments. So now that you guys are going to have to kind of reassemble them in PA. What are your thoughts on how you're going to do that? I have the signs. We have some very nice signs. Connection. They're yours. I'm happy to take the signs and put them in my garage as well. Got to run around for my as it says tonight. You can't put them up early. Perfect. Okay. On that note, am I going to go ahead and adjourn the meeting? It's an objection to that. Wow. This is another major difference. Thank you. Thank you. You're welcome. I'm so afraid.


The reason why you cannot defeat the narcissist you are not mentally prepared For the situation of defeating the narcissist you can't go toe-to-toe with them You can't directly oppose or compete with them But I'm not saying that you're not strong enough to do that.  Who is willing to do that? You're a warm-hearted person You're kind and sympathetic You want to help other people You don't want to break them down You did everything in your power To make them a better person To give them a better life You did everything you could To show them how much you cared about them But when you do that They just see it as though they have control over you They see it as though you still care And they see your kindness as a weakness They see it as something for them to exploit The kinder you are to a narcissist The more they will take advantage of you And the more they will try to destroy you in the end Once you've realized that you're being taken for granted But the reason why it's so difficult For you to defeat the narcissist Is because you're not a narcissist They can do all of this crazy stuff They can go all out to defeat you Because they are the narcissist And that's just what narcissists do But you just don't have it in you You don't have the interest or desire To criticize, cause harm or harass someone That's what narcissists do They're the ones who hold grudges They're always looking for ways to undermine you Because they see your power They see that you are strong They see that you're able to succeed Without doing anything that they're doing And they hate you for that But they also see it as something That is a benefit to them Because they're willing to fight dirty They're willing to use every possible way to defeat you Especially involving betrayal and deception They don't mind getting in the mud Because they see it as an opportunity to drag you through it The narcissist will sink the ship Just to take you down Even if it means that they're going to drown in the process They will sabotage themselves just to get to you Narcissists are very ruthless and determined They show no pity or compassion for other people They're driven by their own wants, needs and desires While you are driven by your compassion for other people You have strong empathy You don't want to see other people hurt There are two types of people in this world People who build And people who destroy Narcissists destroy While you prefer to build And that is why you cannot defeat the narcissist Thank you for watching I hope this video resonate with you Please like, comment, share and subscribe If you would like to donate My paypal link is in the video description Coach the inquiries You can email me at coach-and-anarch-surviver.uk Thank you for watching and I'll talk to you soon


Welcome back to analyzing software using deep learning. This is the second part of the module on using hierarchical neural networks for analyzing software. And specifically what we'll do in the second part is to look at one application which looks into types and programs and how to predict types in programs using hierarchical neural networks. As most of the applications that we look at in this course, this is based on very recent research and actually this is based on a piece of research I've done myself while being on a sabbatical at Facebook. Together with a couple of other people, if you're interested in more details about this work, you're invited of course to read the paper which you can see here on the slide. Let me start by saying something about the motivation for this work. So as you all have probably seen, dynamically typed languages have become extremely popular. So languages like Python and JavaScript are used all over the place and a lot of software, including a lot of sort of important software is written in these languages. Now these languages are dynamically typed, which means that developers do not have to write on type annotations or types at all for variables and functions and so on. But instead these types are known at runtime only and usually unless someone annotates these types, they are not known statically, so they are not known by just looking at the source code. Now this lack of type annotations has a couple of negative consequences. One of them is obviously that you may have type errors that you may not detect while just looking at the code simply because there's no compiler or type checker that tells you about it, but you may only detect them later while your code is running, possibly while it's running in production. Another problem is that in the absence of type annotations, it may be pretty hard to understand some APIs. So assume you want to use some function that is provided somewhere, but you do not really know what kind of parameters to pass and what kind of return value to expect. Well, this is exactly the problem that you have if your APIs do not have type annotations. Finally, the lack of type annotations also makes it pretty hard for an IDE to provide good IDE support. So for example, if you want to have auto-completion or automatically jump to the definition of a function, all of this is much easier if the IDE knows some of the types in your program, but of course if you do not have any type annotations, it simply doesn't know. To make all of this more concrete, we will look at one specific example throughout this whole part of the lecture and this is the example that you can see here. So it's essentially two Python functions, findMatch and getColors. And as you can see and as it is the case in a lot of Python code, these functions do not have any type annotations. So specifically, there's no annotation, for example, that tells us what type of this parameter is. The same for the return type of this function. So we see that there are some return statements, but we do not really see immediately what the return type of this function is. And the same down here for the getColors function. Again, we do not have any type annotation for the return type and there are no parameters, so there's no need for an annotation here. One way of adding some types to a code base, for example, adding some types to the example that we've just seen, is Gradual typing. So the idea of Gradual typing is that you do not have to add all the types at once, which would be really hard to do if you have a lot of code written in a language like Python, but that you can add some types at some code locations basically as much as you want. For example, you could pick a few functions that you know are very important to your code and that are used widely by other code locations, and then you could annotate the parameter types and return types of these functions. And then what a Gradual type checker is doing is to essentially take all these type annotations that you've added and it will warn you about inconsistencies. Let's say you say the parameter type of one function should be an int, but somewhere you're calling it and you're obviously passing a string into this parameter, then it will tell you, well, this is an inconsistency, this is a type error, and something is wrong and you should fix it. But at the same time, it will not complain about missing information. So if there are some functions that are not annotated, the Gradual type checker will just ignore this, which makes Gradual typing a nice way of adding types to an existing code base. Now, while all of this is nice and good, annotating types is actually still pretty painful because it means you have to go through a lot of already written legacy code and carefully reason about what the types of all the variables and parameters and functions and so on might be, and then add these annotations. So doing this manually for a large code base is a lot of work and it's not really the kind of fun work that developers would like to do. Now, instead of adding type annotations manually, there are a couple of options to do it automatically. Option number one is to use static type inference. So what this basically means is that you have a conservative static analysis that reasons about your code and reasons about values in your code that you created at some point and then propagate this information such that at the end it may be able to tell for some code locations what types, for example, a return type of a function is or a parameter type of a function should be. These static type inference approaches typically guarantee type correctness, so they only tell you that this program element has that type if they know for sure. The downside of this is that it's pretty limited in terms of how many types they can predict. So we have run some experiments and have found that many types can actually not be predicted using static type inference. So it's a good approach that gives you some types, but it also misses a lot of other types. The second option is dynamic type inference. So if you're running your code, you can see the types, right? You see what kinds of arguments are given to a function. You can see what kind of return values a function produces. And because this is at runtime, you know for sure that this is a possible type that can be passed as an argument or can be returned by a function. The downside is that doing so, so actually running the code requires you to have some inputs that run this code. You may have a test suite, you may be able to run your code in production, but you somehow rely on a way to get these inputs. The other problem is that it may miss some types because of course not all code will be covered and even if a piece of code is covered, it will be covered with one kind of type. So for example, you may see a lot of ins given to a function, but maybe this function also is able to receive strings, but if you never see it, you will just miss this type. So it's a good approach, but it has some limitations. So this is also not the full solution to our problem. Option number three, which of course is also not perfect, but that's the focus of what we are talking about here, is to have some kind of probabilistic type prediction, for example based on a neural network that learns from existing type annotations to add the missing type annotations. And this third option is what we look at here and we will look at how to use a hierarchical neural network for this purpose. So the approach that I want to describe here is called typewriter, because it's basically writing the types into your code. And this typewriter approach consists of two big components. So the overall input is a program, and actually for training the model, we need more than one program. We need a lot of programs, but once you've trained it, you just pass one program and then the output is the same program, but now with type annotations added. And this happens in two steps. So the first step is a probabilistic model that predicts the types that are missing, based on a lightweight static analysis that extracts some information from the code. We'll see what kind of information this is, which is then used in a neural network to do the actual type prediction. What this gives at the end is a type vector for every piece of code where a type is missing. So for example, for a return type of a function that is not yet annotated, it will give you a vector that you can interpret as a probability distribution over the possible types that this return type may be. Now, given these type vectors for different parts of the code, the second step is to combine the different predictions for different elements in the code using a search approach that searches for a consistent way of adding types to this program. Consistency essentially means that the program at the end is type correct and type checks according to our Gradle type checker. And how exactly this works will also see in a few minutes. Let's start by having a look at this first part of the approach and specifically let's start by having a look at this lightweight static analysis that is extracting some kinds of information from the code that we will then use for the neural type prediction. So the information that we extract here is of two kinds. On the one hand, we have natural language information and this is something that a lot of program analyses are actually not doing but which turns out to be pretty helpful. So specifically what this analysis extracts are the names of functions and arguments and also comments that are associated with functions. And the intuition is that both these names and also the comments often tell you something about the types that the function parameters or the function return value will have. So getting this natural language information is pretty valuable. On the other hand, we are also extracting programming language information basically what a more traditional program analysis would also do and specifically we extract two kinds of information here. One is occurrences of the code element that we want to type. For example, if you want to type a parameter, we will extract occurrences of this parameter in the function body and we also extract information about which types are imported by the current file because that tells us something about the types that the programmer is actually using in this file in which we may want to use as the types to add as annotations. So let's look back at the running example that we have already introduced earlier. So now we have the same piece of code here. The only thing I've added is this comment which actually does exist in a lot of real functions and which we will here use as part of the input to make a type prediction. So one thing that one kind of information that we're extracting, as I said, is the natural language information. And for this specific example, we will extract a couple of pieces of such information. One is the names of this function here and also the name of this parameter. The same for the other function where we also extract the name and then we also extract this entire comment here. So basically all of these words that are here in this doc string or this Python function level comment will be extracted as natural language information. In addition to the natural language information, we extract information about the program itself. And specifically here we look at the occurrences of the parameters and at the return statements in the code. So the first function has this parameter color. So we look at occurrences of color which occurs here and here. And then instead of just extracting this one token, which wouldn't be very useful, we look at the window of tokens around this occurrence of the parameter. So we go back a couple of tokens, let's say until here, and then also go forward a couple of tokens until here and extract this window of tokens as a short sequence of tokens that tells us something about how this parameter is used. And we do this for every occurrence, so not just for this one, but also for the other one. For return statements what the approach is doing is to extract the entire return statement. So basically we would extract this and also this and then for the other function this whole long statement with this array expression here. And each of these are represented as a sequence of tokens which we will then use in our hierarchical neural network. So after extracting all this information, it is fed into a neural model and this happens to be a hierarchical neural network. And what we'll do with this neural network is to predict types. So this is for type prediction. So as I said we have different parts of the input. We have the code tokens that we extract from all the occurrences of a parameter and also from the return statements. We have the identifiers that are extracted for every function and every parameters. We have the comments that are associated with a function. And then we also have the available types which we know from the imports that exist in the file. Each of these pieces of information are extracted for every code element that we want to predict a type for. So for example for every parameter that does not yet have type we're extracting all these four parts of our input and then try to make the best possible prediction based on all of this information. Now to handle these different kinds of input we are using different sub-models because this is a hierarchical neural network. And specifically what we will have here are recurrent neural networks that reason about the sequence of code tokens. We will also have a recurrent neural network that reasons about the different identifiers that are associated with a specific program element because this is not just one identifier but it may be multiple. For example for a parameter we have the name of the parameter but also the name of the function and also possibly the names of other parameters that also go into the same function. Then for comments we will have yet another recurrent neural network that reasons about the sequence of words in this comment. And for the available types we actually do not really need any specific other model because there are not so many types that may be available and therefore a short vector representation is actually sufficient. Now in order to feed the code tokens and the identifier names and the words in a comment into recurrent neural networks we need to somehow represent them as vectors. And for this we will use embeddings. What exactly these embeddings are goes beyond the scope of this particular module of the course but there will be a different module that just talks about embeddings. For now all you have to know is that we have some function that takes a code token or takes an identifier name or maybe takes a word in a comment and maps this word or token to one short vector that represents this token or word in a way that somehow preserves the semantics of the word or the token. So to do this we have one kind of embedding that works for code and for identifier names and this is what we'll use for the code tokens and also for the identifiers and then we have a different embedding that works well for natural language words and this is what we'll use for the comments. So essentially what happens here is that each of these individual tokens and words is represented as one vector and then the sequence of vectors is given to the corresponding RNNs. So this here is basically a sequence of vectors and the same also here and also here. Now what each of these RNNs does is to summarize this given sequence into a vector. So we will have one summary vector for the code tokens another summary vector for the identifiers and yet another summary vector for the comment and all of these basically compress the information that is given into a relatively short vector. What's important is that each of these RNNs is independent of the others. So they will be jointly trained but apart from that they each can do well their own reasoning about the given sequence of tokens or words so that the way we reason about the words in a comment is of course different from the way we reason about tokens in code. Now given these four vectors they are concatenated into one large vector that essentially represents all the information that we have about the code element that we want to type. And then this vector is given to a feed forward neural network which may have a couple of layers and then in the end predicts basically a vector that we will talk about in more detail in a second which is called the type vector and essentially what this type vector contains is a probability distribution over the possible types. And this here is just a feed forward neural network plus a softmax function in order to turn this type vector in something we can interpret as a probability distribution. Let's now look a little more into this type vector which is the output of our hierarchical neural network. So what we do here is to formulate type prediction as a classification problem where we basically say the possible classes to pick from are the different types that exist in our code base. Specifically we do not look at all possible types because in a large code base there may be many, many such types but we just look at the top 1000 types which turns out to cover more than 90% of all the occurrences of types in the large code bases that we look at here. So this output then is a vector of 1000 numbers and each of these numbers basically tells us how likely it is that the type we want to predict is this specific type. Now what we do during training is to set all of these values to zero except for the one type that we want to have as the type that is going to be predicted. So we are basically saying all of the types have zero probability but there is one type that should have probability one and this is the one you should predict. Now the network is trying to learn this and getting as close as possible to this perfect prediction. Of course it's not perfect so what we will then do during the prediction is to take the type vector that is actually predicted by the network and we interpret it as a probability distribution over the possible types. So there will be some type that maybe has a 60% probability and another one that has a 20% probability and a few other types with even smaller probability so that all of this sums up to one because we have used the softmax function to make sure that this actually looks like a probability distribution and what this essentially gives us is a prediction of how likely the different types are which we can then interpret as a ranked list of possible types with the type that has the highest probability at the top of this ranked list followed by the type of the second highest probability and so on. So now you know how to build a model that can predict types let's now have a look at how to actually train such a model. So as usual for training we need some kind of training data and the training data that we are using here are the existing type annotations that already exist in the code base that we want to analyze. So fortunately some people have already annotated types either manually or using some other existing approach for example the static type inference or the dynamic type inference that I've talked about earlier. In practice this typewriter approach has been applied to a multi-million line code base so really a lot of code in which some types depending on what kind of types we talk about so it's between 20 and 50% are already annotated. So there's already a lot of data to learn from but at the same time like 50 to 70% of the types, 50 to 80% of the types are still missing so there's still a lot to do for the type prediction model. And then what the model basically learns is to predict the missing types from the existing type annotations because we use the existing type annotations as training data and then once the model is trained applied to all the code locations that are still missing types in order to hopefully fill in many of these missing types. So to make this more concrete let's get back to our example. So here is again the code that you've already seen earlier and now once we have trained our neural model in order to make type predictions we will get for every missing type a list of likely types that the model things should be annotated at this code location. So specifically what we'll get here for the parameter called color is a prediction that says well I think this is an int and if it's not an int then maybe it's a string and my third option would be a bool. Similarly for the return type of this findMatch function we get also three actually more likely types the first being a string, the second being an optional of string which basically means it could be a string or none and then the third one is none. And similarly for the return type of getColors we also get some predictions for example the model is guessing or predicting that this could be a list of strings a list of any's where any just means any possible value that you have in Python or just a string. Now if you carefully look at this code you see that some of these predictions are wrong some of them are right and one challenge is how to actually find out which of these predictions to use. So talking about challenges let's think more about what we can now use this neural network for and what kind of challenges we still have to overcome in order to make it really useful. So one challenge is what I've already mentioned it's this imprecision. So some of the predictions that we get are just wrong because any neural network will be imperfect for any kind of interesting prediction task and in practice this here means that someone for example the developer must decide which of the type suggestions that the model makes to follow. Now this could be done but we'll see a better way of handling this problem. Another challenge is the combinatorial explosion that results from this imprecision. So for every missing type we will have one or more suggestions by the model. Now because there's not just one type missing we have different lists of suggestions for every location where a type is missing and in practice you can combine all of these different type predictions so maybe you want to take the first predicted type here and the second predicted type there and maybe the third predicted type here but there are a lot of different combinations and actually there's so many that it's practically impossible to really reason about all of them so we need some more clever way of finding the right predictions out of these ranked list of predictions that we get from the neural model. So these challenges that we just discussed lead me to the second part of typewriter that we look at now. So essentially what we get as the output of the first part as the output of the hierarchical neural network is this type vector that gives us a ranked list of likely types for every code location and now we need to find a set of types that we actually want to use so that at the end we can decide which types we want to add or at least suggest the developer to add to the program and this is done in the form of a search where we're looking for consistent types so basically types that if we add all of them together still lead to a type correct program so the gradual type checker does not report any type errors and in order to do this we combine two things one is we use an existing static type checker which is this gradual type checker that I've already mentioned and the other component is this feedback directed search which is a new technique that we are proposing here in this typewriter work. So let's have a more detailed look at this search for consistent types. So what we get from the neural type predictor is a list of top K predictions for each missing types and what we'll now do is we will use an existing radial type checker essentially as a filter to find the predictions in this top K list that are actually correct in the sense that these predictions if we add them as type annotations will lead to a type correct program. In practice there are many of these gradual type checkers that we could use for Python for example there's Pyre and also Mypy or for JavaScript there's Flow in the concrete implementation that we talk about here we've used Pyre but in principle any of these gradual type checkers could be used. Now if you think about this problem of picking the right type from this ranked list or these ranked lists of different types then this is actually a combinatorial search problem. So we have some number of places where types are missing and we call those places type slots and for each of these type slots we have some number of predictions which is K if you look at the top K list of types. Now if you think about how many possible type assignments you get then this is K plus 1 to the power of size of S the plus 1 is because we can also just not add any type so this is always an option on top of the top K predictions we get from the model and if you think about how many possible combinations these are then it's basically too much to really explore exhaustively so we need some more clever way of searching the space of possible types to add. To navigate the space of possible type annotations typewriter is using a feedback function so there is some function that basically tells us for a potential assignment of types to type slots how good this assignment of types is. The goal of this feedback function is to minimize the number of missing types so we want to add as many types as possible without introducing any type errors because a type error is obviously something that a developer does not want to have. Now in order to put this into a single value that can be returned by our function we return these two numbers here we combine these two numbers namely the number of missing types and the number of type errors into a weighted sum and specifically the weights that we are using here is a weight of 1 for the missing types and the weight of 2 for the type errors so we give a higher weight for type errors which basically means if you add a type and this is wrong and it leads to a type error then this will not be as good as not adding the type. Now given such a feedback function a search can basically go through the different type assignments or the different ways of putting types to the missing type annotations and the different strategies in which the search could work so we have explored a couple of different strategies that are summarized here so one dimension to decide how the strategy should work is whether the strategy should be optimistic or pessimistic. Optimistic essentially means that we are adding the top most predicted types everywhere in the code optimistically assuming that those are correct and then if you see that some type errors have been introduced that way we start removing types again. The pessimistic approach in contrast is assuming that probably most of our types are not that good so we should add one type at a time and be very careful about not introducing type errors so if we add one at a time this is very easy to do because if we add a type and it's wrong we just step back and remove it again. The other dimension for the strategy is whether we want to be greedy or non-greedy while adding these types Greedy and non-greedy here is with respect to the score that we get from this feedback function. So the greedy approach means whenever the score decreases so when things are getting better we keep the type and we just keep adding and modifying other types but this type we know that it's good so we keep it. The non-greedy approach also sometimes backtracks basically it means even if it has added a type which has decreased the score it will sometimes remove this type again and the reason is that otherwise you can get stuck in local minima which basically means you have added a type which looks correct at first but then it prevents you from adding other types which would actually be the correct types and in order to not get stuck in such a local minima the non-greedy search is also backtracking sometimes. So let's get back to our example and illustrate this whole process a little bit. So what you see here is the ranked list of predictions in the neural model and as we've already seen earlier not all of these predictions are correct and it's not always the first prediction that is actually correct. Now let's assume we do have an optimistic strategy that basically adds the first predicted type to each of these type locations. So we will say this color parameter is an int the return type of findMatch is a string and the return type of getColors is a list of strings. Now given this set of types we will give this code to the gradial type checker which we'll then see if there are any inconsistencies and in this case it would basically see that yes there is an inconsistency because if getColors returns a list of strings then this means that this candidate here is a string because this is what we get from getColors and then we're iterating through these candidates but at the same time if we say that color is an int it means that here we're comparing an int and a string and this is something that is not correct so this will give us a type error and the gradial type checker basically tells us that this way of picking the types is just not correct. So now let's assume that instead our search now takes the second option for the parameter called color so it says it's a string but it sticks to the other first option here and what we now see is that this type error goes away because this now is a comparison of two strings which is fine but there's one other warning that we'll get or one other type error that we'll get from the type checker and this is about this return statement because if we annotate findMatch to return a string then we cannot at the same time return none because none is not a string. What we instead actually need to do so instead of the string here we need to annotate the return type as optional of string which means it could be a string or it could be none and once our search has found this assignment of types we will see that we have added all type annotations wonderful so we've minimized the number of missing types to zero while at the same time having not introduced any new type errors which means we are done and our score goes down to zero and we have found a perfect solution. Finally let's have a look at some of the results of this model in practice so if you just look at the neural model alone so just a prediction but not the search then we can measure two things one is precision which basically means whenever the model makes a prediction of a type how likely is it that this type prediction is actually correct if we only look at the top one prediction this precision ranges between 58 and 73 percent if we look at the top five prediction then the precision goes up to 92 percent the other thing you can measure is recall which basically means how many of all types that we would like to add can actually be added by this model and here again we see a difference between top one and top five so if you just look at the top one prediction it's between 50 and 58 percent and if you look at the top five prediction then 69 to 72 percent of all types can be successfully predicted now this is only the neural model on its own if we combine this neural model with a search guided by the by the radial type checker then the results get even better so what we see here is that 72 percent of all types that we would like to add can actually be added in a way that makes the program type correct if we look at the file level then we find that for 44 percent of the files the approach can add all types in this file basically completely annotating the specific file as a result this typewriter tool is now used at Facebook where it's basically deployed as part of the development infrastructure and now makes suggestions to developers saying that hey maybe you want to add this type and by now a couple of thousands of these suggested types have already been accepted by the developers with just minimal changes and have become part of this code base which at the end makes products like Instagram and WhatsApp and also Facebook itself more reliable because it's adding these annotations to the Python code of these projects alright and this is everything I have to say in this second part of this module on hierarchical neural networks so what you've seen now is one application of these hierarchical neural networks for type prediction and what you've also seen is that just using a neural model alone may not be sufficient because you can actually use some other techniques in this case, gradual type checking to get better results that at the end are more useful to developers so sometimes it's worth looking beyond just the neural models but also combining it with more traditional program analysis techniques thank you very much for listening and see you next time