Hello and welcome. This is the Education Committee in the Vermont House of Representatives. We are continuing our conversation about the potential move to adjust the existing weighting factors and with us today is Professor Tammy Colby who wrote a report to the legislature that was introduced to us last year. So welcome Professor Colby and thank you so much for joining us. Thank you for the invitation Representative Webb. Just for the record I'm Tammy Colby. I'm an Associate Professor of Educational Leadership and Policy Studies at the University of Vermont. I was one of the people who prepared what is also known as the weighting study. That study was commissioned by the legislature as part of Act 173. So in speaking today I am representing a group of individuals who worked on this work that include Dr. Bruce Baker at Rutgers University who is a national expert in education finance and Jesse Levin and Drew Atchinson who are also school finance experts at the American Institute of Research. Who is there a question? Yeah no go ahead. Okay all right. So today what I thought so based on my conversations in advance with Representative Webb was to sort of go back to the beginning and talk a little bit about what the weighting what the purpose of the weighting study was what the charge was what our what our methods were what the findings were and what the conclusions are and then that I think they'll take about 15-20 minutes and then to open it up for discussion and questions that you may have for me about the weighting study I am not prepared to provide specific recommendations or commentary on any one bill you know in our role as the contractor and also this sort of independent objective outsiders on the process you know we're we certainly can provide policy analysis on the bills at one at some point if that's if that's requested but at this point I think we are most useful to you at this point to talk more generally about what the studies findings were and help you to put some of the legislation that's crossing your desk in context of that study so that's how I see my purpose here today so I think Jesse did you did you give me authority to share a screen and all of that yes I did okay thanks let me I believe that everyone should have this is a PDF correct we all have we all do have it so okay great let me just play from the start um great so as I said the the weighting study response to a specific legislative request that was actually part of Act 173 which I know you've also taken testimony on today and other days and that's just to be clear that that the legislative request drew a box around what it is we were supposed to do and really three things one we were to undertake a study that examines and evaluates the current weights that are in our equalized pupil calculation for economically disadvantaged students English language learners and secondary level students and to make recommendations regarding whether or not those weights the values of those weights should be modified two to consider whether new cost factors and weights should be incorporated into the equalized pupil calculation and three to consider whether into what extent the special education census block grant that's interesting should be adjusted for differences in the incidence of and costs of students with disabilities across school districts so we had three purposes look at the existing weights consider whether or not there should be new weights and evaluate and consider whether or not there should be adjustments made to the special education block grant for differences in incidence but before I go into sort of what we did I think it's important to step back and and ask and ask the question why would we even ask these questions in the first place and I think talking about that for a minute is particularly relevant as you evaluate these different bills that are crossing your desk so the first big thing to keep in mind is that states are states including vermont are responsible for ensuring equal educational opportunities for all students that's a constitutional responsibility however equal opportunity doesn't necessarily translate to equal educational resources so students become school with different learning needs socioeconomic backgrounds may require different types and levels of educational supports for students to achieve common outcomes and you're gonna hear me talk about common outcomes a number of times today and we also recognize that schools in different contexts may also require different levels of resources or provide equal educational opportunities for example differences in scale or the prices they must pay for key resources so right at the outset we can recognize that schools by virtue of differences in their student population and their contacts may need to spend different amounts of money in order to bring students to a common level of outcome right all state education funding formula included adjustments for these differences in educational costs across districts right and so we call these things cost factors and cost differentials in vermont's school funding policy right the state largely relies on localities to make appropriate adjustments to their annual budgets for cost factors and then it subsequently adjusts for differences in costs in its funding policy through categorical grants and waiting so there are two really important points to take away here first in vermont the way our formula works is that obviously school districts develop pass and and implement their own budgets the assumption in that model is that school districts are spending the appropriate amount of dollars for students to achieve to to achieve at certain levels stipulated by the state right we delegate that responsibility to to to local school districts the second thing is is that we also recognize that in passing those budgets with the assumption that they're spending what they should that certain school districts are going to again have differences in costs and that the for a fair funding formula to operate then it's the state's responsibility to equalize those costs particularly cost to taxpayers there are lots of different ways that states can do that in vermont we use two primary approaches categorical grants that provide supplemental funding for specific programs and services and waiting and in vermont waiting is for a district's average daily membership for certain cost factors and then we use this weighted membership to equalize local per pupil spending for the purposes of calculating local tax rates so that means that our weights operate differently than many other states right our state in vermont the weights implicitly adjusts for spending differences by equalizing per pupil spending across districts according to differences in educational costs and this then impacts local tax burden or the ability to pay for these additional costs of ensuring that all students achieve common educational outcomes in vermont weights do not generate additional state revenue for local school districts rather they're just impacting the price that local school that local taxpayers pay for spending so in a hypothetical example you can think about it this way if a budget was unchanged year to year which that never happens but let's just pretend that happens for a moment right and the number of equalized pupils goes up in that district then the homestead tax rate would come down the price that they would pay for those dollars goes down if you had the same amount of spending right and the equalized pupil calculation goes down and again the equalized pupil calculation is generated with the weights then you would have a higher homestead tax rate notice in all of this all we're doing is we're talking about how the equalized pupil calculation the weights are used to equalize costs or spending across districts for the purposes of calculating tax rates this was the tax equity mechanism in the school funding formula which was a requirement under Brigham so we have four existing weights three of which were the focus of our study economically disadvantaged students which are weighted 1.25 english language learners which are 1.20 secondary students which are defined very broadly as grades 7 through 12 as 1.13 our formula also includes a deflator for pre kindergarten students at 0.46 we were in our in our scope of work we did not consider the pre-k weight it was not it was not it was not something that was asked of us so it's not part of the study so when we set out to answer those big questions that the legislature posed we recognized that we needed to use multiple methods to do this so we did two things one we spent a lot of time talking to people around the state about the school funding formula cost differences the impact that the weights have on spending decisions we talked to stakeholders from organizations we talked to districts we talked to some members of this committee right we talked very broadly and in the report itself there's an extensive chapter that talks about all the things we learned we learned some things that I think are useful to this committee that have nothing to do with the weights so I would also encourage you to read that because there are other things in there that I think are useful to provide context about what's going on the field we also then undertook quantitative analysis where we use statistical modeling which is in our modeling techniques our state of the field in school finance nationally and to identify the factors that account for differences in educational cost across districts so what are these things that we should be accounting for estimate the cost differential for those things and then translate those cost differentials into weights and this is important appropriate to be used in Vermont's existing school funding formula if you open up statute there's a very prescribed there's a prescriptive way that weights are used in the calculation our job was not to rethink how the weights were used in the calculation our job was to estimate cost differentials and translate those into weights that fit into that existing calculation and that's going to come up again in a minute so I want to talk about two these two sets of findings quickly and then we can open it up for discussion I want to talk about the stakeholder findings that's that directly sort of speak to this conversation around weights and the first set of findings is these stakeholder perspectives on cost factors and weights in the formula and what we found was broad agreement in the field I mean remarkably consistent agreement in the field which was the cost factors incorporated in the calculation do not reflect current educational circumstances that we are that the formula is not accounting for all of the things that account for the differences in costs across districts that are outside their control two that the values of the existing weights used in calculating the districts equalized people counts have weak ties with the actual differences in the cost of education students with disparate needs or operating schools in different contexts what does that mean it means that the weights aren't correctly are miscalibrated with respect to adjusting for the actual differences in costs across districts for things again outside their control three the state small school grant programs problematic and its current design and operations and four that there's a need for a specific and targeted grant aid program to support schools struggling to meet different and increased levels of need due to childhood trauma and mental health concerns we also talked to stakeholders about their perspectives on the census block grant calculation remember that was third piece of the work we were doing and frankly stakeholders were pretty mixed in their perspective on the need for potential adjustments to the grant at this time and I'm going to stress that at this time in this words at one end of the continuum and these are direct quotes you know we heard the sky's not going to fall when you when you implement this at the other end of the continuum we heard the correlation between poverty and disability is strong somewhere in the middle and frankly a lot of the conversation was in the middle which was it's just too soon to tell whether the grant's going to be problematic because it hasn't been implemented stakeholders who are concerned about how the census grant will be calculated also however recognized that in part their apprehension was tied to concerns about the challenges with the existing system for pupil weights these two funding formula work together right we can't think of these things as distinct and what one example is is that i'm a stakeholder shared with us is that you know if the weight for poverty was adjusted to reflect what they thought was actually the true differences in costs for adjusting economically educating economically disadvantaged students and students with complex socio-emotional needs then the stakeholders were a lot more comfortable with the census block grant and if you go back to the study that we did on special education that predated this one one of the key findings in that study was school districts may be referring students to special education at higher rates because we have a reimbursement system and that really has been an escape valve for them on the budget side stakeholder perspectives on the small school grant were pretty consistent stakeholders were uniformly opposed to continuing the small school grant program the way it is in the words of one stakeholder everyone is looking for a better way forward nearly all interview participants view the small schools grant program as fundamentally at odds with the policy goals of act 46 and that's that's the school school consolidation legislation and there was general agreement though that the state needs to support geographically necessary small schools so the issue wasn't that the state shouldn't be providing additional dollars to support again geographically necessary small schools it's that the way the small school grant is working is ineffective and problematic so in the words of one stakeholder we don't want to create disincentives with respect act 46 but we want to address factors that stress schools and pass risk to opportunity in general stakeholders felt that incorporating weights for school size and reality which is essentially population density and the equalized pupil calculation would alleviate some of these current concerns because that would be systematic it'd be transparent and it would adjust in a fair way differences that school districts differences in costs and kind of by school districts operating in rural context there were a couple other considerations identified there were concerns about the impact of remodeling college program on districts long-term weighted membership which is part of the equalized pupil calculation there's a general consensus that ECP students students should still be counted somehow in districts weighted long-term membership as a fraction of a full-time student it's not that these students just go away right they're still receiving counseling services participating in athletics but when they go to early college they come out of the count entirely and there were underlying concerns the efforts to update the equalized pupil calculation to better reflect cost differences right which is what the equalized that's all the equalized pupil calculation does cost differences right and introduce more equity in the system may not translate into increased levels of spending in districts with higher need for example in some low spending districts the additional tax capacity generated through a higher equalized pupil count could be seen as an opportunity to reduce taxes rather than increase spending right there is no backstop or sort of floor in the existing legislate in existing statute that would prevent a district from using an increase the tax capacity generated by potentially an increased number of equalized people as a tax cut and still spend at below optimal levels and that goes back to the local decision-making authority around how much is spent on a poor pupil basis in a school district in our quantitative analysis right we used we we we ran our models for the quantitative analysis using three separate data sets in order to triangulate findings across these things and the reason we did that is is we were concerned about you know the Vermont effect right like it would end up that that because we're using Vermont specific data that we might be picking up Vermont specific artifacts of spending patterns and so we ran our models and did our analysis using Vermont data regional data which in course the northeast region with the exception of the Boston metropolitan area because that's not very comparable and and the cost differences are different there and then we re-ran them using a national panel of data so three panels of data in order to check this all out and what we found was is that there were five cost factors that we identified that are related to differences in educational costs across Vermont schools that are outside their control economic percentage of students who are economically disadvantaged percentage of students who are English language learners the percentage of students are enrolled in the middle grades and the secondary grades rather than that seven to twelve splitting that up into six to eight and nine through twelve indicators for geographically necessary small schools that's those are small school those are schools that enroll below a certain threshold that are also located in population sparse areas and independently from that population density of the community in which a district is located based on those five things we then used spending data again from these three sources we modeled out what those cost differentials are like for example how much more does it actually cost for a school to ensure that it's economically disadvantaged student achieves common outcomes with non economically disadvantaged student and in the appendix of the report you can actually see the dollar values for what those are so for example the difference for I believe for economic disadvantages like $3,500 so we calculate those out but remember then we had to take that additional step of then figuring out what those cost differentials translated into with respect to a weight that work that is consistent with the existing approach and statute for applying weights for equalized people to calculate equalized people calculation so what you see here in this slide is the recommended weights for the equalized people calculation these are the adjusted these are the these are the weights that we calculated these weights are based on remember I said we did Vermont regional national data these are actually based on the Vermont data and we're confident of that because when we did the regional models and the national models there was remarkable consistency with Vermont with the exception of English language learners students which I'll talk about for in a moment so what you see here are our weights right in this far on my screen far right hand column right weights recommended by the cost function models would be 2.97 based for applied to the poverty rate which is the indicator in the formula current formula for economically disadvantaged students 1.58 for percentage of ELLs two separate weights for enrollment and why would we have two separate weights we'd have two separate rates because we could model this and see first of all at one point did size no longer matter and what we found then is within that that there's an inflection point where there really is a different first of all after 250 students the size doesn't really matter for cost differences but below 250 from 101 to 250 the cost differential is very different than a micro school less than 100 students which makes sense right like you would expect economies of scale to perhaps be different we found the same thing for population density you'll see that there are these different weights for different densities and again that makes sense too and the weights are point two three point one seven point one one depending on the number of persons per square mile in a district we have a separate weight for middle grades enrollment and secondary grades enrollment so what we found contrary to the existing weight where we say sort of seven to twelve is all one we actually found that there was a slight difference in costs for middle grades versus high school one point two three versus one point two well um so stop you for just just sorry i can get a drink yeah good just notice that you had a question there this is a very basic question but i think this chart what is confusing to me is that when i see zero point two five and zero point two three that says that those students cost less than that they're a fraction of so i don't understand the zero points yeah so another way to think about this is that it would be to make this equivalent to how it operates in the formula which is really one point two five it's a multiplier then then the corresponding weight in the following column be three point nine seven so you need add one the reason the reason it looks this way is because because in the formula itself whether or not the weight is additive weight or a multiplicative weight matters and so whether or not you you multiply by one point two five or two point two five is driven by the formula but it is not a fraction it is not a fraction of a student it's two it's point two five more well we went through that how we get to the weights and i think it must have been on friday that showed how that works excuse me okay representative james does that answer your question though yeah add one yeah add one add one add one um but but the reason but there's a second piece to the add one and i want to make this real clear that's because some of the weights in how the existing statute calculates the equalized people calculation are centered on one and some are not so that's why this gets a little tricky because some of our weights are additive and some of them are multiplicative and so remember how i talked about we calculated the dollars and you got to sort of backtrack into what that how those dollars work in a formula that's that's why that that's why we have to do these kinds of things okay and it wasn't isn't something we came up with it's just this is how the existing statute does the calculation and if you we have to think about if we're taking dollars and making making them into weights that work within the formula we have to take into account the structure of the existing formula okay which can make your head spin it's okay it makes mine spin all the time i'm sorry representative that was when you were up in appropriations that coli went through all this so that's why you missed this i'm sorry okay thanks so it's basically three times in a sense is three times yep yeah representative back hey professor colby good to see you it's good to see you got a good question for me i don't know i have a question now regarding population density yeah i understand your measures and your weights but let me just throw an example at you and you tell me how this would apply in this real world example okay i'll use a district that's next next to me um it includes lindenville and several very small towns that surround lindenville they make up one district now i'm certain that lindenville would not fall within any of those population density measurements they're they're much bigger you know they're st johnsbury like it's an actual town but i'm sure that every one of the towns that surrounds lindenville meets one of those criteria so in that situation let's say that um when you meld that all together that the population density of the whole district was if it didn't it didn't trip any of those different measurement trip wires and then that district would get no help with population density or maybe the other circumstance where it does trip the one of those but lindenville doesn't even come close to tripping it and that's where most the kids are they go to that one school in lindenville how would that scenario work out with your recognition about population yeah yeah so so so there are a couple considerations here right so and i think you've hit on an important nuance which is that the population density weights are at the district level let me explain so why is that um there's an empirical reason and then there's a conceptual reason and this is something we certainly talked through with the agency of education and and um in in the analytic process so the empirical reason is is that trying to calculate population density at the school level in a meaningful way is almost impossible to do in a reliable way and it also is not it's not so that's one two related to that is that the agency of education currently tracks population density at the district level so to create a school level population density metric would both be difficult and something new that the agency would have to do so that's that's a consideration right the second thing the more conceptually though is that with act 46 and decisions about schools and distribution of schools and access to educational opportunities within within within this larger area is actually being consolidated at the district level and there was a conceptual concern that if you started just like with the small schools grant if you started to try to weight on some of these at these more micro level or sort of school based factors that we'd run into the problem where we're which we have in the small school grant right now which is you may be providing in fiscal incentives for geographically unnecessary small schools to remain open so the so the idea of both empirically one we have these empirical challenges and conceptually was to weight at the district level and then districts would have the flexibility to make these decisions is there anything called a perfect weight no there isn't and there are trade-offs with this right and I think you've highlighted right so and and that's not the only district that might have one of those challenges right this is also a challenge I believe for representative Conlon with down in Addison County with Cornwall chime in if I'm wrong on that representative but you know like we have we have areas that are more or less rural and some of these districts and you know that that is a consideration it's a valid policy consideration but I would say that those considerations have to be weighed against the ability to operationalize something in a meaningful way that's objective and systematic that can be that can be maintained right and two that that we need to be careful that we don't inadvertently build into this system other kinds of incentives that are working against policy goals and other statute name namely act 46 so it's it's a valid point and you know I think that's something that US legislature leaders need to discuss but there what I'm just trying to lay out is there there are some there is some rationale around that and you know that's open for discussion is that helpful representative yeah thank you and represent Conlon I I know I I sort of brought you into that conversation I do want to make sure that if you want an opportunity to comment you okay I know I had the exact same question and really appreciate the thoughtful response okay it's not intended to be penal it's right like it's when there's there's science and then there then there are some policy decisions that have to be made around these kinds of things and what we try to do in our our thinking is be really balanced and thoughtful across all of these trade-offs where you know some of these things are pretty empirically clear some things involve some trade-offs and this is certainly one of them and and there could be there's certainly room for policy discussion here the one other thing I do want to point out though is that this these weights because we've used regression modeling they go as a package right so when you pull one out we have to right you have to recalculate everything else and that makes sense right because what we're trying to do is comprehensively adjust for differences in costs across districts in a systematic way and because we identified what the cost factors are if you pull one out for example then there might be spillovers to another cost factor right you understand I'm saying so it's important to keep in mind that we started by identifying cost factors these are the things that that account for spending differences across just across schools and districts outside their control and here are the adjustments for those so I just want to put that point out there representative Austin I think I see your hand up yep just want to yep um I'm just wondering why minority students weren't a category um because when we did the cost fat when we looked for cost factors things that predict differences in cost race ethnicity did not account for differences in costs we didn't find that right but we also have to remember and this is unfortunate but is that there's oftentimes an overlap between economic disadvantage and race ethnicity and it makes it very difficult to parse apart the cost differences that might be due to systematic racism right and the cost differences that are due to economic disadvantage that does not imply that that that that those those factors might be a part of it but what I would say is because those things are oftentimes highly correlated it's likely that that the cost variation is picked up in large part there at least the portion that's due to economic right the portion that's due to economic disadvantage um but you know these questions of how do we account for systematic racism and in um school finance formula is I think is going to be an important question for everyone nationally going forward um and certainly something that the field is thinking about and working on but it's we're not there yet frankly with being able to estimate that in a reliable and valid way thank you professor Colby um I have a question and it relates to uh the the enrollment and population density question um and we also have um a categorical aid right now it's about 20 million dollars for transportation you factored that in when you did this right because in the way this works right when when you when before the weights are when when we're applying weights we've taken off all the categorical right before you start to get into tax capacity so all of that has come out so when you're saying if you pull one you you've just pulled a thread that's going to sort of uh the whole thing so um I understand the population density challenge and what that means but it looks like you sort of double counted it here one is your density and the other is you're a small school thank you these are the weights for those things and so one of the things you could do is and we recommend is that you conditioned the enrollment rates on population density and you are able to do that and that's in one of our recommendations right so there could be so so if you think about these are independent cost factors right so we found um in fact we just published a paper on the top journals in the country is that we found that there's actually a difference in costs that population density is a cost factor independent of school size and oftentimes we conflate those two things right we say that they're small in rural and we we don't it's actually they're actually they drive costs in really different ways so what it means is that a large school district operating in a rural area may also may have higher costs by virtue of being in a population sparse area teacher labor um higher wage higher um higher rates for contracts things along those lines right we also know that we have economies of scale over here so what to be consistent from a policy making perspective was to condition the enrollment weights on population density but you would still apply population density weights independent of that does that make sense I think so so but but to so it's not taking weights out all you're doing is you're saying that that the that four small schools that you would condition turning on those weights weights only in just only in districts that were population sparse great thank you representative arson sorry for the delay I'm following up on uh the chair's comment on enrollment correct me if i'm wrong I mean this isn't per building this is per district and we don't have any districts with less and these are course these are per school so building yeah these were per building and the way this would be in the in the way in the calculation it would be the percentage of students in a district who attend a school for example with less than 100 students and then best weight would be applied to that count of students yeah but isn't the population density per district the way it is the weights are all applied at the district level right well not district they're all applied at the lea level right and so any school level weight has to be calibrated up does that make sense am I in which I explain it a different way so what I mean by that is that so let's take um let's take my school district right so so let's take washington west right we have a number of small schools in our school district and but the weights all are calibrated right the weights we have right now are at the district level if in fact we wanted to include a cost adjustment for small geographically necessary schools in a district what we would do is we would say as a percentage of students attending that districts how many are in how many schools that would give you a number of students right and to that number you would apply this weight but all the weights we have right now are at the district level at the lea we don't wait at the school thank you can keep going I know you have a little bit more okay so just a couple high level conclusions um so and I think this is important is that we have to remember that the weight we have in the current formula haven't been updated in 20 years and so one of the things I've heard over the course of the last year is boy those feel like big differences well there's been a lot of changes right since 1997 in educational costs and so that's one thing the second thing we have to we have to remember though is that the weights when they were originally put in place in the wake of x 60 we're basically carry over holdovers from the foundation formula previously and we could find no evidence that those weights were empirically derived in fact in talking to folks around the state what we heard is you know there's more kind of guesses about like what we thought the cost differentials were um and they weren't estimated using what we would have considered at the time or even now sort of contemporary methods for estimating cost differentials so I think we have to remember that you know we've had sort of this stagnation in the state's funding policy has been a source of concern the weights are widely viewed as out and dated and sort of out of step contemporary practices and part of that reason is that the weights just haven't been recalibrated or reevaluated in a very very long time the second thing is is that the weights and sort of our existing funding don't really recognize some pretty significant shifts in the state's educational policy and practices um you know small school grants for example stakeholders talked about like this is really inconsistent with other things the states are doing and so all together our findings suggest that it's time to incorporate new cost factors and weights into the formula that the existing weights for economically disadvantaged ELL fall short of appropriately adjusting for the costs of educating students to common standards we need two new cost factors school size and population density we need to think about refining the school level I mean the secondary level weight and breaking it into middle level and secondary grades and we also need in doing all that need to recognize that modifying the equalized pupil calculation may not translate into increased levels of spending in the districts with higher need in fact the equalized pupil calculation in the current funding formula isn't tied at all to expectations for what spending should be or quality that's the existing formula right like that's that's not us making that up that's just the way the formula works right and so the formula in the weights the way the role of the weights play right now is in terms of taxpayer equity which was a key thing or a key point of response that the legislature was charged with dealing with in the wake of Brigham right so the weights are miscalibrated this part of the existing formula is not operating the way it should operate because it's not equalizing costs um then the other the other finding just again is there there may be a need for new sources of cat sources of categorical aid for student mental health and trauma based instruction let me just stop there and I'll unshare my screen thank you I think as you know implementing something like this uh next year would be uh quite traumatic um in addition uh and if we implemented it over a period of time we still are dealing with pulling one thread of education funding and not addressing some of the other very complex issues related to education funding so um this is sort of a sliver of the challenges related to education uh finance so I guess what I'm looking at um what questions remain so so it as this is one of the the things that that you presented evidence needs to be addressed what are we missing going forward that we need to need to address that we could set up sure sure um well so our the charge for our study was to do what we did we evaluated the weights we calculated new weights we talked to stakeholders we came up with recommendations the scope of our work did not include a plan to implement anything right and I think you and many others on this committee and people who've asked me questions asked good questions about well how did what do we do next with this piece and I think there's room for good thinking around that um I certainly be welcome the opportunity to think that through with a group or a task force or with anyone but that wasn't in our charge right and so we we were asked to do but I you know I think you're right and I think there's we we do need to be thoughtful about this right like these are big changes and and we didn't be thoughtful about that and I know that you have a number of bills to think about this different ways and and you know I think that's open for discussion and debate um and and and should be right so that's one right but what I would say is that if no other changes are if if the existing formula isn't thrown out essentially right like you know this is still the formula and the formula a key component of this a key I'm sorry a key requirement of the existing formula for it to remain in compliance with Brigham is taxpayer equity we're not we are not adjusting for differences in costs across districts that's all the weights do but that's not happening right so that said like you there are other things that we may be wanting to consider right so what are some of those things we don't have a direct link between sort of state authority and local spending decisions which means that we may end up with districts that are spending at less than optimal levels and that can be under spending and over spending right ideally we would want all districts to be spending at an optimal level for for right there for what their students need but you know that's not connected to for example our EQS mitigation quality standards right we so that's that's one thing we have to think about another thing we need to think about is you know there's operating support and then there are targeted programs and we have to be careful not to conflate how we fund those two things right like general operating supports is sort of a basic school finance principle is like you really have to be careful about thinking in terms of general operating support is categorical grants right because what you do is you create all these silents right and these micro programs and requirements and and things like it's it's problematic it's part of the reason right for the moved act 173 where we're trying to allow good decision makers at the local level to more flexibility and funding categorical grants are the antithesis of flexibility and funding right because categorical grants are always targeted for a specific purpose and if and they come with their own monitoring and accountability standards so when we want to think about using categorical grants we want to think about that when we want to do something really specific and targeted either for all schools or maybe select schools so for example you know one of the findings in our report was around trauma and mental health we know that schools are really struggling with that and different schools are struggling in different ways right but that's not necessarily general operating support and then the third thing is is that I talked to this committee about this a couple weeks ago I'll say it again we need to make sure that we're spending the dollars we have access to already right yes we're turning back title one and cis dollars but I think before we start start sort of beating up on local school districts about that I think we also then need to ask the question why is that happening right and how can we build capacity in local school districts and local school boards to know how to invest dollars wisely I know there's some school board members on on on this committee it's a tough job right when you look at a budget and you're trying to say like where do we invest dollars wisely where is my where am I going to get the biggest bang for my buck how should I prioritize my dollars right local school districts need help with that you know and so that and then the fourth thing I said the other day which is with respect to one time money carers act money as their money we need to be really really careful about making sure that we use those dollars for one time costs right so using those dollars for example for a categorical aid program for students in poverty as sort of a back door to you know let's not do something about the weights let's do this what you've done is you've recreated a recurring cost to the state that you may not build a fund going out and that now comes with additional monitoring requirements much like title one which is federal dollars which we're already having trouble administering because it's a categorical funding program with specific requirements and accountability standards so I don't know I don't okay does that is that it's a guardrails around this too like I think there's a lot of room here for good thinking of policymaking but you know there are some places where I think we could really make things worse let's let's not do that I've got some other questions I have three people lined up here at the moment representative Erison then Brady's and Conlon yeah my question is on the study itself uh earlier in your presentation you alluded that other states use a similar system and I'm curious if your results fell in line with theirs so no other state uses a funding system like for months we're inventing the wheel we have reinvented the wheel most states operate a foundation formula and that foundation formula where there's there's a minimum and then right applying weight the weight what the weights do is they generate additional revenue ours doesn't do that so when I say when I say all that said all states have funding formula that adjusts for differences and costs they do that in different ways we are the only state in the country that has this tax equity adjustment where our weights operate to um calculate equalize pupils for the purposes of adjusting tax tax capacity thank you if I may I have one other thing um we recently just did a study for for New Hampshire who's also if you're following really working hard on this issue too and in that study and I'm happy to for that alone in our report we actually review all 50 states and have like some matrices talk about what other states do and so if you'd like to see that I'm happy to point you to that report thank you that would be great representative Brady that's a good segue to my question actually um and maybe I just need to look more closely at that section but I've heard Paul from the public assets institute um in talking about education finance a lot say obviously there's a lot of work to be done here but on the whole Vermont's education funding system is pretty equitable compared to a lot of states which doesn't mean that there's not a lot to work on here and some some huge challenges but is that do you think is that a fair characterization like that you know even from where we are right now is it work yeah okay there are states that are much worse than us I don't want to I don't want to put us I don't want to rank us in terms of good guys but I can tell you that that there are states that are much more inequitable and inadequate remember that there are two two pieces to this right like we want to make sure we're sufficiently funding and that the funding is fairly distributed right and so there are states that are far that we're spending is very low raising real questions about whether or not sufficiency and in some of those states it's also inequitable there are states we're spending as high where we have inequities right that are usually tied to differences across you know geographic areas right where you live matters so there are states that are much worse than us on both of those metrics represent of common thanks a topic that I brought up previously I always need to be made clear on this so your study talks about students with disabilities we had the committee the secretary in today and he referred to special needs students and so I get a little confused as to where students on IEPs fall into this I think you have said repeatedly your study really looks at students those that students on IEPs are separate and apart they are separate and apart in Vermont right our we run our state of Vermont operates a categorical grant program for that provides supplemental aid to local school districts to offset the cost of providing special education and related services to students with disabilities those are students with IEPs not students on 504 plans there are other students who are struggling students who might be struggling because of economic disadvantage who are non-disabled right so those those students are covered in this analysis our analysis did not reopen rehash the question of special education funding in the state the only thing we did is we went in and we said does it make sense at this time to include a poverty adjustment to the census grant block grant which would allow us to acknowledge that there may be differences in the in sort of the incidence of disability in a community largely due to poverty and the consensus on that from our perspective that it's too soon to tell that you know certainly a weakness of a there are strengths and weaknesses there's no such thing as a perfect funding formula for special ed too right and certainly you know we want to monitor that but part of our comfort level on not making that adjustment yet and waiting to see comes from two places one on average Vermont spends two and a half times the national average per IEP let me say that again this is from the study we did in 2017 on average we spent two and a half times the national average per IEP we have a lot of money in our special education system right now whether that's appropriate or adequate it's hard to know right but there's a lot of money the second thing is is back to that other stakeholder observation is you know these two systems are really closely tied and we tiered systems of support which we really hope that school districts are putting in place you know early intervening services etc you know the question needs to be what happens here if we were to take care of the miscalibration over on the waiting side so that right what would happen then because for some school district what that would do is create additional tax capacity to actually spend more on the general education side without changing tax rates and that might change how those school districts feel about us about whether or not a sensitive block grant is sufficient right right and I think what you said is that if you sort of wait for poverty as record as brought out in this that they would probably feel more comfortable with the census block grant system that was we heard that from many stakeholders in the field right and some of those stakeholders are superintendents some of like broad range range of people said you know and I think that's an important consideration in all of this all of this discussion is that you know sometimes we like to break these things apart because we got a categorical grant we got wait when it comes to being in a school district it that's like money's fungible right money's fungible and so it it's not that we can just sort of peel off one without thinking about the other these two things are unrelated and I think that was part of the reason that it was actually in act 173 that the legislature called for the study of the weights and just one quick question uh how do we how do we determine what is a geographically necessary small school in our study we recommended one approach you might use which is use the cut points so a small school would be one that's below the 250 because we find in our study is that over that number of students that we just don't see a difference in costs so it would under that and then we also find that there's really no difference in the right cost over a certain certain level a certain number of individuals per square mile and so a geographically necessary small school would be one that met the criteria of having higher costs outside their control and that was in a geographic area where we also know that costs are higher thank you I think one of the the questions that also exist here is um we have high poverty districts that are high spenders and low spenders and we have low poverty districts that are high spenders and low spenders and always there's always the question and always the call uh to reduce education spending in the face of declining enrollment um um remembering the implementation of act 60 and seeing how that went for the for the how that that sugared out I think that there is a concern in adjusting these weights that it might certainly just end up increasing education spending I think it's hard to know and I you know this is actually a good question for when mark perot comes back off of leave mark did a nice analysis in the wake of act 60 about what those short term decisions and what that did and I am remembering what mark told me right so correct mark mark will need to correct me but I believe that that there was a period of compression where where where the weights did their job in a way right like they made it too expensive to spend dollars and just for certain districts and so spending came down and they they encouraged spending to come up in districts where you want to spend my suspicion is and this again comes from talking with mark and please talk to him further so he can clarify you know my suspicion is is that that compression went away over time because the weights weren't really working very well they aren't adjusting for costs and so money is really cheap for some districts and it's really expensive for others and so unless those weights are calibrated properly then the tax the tax capacity isn't right like if you think about tax capacity as a potential governor on spending it's not working because money is too cheap because all we're doing is rearranging deck chairs we're not we're not we're not bringing new dollars in and so if the weights are balancing this all the way around in a way that's fair and equitable then you're going to see sort of these anomalies at both ends of the distribution now can can I say with any degree of confidence that that you change the weights and spending goes no I can't do that right like that would be irresponsible and I don't know that anybody can do that the legislature however does have the ability to however put in guidelines around you guys get to make the loss right so like so you can put a floor in it says below you can't spend below this and you can you can talk about what you can talk about that excess spending cap right there's no reason that you can't do some other things here that might help to maintain or to put pressure on compression but those policy actions are separate from calibrating the weights we heard from the secretary today looking at our various proposals and had recommended a phase in plan but also perhaps using some professional guidance on the missing questions the unanswered questions yeah so I'm sure that we would love to check back in with you at some point as we consider what those questions might be well you know here at UVM we're we're a land grant university and we feel deeply committed to supporting the state of this you know we we're not we we have no profit motive here so you know we are happy to support the committee and the general assembly in any way we can in this work so then that can include bringing in outside experts like we did for this study so you do let us know if there's some way we can support representative back I want to just make sure that do you have anything that you wanted to add at this point no I really I really appreciate your comments Tammy especially those that you know towards this is this isn't just a one it's not just waiting it's not it's a lot of different things that are working together to get us a result that's different from what we want thank you um we have um Colin Robinson in at the moment with the teachers association and I think it's going to speak to the concept of waiting Tammy Kobe we'd love to have you stay with us I understand that you actually have to go teach first week back in classes so I'm going to put on my other hat but um if comments or questions come up to which you would like my response feel free to email me or happy to come back but it's thank you again for the opportunity to speak with all of you it's nice to see everyone um and again happy to support in any way we can and do send us the new Hampshire I will have some people in this committee that read everything that's great that's great before you go professor um I do apologize for the day that we were supposed to have a radio interview and there was actually a structure fire I had to be at but I remember that I remember that representative Sebelia is coming on on the 27th so I'll see if I can maybe coordinate with you too that'd be great that'd be great yeah we were he was one of the people I was going to interview and he called me from a fire truck saying I gotta go I'm like I think you do I mean let's be clear school finance is important but burning structures I think takes precedent do we all agree with that so thank you again for your willingness to participate I appreciate that thank you so very much thank you good afternoon everyone thank you committee at three o'clock I've been taught kind to power through with Colin as opposed to coming back if you need to take a little break just go off your screen and um I think it would be helpful if we just continued on through uh Colin Robinson we're really interested in hearing uh what the NEA has to say about the question before us wonderful thank you for having me chair web for the record Colin Robinson um political director Vermont NEA I think that's my first time in the committee this year good to see um returning faces good to see some new faces and thanks so much for having us here I recognize you just got a ton of information from professor Colby and um knowing that I was coming after her I intentionally made my remarks um relatively brief and just wanted to sort of highlight some specific points and just so everybody um remembers Vermont NEA we are the union that represents teachers and school support staff all across the state so to that end specifically I want to touch base on sort of four points so obviously waiting is is important and we believe generally uh making an adjustment now uh is wise whether that happens exactly in this moment or in future years is obviously going to be the conversation that you're embarking on but addressing and adjusting weights is important you know one of the things that we hear a lot from our members and we've been hearing them for some time and those of you who have been on the committee in previous bienniums have heard them speak to this and quite frankly professor Colby just spoke to this as well is that you know students are coming to school with incredibly complex needs trauma and mental health needs and schools are doing what they can to support those learners and those students to be able to access their learning and waiting is a reflection of those increased needs and I think obviously professor Colby's report and that are for other colleagues really dove into that in a comprehensive way you know the specifics of the numbers in her report as you heard from Secretary Francis morning speak for themselves you know we don't feel like we're in a position to speak to whether those numbers are are perfect but she did obviously speak to how they intersect with each other and it's sort of an all or nothing as it relates to how they're laid out so we definitely think this conversation is critical I think the other point that we wanted to lift up and again Secretary French as well as Professor Colby have spoken this today is the intersection with Act 173 obviously this study was enabled by Act 173 of the 2018 session and Act 173 has had up until this point I think two sort of delays in the implementation of the shift in the funding and there's a whole lot of reasons that I'm sure your committee will at other times talk to or and speak to and engage on that issue but one of the critical components of Act 173 in addition to changing educator practice and how students and with disabilities and struggling learners are supported is how the census block grant works and you know remembering back to the conversations of this committee and working on 173 and why as Professor Colby said this waiting study was in there is there is an incredibly strong intersection and between Act 173 and the move to a census block grant structure and the waiting study and I think it's just really critical for you all in this committee and policymakers generally to not lose sight of that because one of our concerns is that as Act 173 moves forward if it doesn't take into account some of these issues raised in the waiting study what could be unintended consequences on students in districts with some of the most significant needs and once again there are a lot of moving parts and the moving pieces and synchronization of 173 with this waiting conversation we feel is really critical so that's sort of point two point three is just wanted to highlight it some specific comments on H54 that I think are as you dig into this issue more broadly are important to recognize and we kind of pulled out as as valuable notes one is the phase in approach that is contemplated in H53 seems appropriate and I think once again the complexity that Chair Webb spoke to about implementation is something that we're very sensitive to and we recognize and whether that is a phase in approach in H54 or some other implementation structure that makes sure it's done without negative adverse impacts on students in districts well at the same time making sure districts and students have the resources they need to support student learning so it's sort of a go slow message and and I would say that that I think is something I'm going to lift up from H54 the other note on H54 is that there is a moratorium on other changes to the education funding structure and well I would say I understand the reason for that I think it's actually makes a lot of sense right because in order to if you're moving too many things at the same time you don't necessarily know what is having what impacts and quite frankly I think the education system over the past 10 years has had a lot of adjustments and changes and it's it's not always clear what has had what impact on on students and how we're supporting their learning and so I understand the intent of the moratorium but recognizing that there might be conversations about broader and as chair Webb spoke to broader education funding adjustments we're concerned that putting a moratorium and any further changes would preempt other conversations that might need to be happening right now about other adjustments in our education funding system the third thing I wanted to highlight in H54 is generally the the construct of studies and assessments of impacts of this change one I think and we believe that's a good thing one thing that often I wouldn't say often happen we've seen happen in the past is sometimes big seismic changes happen and there aren't built-in structures for accountability to make sure that those intended benchmarks are met and that the intentions of the law are actually realized and we appreciate that H54 actually has that baked in because there's a recognition these are big adjustments we want to make sure that they're having the intended impacts and if they're not creating ways for those adjustments to be addressed or need perhaps needed adjustments to be addressed along the way in the out years and so I think that is a core principle as you dig into this at whatever time horizon that looks like is one that we think is important to lift up out of H54 so the final point we want to make on this is just how this intersects with the broader conversation about education funding in Vermont and specifically the recommendations of the tax structure commission that have come out obviously as I'm sure you all are aware they're presenting and recommending a fairly comprehensive restructuring of our education funding system now that we're 25 years out from Act 60 and it specifically is eliminating the residential property tax and making sure that everybody pays based upon their ability to pay not just the two-thirds that are currently receiving income sensitization and that's something we support we've long supported that and we believe that that is a fair equitable way to go for districts for students as well as taxpayers and so just wanted to flag to my earlier point and also chair Webb's point that there's a lot of moving parts on this right now and waiting is a critical component of how we support our students as we move through into the next decade and emerge from the pandemic how we effectively support struggling learners and students with disabilities as we continue to move forward with Act 173's implementation and the professional supports that educators need in addition to the education census block grant funding that goes along with that and finally as we look to tackling the recommendations of the tax structure commission so those are some high-level thoughts on the waiting study generally the waiting conversation that you all are embarking on and some specific things that we want to kind of lift out of H54 that we thought were pertinent to wherever your conversation may go I thank you we did have another bill presented to us today H184 and would love to have you take a look at that as well is that that's representative Becksville I apologize I saw was on the calendar and had not had an opportunity to dig into it just yet okay we will appreciate hearing from you particularly on section 11 okay let it be a bit interesting you um anything any questions then for Colin Robinson okay um this has been quite a day um representative Conlon is going to try to pull some of the things together related to the waiting study that we've heard oh representative Brady please thanks sorry I'm just looking at the testimony you shared Colin and in the section about Act 173 you said um whilst far from being fully implemented the act requires educators to get trained and how to change their education practices to better serve all students and that training simply hasn't happened yet and I just wonder if you can say a little bit more about that about what the roadblocks are to it or what the possible solutions would be to get to that training yeah um so thank you representative Brady for that question I think that and I will admit perhaps I I overstated it a little bit in there there are some districts where quite frankly like the district you where you serve in the school board um in Chittenden South has been working with the DMG group to implement some of this work for a very long time um I think what we have observed and what our members have seen is that it's very very different and it all across the state and we just are concerned that if there is a date and time when the census block grant actually um comes into effect that all schools and all educators are prepared to make that make that shift recognizing that some some are much further along in that path than previously and I think when Act 173 was originally passed there was hope that the agency would be able to provide some high-level professional development support to school districts and that is what has not happened it's been sort of issued district by district have been able to tackle this in in different ways I think the other I think is some of you probably know I don't want to assume all of you uh multi-tiered systems of support MTSS which Act 173 is built upon is also not fully realized in all districts either and in order for 173 to be to realize its potential as envisioned it's built upon MTSS being effectively delivered and integrated into the instruction in all school districts and similarly districts are in different places in that process as well so it it perhaps my statement there was slightly overstated but it's not overstated as it relates to the agency of education providing support there is disparate levels of sort of training capacity at different districts and different districts being in different posture to transition to the 173 structure so Colin the secretary was in the other day said he had done some work on the scheduling challenges that are identified in the DMG report and they have been providing some support there um as you know oh excuse me no absolutely my next my question is MTSS what can we do to help we understand that it's what can we do we're excited tell us what you don't want us to do I I mean you know I think that this is a theme that this can members of the committee who have been here before have heard is that the agency of education quite frankly is is under resourced and I think that there is perhaps an opportunity with some of the federal dollars to create short-term positions to perhaps provide technical assistance and support the districts I think also obviously H101 I think is intended to provide some district level coordination I know it's not necessarily MTSS specific chair well that's that in mind but exactly yeah um so I think those types of things but quite frankly I I do think that having greater capacity in the agency of education to provide guidance and support would be would be useful so one time money to either bring in some expertise through the agency or professional support provided regionally yeah yep I think that could I think that could be it and also it could be would you would you do me a favor would you send us something like that in writing yes it everything gets gets lost in the virtual world so I appreciate that yes happy to do um anything else thank you um representative conlan and representative beck would love it if you would have a conversation to help us figure out how to move forward since you have been following this for a long time we're gonna try to look at trying to find time yeah um this has been a very interesting day that's for sure appreciate the committee's time and attention during this you came to the education committee and it's we've got it's it's complicated and I appreciate everybody's willingness to stick with and show up every day and I think with that if there's any not anything else we'll go offline thank you so thank you so much


Welcome. Thanks for joining this session of Blue Team Village Project Obsidian, Incident Response and UDA loop. My name is Julianne and I will be your guide for this walkthrough. It will focus on the UDA loop and how to use it during security incident response and illustrated by two alert case of the same incident. This is part of multiple walksthrough around incident response but also find sync, threat intel and various other aspects of cyber defense. We encourage you to attend another session between real life or virtually. One of those is around triage and scoping for which we will do a very quick reminder. Our agenda will be a quick introduction, the UDA loop, some key questions for incident response or two alert case, a lesson learned and conclusion. For incident response, we will start with the triage and scoping. This is developed in a different session but as a reminder, the six phases of a security incident are preparation, identification, contentment, eradication, recovery and lesson learned. And as a fundamental part, always keep in mind that you should document, document and document what you are doing. Be it for your colleague, be it for an auditor, be it for your management. This is necessary for any security incident response. Please note also this presentation and the corresponding notebook will be available after death count. Let's move to the UDA loop. The UDA loop has been developed by US colonel John Boyd from the US Air Force in the 60s following his experience during the Korean War. It was initially for dogfight, but the concept is widely applicable, most notably for us to security incident. What does UDA mean? Observe, orient, decide and act. It's a quick sequence of steps that during a dogfight helps you to assess the situation and evaluate where you are, what action you want to take and take the action. And it's usually operated multiple times inside a dogfight to help you win against your attacker, ideally because you are operating this loop faster than the attacker. It's kind of similar to the Deming Wheel in ISO standard, the famous plan to check act, but more at the micro level versus the macro level of ISO standard. Obviously, all the observation, orientation decision will be based on the context of outside information of cultural information and past experience, whatever has been analyzed during this event. I have a quick question. During an incident, what is very important is identified what you know, and what you don't know. To quote Donald Runsells, they are known knowns, they are known unknowns, and they are unknown knowns and unknown unknowns. What is often, is it a false positive, is it a known legitimate activity, known testing, be it a pen testing or something else. Do we have enough visibility to assess the situation and at a sufficient confidence level. It's very essential to say, we are not making confirmation bias bias and mis-evaluating things because we have just, for example, one source of information, which would maybe not be really able. To be sure that evidence can be preserved, to be sure that your colleague can do the same assessment, or if things go to court, that court can evaluate the same situation with the same information. We want to know obviously which user are impacted, which role and permissions they had. Same thing for system. We want to know what stage of the attack it is, and is it live active versus the attacker has left and you are only doing basically the cleaning or preventing the attacker to come back. How did it start? Who or what is the patient zero? Did lateral movement occurred? Do we have just a single system impacted or are there many? Which data attacker has access with some legal impact? And last but not least, what is the business impact? Where is the service unavailability, some information disclosure or data tampering? Let's move to an early alert. As part of project obsidian, team has simulated different scenario and kill chain. Now we will see what happens is during an early level of this attack, we got a notification. In this case, user brand social got an email notification that he logged in the company jump host, but he didn't. In a very good company member in notified security. You are security. What are you doing? So identification phase. I said for the older loop, we will do some question, each question will result in one, or maybe multiple observation. We'll orient investigation and a result in decision and action. One quick reminder as a start for any incident or alert. Have a ticket where you will document things. I can only see alert to the user when it's the manual reports and share the ticket reference. Depending on the context, you might or might not share more information. Document what's known, what's unknown, what's your understanding of the initial reporting of the alert. It's important to know that not all user are reporting things the same way. And one user can say, can share a lot of details when another will give you my computer is crashing and you will have to ask the question to say, yeah, okay, is it a serious incident and why what into you to make a security report. Following this documentation and as part of avoiding any bias, you want to regularly have peer review. First iteration of the other loop. From an observation perspective, we look at the ad authentication log, which show a password spray attack has happened and multiple accounts were compromised. We can orient ourselves by reaching out to it or help us to validate. There is no related activity pen testing testing or similar that could be associated to this behavior. After we want to decide do we want to move to the next phase or to more identification. Let's review how we can do this authentication. As part of our Jupiter notebook, we are using mystic by library to do a splint query. As an example, we collect successful and failed logon pair user account and source IP. If we try to visualize data, we have a nice chart on the ad authentication by user. And we can see this line, which is nearly always associated with the password spray. So normally this graph is interactive inside the presentation mode it does not render as well as we want, but we can change things. Let's see a snapshot of the rendering where we have more user but we can see that all the user of the company basically as an attempt to log in on the chart we don't see if it's failed or successful. But if we review another visualization pair IP user versus IP, we can see that all those login are coming from a single IP address, which for some reason I can't change in the presentation sorry for that. With all the information, it's easier to say from the log, I want to get all successful login from this identified attacker source IP, which from your security documentation and all your IT documentation, you can confirm if the source IP is related to your company or not. In this case, we can find that from a single IP, we have five different users that have login and public IP. So it's likely that the attacker is using this as an entry point. Just with authentication log. We know the attack vector is a password spray. We have the source IP of the attacker. Maybe there is a single one maybe there is more and we have identified that it's not just one account compromise but five. It's important here to have not jumped the gun compared to the user report where if you had actions that directly you would have only one user contained or disabled and which would not have blocked the attacker because there were five access. On a few things we don't know. How does account privilege. Are there any other entry point and what activity happened after that for further iteration of the UDA loop. We can check if there are where password reset if some MFA change happened if some cloud app token change. We may want to collect more information be to a VM snapshot or more granular artifact collection like process activity, network activity, auto runs. Depending on your environment. We may want to add telemetry if it's missing our stage and log collection network collection like ideas IPS. And depending on your assessment of the incident notify management in depending on your companies that may happen earlier that may happen later. Depends on your process. We won't dig further in this early alert case because we will review in the latest case. Now moving to the contentment contentment should be proportional to the type of incident and the business impact, you are unlikely to stop a service that brings million of revenue if you have a small fishing from a Nigerian president. And if you are if the attacker is extricating your customer data or your company intellectual property. It's very likely that action should be taken immediate. Those questions should be asking you to your leadership early on and ideally in preparation phase, not during the incident. First typical action for contentment lock disabled the affected user account. If you evaluate is it enough. If it's not enough, we can decide to content or isolate compromise system or suspicious one. If email propagation is used we may want to freeze the mail server queue or recall or delete some email. That's only a few of the actions that may happen or maybe needed during an incident. So for contentment, we are doing a eradication. Normally we always want to rebuild compromise asset because from a security analyst perspective that's the only way to ensure that an asset is clean. Sometimes business will decide otherwise because the asset is a legacy it can't be revealed or it will take too much time. So that's an executive choice decision to balance between the risk. After any rebuild, you should ensure that the system is functional and protected. Depending on the assessment, we ensure that patch or configuration has been fixed to ensure it can't be attacked. And we obviously want to be sure that the business service is operational for recovery. We declare return to normal service validate again that service are functional, normal monitoring is functional, no abnormal behavior observe for some grace period and ensure attacker is not coming back. Let's move to the last alert. We are more at the end stage of the attack. The previous alert has not happened or maybe was miss or misclassified. This is an automated security alert raised on the company jump post title mimicats execution detection. What are you doing. Let's do a quick reminder of what is mimicats. It's a very popular security tools that allow to extract plain text password hash and value secrets for memory. It can perform various type of attack, including past the hash past the ticket. And it's very often used by attacker as an open source tool. We start again from identification. We talk about it staff or business staff. It's very important that you have a good documentation, the inventory identity matrix network matrix that will help you a lot. But clearly, it never will never replace helpful staff. You still want to have documentation that will most often make your work easier and faster versus trying to wake up the engineering on call. And just wait somebody on a bridge. First iteration of the older loop. Again, we try to validate. Is it a pentest or legitimate activity by it or by some non user, depending on your company. You may have user more normal or more abnormal than others, and doing some activity which may resemble an attacker. So it's part of the knowledge and the baselines that you build over time in preparation of incident. Next, we confirm that the mimicats execution is not legitimate. We want to know which credential was stolen and are they reusable as is typically is it end user accounts service accounts some cloud accounts. Are they all credential versus domain credential. Are they domain admin or are they standard user. Are they some group policy object restriction in place, typically to prevent the use of a domain admin outside of certain system. We're using laps, which is a Microsoft tool to ensure that only each local administrator password will be unique. And this way they can't be reused to attack another system. We want to know what that action occurred and whether data filtration active or not. Back to our notebook and to a splint query. We try to check the process activity. Here to example query with either event ID 4688 net native process command line auditing in Windows or with this month event ID one, which is an extra tool but often recommended by security people. We can see the visualization perspective when we build the process tree of the jumpers. We can see various activity attached to one of the user, which is named Pat resist. And you can see that there is a cascade of activity that's mostly related to or shell but we will see other activity. More on this screenshot where you can see some task list call some run DL 32 with the mini dump argument. Who am I common? What we know from this process tree we can say reconnaissance activity has happened. We have some who am I task list, possibly other common. We have a process memory dump, which is unlikely to be any time legitimate. The one my or task list, maybe your system in can be responsible for it. But doing a process memory dump, very unlikely. And we know that lateral movement occurred. The cascading of poor shell and CSC with a computer list argument is a result of postploid invoke port scan. And CSC is because poor shell is calling a command let, which is using teacher. And as such, each call is resulting in a CSC children process that will do this live compilation of the blog. One thing that we don't know or we don't see in this process tree was mimi cats or some PS exec that you can be in in the financing was true. There can be various reason, but one thing obviously the data can have been tampered by the attacker or inadvertently, it can be wrong filter by the analyst. It can be a software bug or configuration is issue. That's one reason you want to have multiple data source and validate they all goes into the same direction. And if we review another log we have identified that the activity is occurring on the RDP jump post. The was deployed in this environment. We can try to filter some of the activity, either by some top DNS list top 1 million top 10,000. Let's say, what are the domain left from some filtering action, we can identify those domain interact s h dot com file pizza, financial magnum tempus financial dot com web warm all dot IO transfer dot s h. Likely the attacker has done an extra session or maybe he's on doing it currently, depending on on your timeline. The only case, and that's where it's important to know your company context is maybe there is a leg is limited legitimate business use case, but usually it will be for one service thing for different service is less likely to be legitimate. We also have seen some type of request on company domain magnum tempus financial, which suggests that the attacker was doing a manual execution. Sometimes it can be also a way to mislead the analyst by the attacker, but at least as a first appreciation that seems the attacker was doing things manually, not an automated, or at least not a fully automated that you can review the you can review the financing station works for more on containment. Depending on the pace of your investigation, you may stop the attacker before during the tax situation before data destruction and log whiting. Again, we want to lock or disable affected user account. We may also want to restrict network access bit by domain IP address to block common control access or to prevent exfiltration of data that will depend a lot of what are the option available in your company. For example, you have a proxy or only an ideas. If you are arriving very late during the incident and you already observe that we are on a stage where the attacker is doing event log clearing on many server and including domain controller, which means that the basically on most of your network. There is no time if not already to escalate to crisis. This will usually include someone from each executive branch legal communication HR potentially more. If you have not already done so but hopefully it is done in form the management. We may want to unplug from the network or shut down impact system. The network is usually preferred as it will help to preserve evidence, most notably in memory, but it will depend a lot again on your context and what can you do, especially in a really remote setup. In the worst case, more global network shut down, typically at office or data center level, maybe something needed. So how to switch to out of band communication. Usually if an attacker has control on your domain controllers that means they also have control on your exchange or mail system, your chat system, likely. You want to involve external partner, your insurance, your external returner, which maybe required or recommended by your insurance. Information sharing an analyst center, K Isaac, and nationals or sector set. Those last two are very important because they help structure your sharing of information and eventually share things in a way which are not liable for you and share this information in a way which is not necessarily identifiable. It's important that you can help that if other company have made similar threats, you can get more input to react faster. Typically to do the containment or to know that this threat actor is respecting their engagement when you pay them or they are not respecting those. You want to validate your backup. Normally that's an early recovery step. But, especially in run somewhere case in can influence contentment, meaning results should be known only early if you are attacker has destroyed your backup or if your backup are corrupt or I've never been tested. You probably have no backup and as a result, you probably want to make more care on protecting system preserve evidence, keep a copy of encrypted data. If maybe a decryptor is found, maybe in a month, maybe in a year, whatever. Ultimately, a very hard decision in a run somewhere case is, do you want or not pair run somewhere. That's a very trolley problem or Cornelian dilemma that you should think carefully and discuss with relevant parties. There is no right answer from a security or law enforcement perspective the recommendation will be always not pay. Basically, if your company life is at stake. That's how they're called, especially if you are at risk to finance terrorism or cyber criminal. You also want to monitor Internet, the different values media and form to ensure that your incident is not leaked to early and that you have time to prepare communication. We want like previously rebuilt assets, but in a run somewhere case, you have probably very high number of system impacted. And first, you need if you are not already prepared it. You need to establish your priority for service recovery. What should be repaired first. Is it your business system, your payroll system, your active directory, whatever else. Ideally, that's come from your disaster recovery plan or business continuity plan, and it's done before any incident. But if it's not the case, that's your first date after rebuilding assets. And like in the previous case, we want to validate system as functional and protected. We don't want the attacker to come back. Recovery stems and previously returned to normal service, validate. Finally, the lesson learned. You should have a final communication to close an incident. It should either include or be followed by the incident report. You should include executive summary, key findings with recommendation, what went good, what can be improved, a high level timeline. Depending on your organization, alignment with MITRE ATT&CK, MITRE SHIELD, NIST, SP800, 53 or ISO 27, maybe good addition. And part of all recommendations should feed your org risk register and be visible to decision maker and executive. You don't want to let an incident and use each incident is an opportunity for improvement. If you are not using it, you leave the door open to an attacker. It could be the same one, or it could be another one. More will be developed in the final reporting work. As a conclusion, security incident response is very often a marathon. You need to take care of your team and yourself, plan for overload, plan for helper, be it internal or external. Preparation is key.  Re-evaluate situation often during an incident. If you have better situation awareness than the attacker, the better. Know yourself, know your enemy, and you can fight a hundred battles without disaster. Uda loop is one method to structure this. Last, make it easier for you and your team, structure and automate the work so decision can be taken better and faster. Jupyter Notebook is one way to structure, to automate things, and it helps to document and show the reasoning of investigation and avoid bias. This is the end of this walkthrough. Thanks a lot for attending it. I also want to thank all the contributors to Blue Team Village Project Obsidian and to project Jupyter Notebook, Panda, Mystic Pie, OpenStrength Research. Enjoy Blue Team Village and Defcon, be it virtually or in real life. Thanks to you and join the conversation on all this call.