You're now live. We are now live and I'll take down the message. Okay, so we are, this is Senate Finance. This is February 16th. And last week, as we were working our way through the waiting bill, which now has a very long number as a committee bill. We got to the section on the waiting oversight committee, oversight committee, waiting tax rate oversight committee. And it switched several, and we also had some ongoing evaluations in there. And I wanted to give the affected parties a chance to speak up and tell us. So we have the auditor. And I know the tax commissioner will be here with us. I see his name. Just to give us any feedback they have on, we've changed some of this dynamic. We've switched some responsibilities around, at least the proposal does, and I wanted to give them a chance to respond. So auditor Hoffer, welcome. Thank you. Nice building. Good afternoon. You like that? Yeah. Thank you for inviting me. I had not looked at this bill. I wasn't aware of this was coming, but I've had a little bit of time and I can offer a few comments. And as I'm sure Senator Brock will appreciate, a GAGA audit in the first instance requires criteria. If you're gonna do a performance audit, you have to measure performance against something that's fairly concrete. It's not like miles of pavement for a AOT, but it's gotta be something you can get your hands around. There's a great deal of subjectivity and ambiguity in this language, and that's not surprising giving the subject matter. So that presents a bit of a problem for us. And furthermore, looking down the road for when you would like this done, I think, unless I misread it, that the complete transition will not be complete until 28. So if that's true, then you're only gonna get one year in that new regime before you ask for a review by us or someone else. So that's something to keep in mind. Second, it's always helpful to the extent it's possible to establish a baseline. And in this case, if you don't have clear objectives yet and the committee hasn't been created, arguably they could do this at your request. The legislature with aid from all the parties that you know are knowledgeable. I would encourage you to think about establishing those objectives to the extent you can now and measure them as best you can now so that you'll have a baseline so that whomever does this down the road, you can show what the progress was. Otherwise, you miss that opportunity, which I think is really valuable. So those are the main takeaways for me. This would be a tough job for my office without, at least right now, without knowing more about the objectives. And many of them are really quite subjective. You use terms like sufficiency. Okay, well, that's clearly in the eyes of the beholder, equitable resources. I'm not sure how to define those things. We could give it a shot, but it seems like the committee that you're creating, which is probably a very good thing, will be comprised, I believe, hopefully, with people who know a lot more about this stuff than we do. That doesn't mean we can't learn, that's what we do. There's a learning curve for audits, but they might be better able to do this. And if, again, even whether they had objective metrics now to measure against later or not, if they did some work on your behalf, we could do what is arguably a better role for us, which is measure the accuracy and completeness of their work. That's one of the things that we're pretty good at. Having said that, I have no principled objection, except that without better criteria, it would be very tough. Okay. I mean, I'm gonna be gone in 2029 anyway, so it's my successor, so not up to me. Ha-ha. Okay. Committee, Senator Brock. I agree with Auditor Hoffer, and what we did in the task force in adding this requirement is it became very clear that we had a whole bunch of things that this waiting study was supposed to achieve, part of it being based upon the work that UVM did to assess how we measure student performance and student outcomes, and that the items that are contained in the waiting study are designed to improve those things. The task force certainly did not have time though to go into creating the actual criteria and the detail level necessary to do the measurement. The purpose of this provision was to, one, recognize that if we're going to make these kinds of major changes, they ought to actually achieve something that's worthwhile, one, and two, we should have a mechanism to determine whether it did or did not. The third thing was who's gonna make that determination? And frankly, we didn't trust anybody but the auditor's office to be able to do that objectively and independently. So those were the things that we sought to achieve in the overall language from the results of the waiting study. Now comes the work though of defining it better so that as the auditor has described that the measurement can in fact be taking place. I think also that we ought to be thinking about interim measures of one kind or another and this committee that's being formed is probably, yes, the right entity to do it but I don't think we can do this kind of major change and then wait till 2031 or 32 to determine whether or not it made sense. So we need to get a middle ground that provides some objective analysis of whether this is working well beforehand as well as provide a mechanism to do performance auditing to test whether it works and that may have to be done in phases rather than in one huge report at the end. Okay. So we may need to do, and some of these criteria might, they really are more in the realm of the Department of Education or working with the superintendents and principals but they're education based but they do need to be measurable. And we've also talked about standardized tests may not be the best measure either especially for some of the things like the poverty weight. So something to think about. Any other questions for the auditor? So we need to have measurable criteria. All right, Senator Hardy. Thank you. So what you're suggesting is this other entity that is created in the bill that the Education Tax Advisory or Education Fund Advisory Committee, that's really meant to be really financially focused not educational outcomes focused as much. So I do think that the Agency of Education would be more of an appropriate body to come up with some of the measurements rather than this new committee. But I'm wondering if we can just create sort of a mechanism in the bill for more defining that because I'm not sure that we have the time to go through and complete all the mechanisms. And I agree with Senator Brock that we wouldn't want to wait until 2032 to do anything with us. So that's what I would hope that we would do is have something in the bill. So if you have a suggestion, Doug or Mr. Hoffer, that would be helpful. Doug is fine, thank you. No, that's all sensible. It's possible if you just flagged this section and proceed as you will in committee and with the full Senate and allow the thing to go over if that's what happens, you could use some of this time if the agency is willing to work with other experts, UVM folks and others to try to get some of this done before the end of the session. If you can do at least some of that, that would help us get a better sense of whether what they've done will allow us to do what you're asking us to do or whether it would need further tweaking. But I'm open to suggestion. I just wanted to let you guys know that and as Senator Brock knows and agrees at present, it's a little soft, that's all. And by the way, one other thing I've learned from us doing, I think three jobs in the education space during my tenure, anything that is related to the schools directly, not the agency itself is very, very difficult to collect good information in a timely manner. It's a big, unwieldy system and the agency will be the first to tell you when you ask them for information that you think is commonplace has said, that's not our job, you have to go to the schools. Once you do that, this timing becomes an issue, but also which schools?  Or in your case, do you really wanna know statewide in which case it's a very big job? So just to keep that in mind going forward. Yes, I think we found that before in that the schools, do we have a common accounting system? Because we didn't and we couldn't tell what the average cost of anything was when we were trying to evaluate money things. And this is educational progress and that's going to be more difficult, so. To talk a little bit about ancient history, I remember during the time that I served as state auditor, which is what 15 plus years ago, I was happy to learn that there was a common chart of accounts that was used throughout the education system. The only problem was they used it differently and what was in one account was not necessarily the same thing that was in another. I hope it's improved over the years then. I don't think we've gotten a common calendar yet. Senator Harky. They're sort of in the process of implementing a common chart of accounts. It's been rather bumpy because they're the least.  But AOE does collect a lot of data, a lot of data from school districts about student performance and financial stuff. So I do think that there is data available on a school district and that the original people waiting study used data from that the AOE had collected. So I think it exists. It's just a matter of defining it. So we really should be talking to them. Yeah, I already got a note for next week. And if I say it out loud, Faith will make sure we do to get somebody from AOE in to help us at least have something in here. This does have to go through the other body. But if it's gonna go this year, it's gotta come out of here. So if we have a placeholder of some kind in there that will allow them to maybe flesh it out a little more, provided they agree with what we've done on everything else, which is not overly likely, but we need to flag that as something that needs attention even if we do kick it off onto a committee or onto the department of ed to bring us back a proposal, us or the ed, probably the ed committee next year and see where that goes. Maybe they can use what, so didn't this bill, we told them to hire three or four more people? Maybe that's what they can use them for. Okay, any other questions? Thank you, Doug, for taking time to be with us today. You're very welcome. Thank you. Let me know how I can help. Okay, well, we will be back in touch. Okay. All right, Catherine, I see there you are. Okay, so just looking at the Ed Fund Advisory Committee and some of the tasks we've set, I know Joint Fiscal Works to get with the tax department to get that letter out. Do you see red flags, issues? I don't, I think a lot of this is sort of, some of this is basically current law. The collaboration with the Agency of Education is trying to formalize I think what's been informal. I have two, I guess, areas of concern. One is just talking about the letters. I just noticed it, I think it's due January, the recommendation is January 15th, and which is later than December one. So for schools that are putting together their budgets, that might be tight timing. That's just a, that's not a JFO thing. It's more, it's the committee that's setting it, but it's just something for you all to think a little bit about. And I'm not sure why the January 15th date was chosen, but school boards are often closing their budgets around that time. So you might want to- Because there's all kinds of posting. They have to post the budget X number of weeks before an election and it has to be finalized. Okay. Right, so that's just one thing to- I'm just gonna let Catherine finish and then we'll have questions. And then the other piece that I am, pondering, is this collaboration with the agents of education and the recalibrating the weights? It's, I don't know how specific it should be, but it's pretty vague in terms of how often they should be recalibrated. And if there's a need to use certain, to use the model that was used before. So that's just a point of further discussion and consideration as you all work through this. I'm just not, I guess I'm not quite sure exactly what this will be. And maybe we go to the committee and the committee agrees to it and we come up with a recommendation. But I don't think doing the weights, I think doing the weights every year is probably a bigger lift. Recalibrating is not what I think was anticipated. So it's sort of figuring out what that process should be. We don't have a problem working with the agency of education. And we do that now as well. So that part should be fine. Okay.  Can you share what the thought was on January 15th? Well, I think Catherine, the December 1st letter is still December 1st. If you look in section 16, that's the December 1st letter. So that doesn't change. The other stuff is just reporting to the legislature. And I think there's language in here that is, and I have to look, but there's language in here about how the weights would be set a year in advance. So that's helpful. Can I just talk about the January 15th date just for a second? Just so on section 15, just, no, that's not it. Yes. No, is it 15? Where is the, where it talks about the categorical aid in the, oh, it's a, no, it's page, it's section 14 in C. And just, maybe I should go back and read it. I'm just not quite clear because it has the weights and categorical aid as necessary. So I just want to make sure school boards have the information. You're right, the December 1 letter is still there. I don't know if this would change anything for the school board's budgets and I would just want to think about that. I just noticed that when I got on here, so I haven't, I just haven't thought about that. I wanted to- Yeah, and there was the year. Well, there was the intent was, and maybe it didn't make it into this draft because I was trying to remember a lot of details when I worked with Jim on this draft, but the intent was that we would set the weights, the weights would, if the legislature was going to mess with the weights or change the weights, it would be done a year in advance. So it wouldn't be during the session, we would make a change to the weights or the categorical aid amounts and then do it for the next school year. It would be in advance. And so if that needs to be clarified, I'm, Catherine, I think any of this language that needs to be clarified, we can certainly do that with the dates, but it wasn't the intention that we would wait till January 15th to get them all this information. Yeah, I didn't think it was. So you're right. So we, I just, we need to go back and just make sure that what you're talking about the intention, doing it a year ahead of time is a good idea if you're making changes to the weights. And then just on the complete recalibration of the, recalibration of the weights, which I think is not an annual event, this is very broad. So I guess one thing I would like to think about a little bit is how much more direction do you want to provide or not provide on how often that would happen and if there's anything specific that you would use for it. Yeah, Professor Colby suggested that the weights need to be, and she made a distinction between recalculation and recalibration. And it was, it's subtle, but actually a big difference. And that the recalibration I think has to happen at least every five years and recalculation could happen more frequently if something major changes and that the model that you would share with AOE would be able to do the recalculation, the recalibration may take a little bit more effort and we certainly could define that more but my recollection is that she thought at least every five years and one of the, they were just recalculated in a fall. We, the task force asked her to recalculate them but they haven't been recalibrated since the study was done. So, and Senator Brock, I don't know if you remember this whole conversation about recalculation versus recalibration, but it was subtle but important. I'm thinking of something, January 15th. We're giving this committee the power to calibrate or calculate, whichever it is, rates. Any change is going to have to be legislative. And that means if we don't get it, well, it's plus January 15th of the first year we may have time to get a bill drafted provided it's not as controversial as this one. But I think we need to look, we've got school boards and budgets and their timing but if we're talking about something that's going to need legislation, we're going to have to have it lined up with our bill drafting and recommendations would have to come to us. I mean, if it's the second year of a biennium that have to come to us easily in November. So somebody could get a bill drafted for consideration. So that might be something to think through while we're working on this. Senator Hardy. Yeah, I think it would be helpful if Catherine, you could just look through all those dates and sort of map them out with your staff and with the AOE staff and see if it all makes sense in terms of and taking into consideration potential legislation and school district budgeting and all that stuff. The intention was not to create Havoc. It was to try to actually decrease the Havoc. Yes, I can do that. And I will look for that one year. That would be helpful. I will look for the language on the one year ahead of time as well. And I, speaking to the recalculation and recalibration on section 17, I'm just, the recalibration is a bigger deal. It's a more, it's really, I think running through what Professor Colby did on her model. And so that's the part that I would welcome further discussion on that because I think that is a bigger, that's substantially more work, I think than a recalculation. And so some of that we just need to, and it's gonna be a question for you all, how much do you wanna, and it may be that we even come back with recommendations to the committee and they, we set up a time, but it's helpful to know if you're gonna be doing this every four years. And I had heard five years also. So that's why I was throwing that out there. So I'd like to just ponder that a little bit more and think about options for it. And we'd have, if it is going to be the same kind of process we went through now, I'm assuming that we aren't gonna have the expertise. So we will have to have a budget like the year before to hire the expertise we need to do the job correctly. Because there's nothing we're gonna do that's gonna monkey with school funding that's going to be non-controversial. Just doesn't work that way. So, okay, so we've gotta do some thinking on the timeline. I will, we will pull together a timeline on just thinking through all of that as Senator Hardy requested and lay out what, we'll see if what issues arise or where, so you all can also see it more clearly. Because this, having this thought out beforehand will increase the probability that it gets done in something less than 25 or 30 years. So, and once we get a smooth system, hopefully it could just be one of those things that goes, gets adjusted and goes into the miscellaneous tax bill, but it's gonna take us a while to get there. Okay, other questions for Catherine? Catherine, do you have other questions for us? No, thank you for your time. Okay, thank you. All right, Commissioner Bolio is with us. And Commissioner, again, this makes changes, gives the, what are we calling the committee, Ed Fund Advisory Committee. They're gonna be putting out that December 1st letter. Just your thoughts, are you gonna miss doing that letter if we take it away from you? That's a learning question, Madam Chair. Craig Bolio, Tax Commissioner. Thank you for the opportunity to weigh in on this. I had prepared just a few high level remarks as well, but it really, I think the earnest discussion that you folks will have is this afternoon with Secretary French. So if it's all right with you, Madam Chair, I might just offer a couple of, what I'm calling observations at this point. And then also speak specifically about the Education Fund Advisory Committee. So a couple of things that I've observed and forgive me because I suspect many of the committee members have already been thinking about these kind of dynamics as well with this whole discussion. But one of the things that we've been thinking about is that with the reweighting efforts, it's likely to be expected that overall spending in the system will go up right in the tax capacity model because you'll have districts who gain capacity and our hope is that they would use it to support their students, but you'll also have districts that lose capacity that won't be able to, likely won't be able to cut costs, commensurate to even that out. So that's one observation and sort of another one that we haven't quantified yet. But as interesting as that, there's a correlation with the towns that are going to lose capacity. A lot of that because of the difference in poverty weights also happen to be districts with high property values. And so our expectation is also that property, the cost for property tax credits will likely go up. Again, we haven't quantified that yet, but just sort of an interesting observation that my analysts had noted and seems to track. And certainly for those folks who are under the 47,000 of household income where their income rate is statutorily set would definitely see property tax credit costs go up for that. So, because of those two things, one of the things that we've definitely flagged for further consideration is the suspension of the excess spending threshold. I know previously the discussions have been that we want to suspend it to ensure that it's not underweighted towns who are getting hit by it. I'm guessing the current thinking now is that you don't want to have towns that have big swings and equalized peoples due to the reweighting become subject to the cap to the excess spending threshold just because of the reweighting. So I don't have a solution for that problem. It's a difficult problem to solve, but just noting that, especially with the other cost pressures that we might see. Specifically on the Education Fund Advisory Committee, I struggle a little bit to see the value that we'd get from moving the December one letter to it. You know, the work is really already collaborative between JFO, Agency of Education and my department. And the bill doesn't call for a change in how the calculation is made, right? So the statute is very prescriptive. And really from my team's perspective, it's a really fairly technical math exercise. And my read of the bill was not that any of the public members of the committee would be taking on any of the actual work for that calculation. So to me, it feels like the calculation process would be the same, but then would have to go through this committee. So I didn't quite see what the value is for moving the December one letter to that committee. I'd also note, I suspect maybe there's larger discussions again that you might have with Secretary French and just heard with Catherine as well about recommendations for reweighting. And so I'm not an expert to weigh in on that area. And maybe there's some value to be derived there, but I didn't immediately see value for the December one letter. I'm also not dying on that hill at this point, but those are my preliminary thoughts. And education and JFO are doing this letter now. Nothing in this bill changes what goes into that letter or the calculation. So your question is, why are we moving it to a committee which does it say how often they meet Senator Hardy? Or I think I'm charged with, oh, sorry, I didn't mean to. No, that's okay. You're charged with calling it. I don't think it says specifically how often it meets. Okay, okay. But I'm assuming that that would take a budget of some form. So why, yeah, why, I guess it begs the question, why are we calling a committee into session if this is meant to be so cut and dried that it's not partisan, no one can play with it. It just is. So I think that's on our questions from the commissioner. All right, any other questions for commissioner Bolio? I'm not hearing any. Anything else you want us to think about? Not on this topic, no. Oh, okay. Not on this topic, all right. Okay, well, we have got some time here. So I'm seeing Jim Demeray's name is up. I don't know if he is available to pop up, but we might, there he is. We might take a look at these sections of the bill and just kind of walk through what it says. Can you do that for us, put it up? Oh, I can, I was thinking a minute though because I didn't, I wasn't expecting. No, you are not there and this is, this is just off the cuff. It's fine, let me just, give me a minute. This committee has got a long break and I'm not predisposed to give them another long one. We have a lot of work to get to. Okay, give me one second and I can get this up for you. Okay, if I can show the screen, so I can, good. Progress. Okay, we're on the wrong screen. Now we're on the right screen. No, we had a screen and now we've got your five auto save screen. There it goes. Okay, we've got something. All right, good. Yeah, hold on, let me get this centered. Systems are giving me trouble today, there we go. Okay, all right. You got to make it smaller again. This one, we're only getting half a sentence. Yep, how is that? Hasn't changed. Hasn't changed? No, it was a good size and then it got big again. That's too big for you, okay. Yeah, it's too big. How is that? It's the same. We only see as far as not even the forward finance. Okay, so something's happening, I'm not sure what because it's, on my screen, just fine. Let me stuff the sheriff in it and redo it. Just see if that fixes it. Because it got in, it was fine for about a second and then it's flipped back to big. Okay, so education, here it is again. Success. Oh, success, yeah. But it fell down. I thought you were gonna say, okay, hold on, here we go. Show that. Success. Come up yet. There it is. Yes, I can read it. So most of the committee can read it. All right. Okay, so for the record, Jim Damer of this console, we are looking at your draft 2.12 of your committee bill on people waiting. I think you're actually gonna take a look at the sections on the new committee. So let me find that. You just have to roll it down. Okay, here we go. Okay, all right. So this slide was actually came over from, what? Came over from, I think Abby Shepard worked on this maybe with you, I'm not sure. So this slide was just dropped in here. It's a change to the tax title, Title 32. So it's adding a section of law. So this is a section of law, not section of law. So this is meant to be permanent. And so it reads that there's created the Education Fund Advisory Committee to monitor for much education financing system, conduct analysis, recalculate, and recalibrate the people's weights and categorical A amount that's necessary and make annual recommendations reporting its finances or assembly. And then membership are the Commissioner of Taxes, the Secretary of Education, and five members of the public. So two with expertise in financing, appointed by the Speaker, two with the same expertise appointed by the Communities, and one by the Governor. So those five people wouldn't need to be compensated for their time. And there's no set number of meetings here in this section. So unclear what the appropriation would be exactly. Then CS powers and duties. So Emily, before January 15th, the Commission make recommendations to the General Assembly regarding the, again, same words, recalculation and recalibration of the people's weights and categorical aid amounts that's necessary, property dollar equivalent yield, income dollar equivalent yield, the not most of property tax rate and the amount of the stabilization reserve. Assistance is from various folks. So we've got from a Taxes A to Z education, JFO, that's a console in Office of Budget Operations. The Commissioner of Taxes to call on the first meeting and the Committee will select the Chair at that meeting. I'm sure it will be a quorum. Compensation is pursuant to 10, 10 of this tiles, I think this is 50 bucks a day for DM plus reimbursement. And that's it for that section. And then you were talking about this section, section 15, which is that under the previous section, the first report of this committee will be only for January 15th of next year, coming to you among other committees. And in that report, it should recommend the upcoming fiscal year yield, now hosted education property tax rate and the amount of the stabilization reserve as if it was maintained at 5%. And then you were talking about section 16, which is the recommendation sort of changes that I guess the process is different from the December 1st there. So we're not just annually, not later than December 1, the Ed Fund Advisory Committee after consultation with JFO, so Calculate and Recommended Property Dollar Equivalent Yield, et cetera. And I should assume that annually on before December 1, the Ed Fund Advisory, oh, sorry, they assume to relate to the next subsection. So that's going to change to A and then C is changing to say annually on before December 1, the Education Fund Advisory Committee, again, with the assistance of JFO should prepare and publish the Ed Fund Outlook. And the rest of this, which is the same as it was in statute before. And those are the three sections I believe you were talking about. Yeah. And I guess what in this section isn't being done now, what are we adding, if anything? Well, the committee's new, obviously, so that's the new thing. I mean, we're forming a committee, but I'm struggling with, and I guess it's for Senator Hardy or Senator Brock. We're setting up and we're paying a committee, which means this will go automatically to a probes. What value added are we getting in exchange for setting up and paying this committee? What's the problem with the present system that we're trying to address? Are you asking me? Well, either one of you, I mean, this was a, I'm just trying to figure out, I understand we may need a group or somebody to watch the weights and let us know if the weights are getting out of balance. But this is taking a lot of responsibility that's presently being done by the tax, the education and joint fiscal, and we're putting up a committee to meet annually to do what I think is essentially the same thing. And I'm just wondering what was the thought behind that? I think there are a couple of things. One is that a lot of the work of over sort of ongoing oversight of the school finance system, which should be done in collaboration with these three isn't well organized and doesn't happen as frequently as it should. And so this would create a mechanism for the oversight, the recalculation of weights and categorical aid grants. So not just the weights, but also other mechanisms of funding in our system. And that there was a sense that there was, wasn't sort of ongoing expertise that was providing this sort of focused oversight. And that was one of the reasons that the weights went so long without being looked at. And the AOE does not have the capacity to do that. There was a statutory requirement that the agency of education recommend changes in the weights. And you'll see in this bill that that statutory requirement is removed and they were never able to do it. They didn't have the capacity or the data or expertise to do it. And that's why we ended up hiring a consultant. So this was really to have the built-in capacity and expertise that would be ongoing. And part of the reason that the December one letter was moved to this in this draft is to have it all sort of centralized in one place where AOE joint fiscal and the tax department are all working together more formally. Cause right now it's fairly informal except for the December one letter. Okay, but we have what? At least five members of the public. There was a sense that having sort of outside expertise from the education community and potentially from the education finance expertise would be helpful. And this originally was a tax commission proposal. That's where this came from the tax commission and the task force looked at the tax commission's proposals and thought that this made sense given the fact that there was- Okay, so can I assume that was Deb Brighton section of the tax proposal? I believe so, but it was also, I believe part of previous tax commission proposals as well. Okay, we'll take a look at that. I just wanna make sure. So what we're hoping to do is get expertise by paying them $100 a day because we don't have it in-house. Committee, any other questions about this section? Comments? Madam Chair. Yes, Senator McDonald. I apologize, I was listening and I don't know the role that the December 1st letter plays along with the other discussions, but I would caution us to keep the December one letter to say what the tax rate would be under the existing law. And not, and the purpose of that letter is to know what it would be if we don't change the law. The December one letter was put in there to keep administrations from using ed money that might be in excess or whatever to craft the budget before we ever showed up in the legislature. So I wouldn't mix that in with a study committee that operated during the months of the year before the legislature showed up. So thank you. Okay, so what was the problem? Perhaps Senator McDonald, you could refresh our memory. What was the problem that was happening before that caused us to put in that December 1st? 60 established the statewide property tax rate at $1.10, I believe. Yeah. And at $1.10, the school, the administration, the secretary of education and others could, they could, they monitored the school boards and what the raises were being, the increases were being made. And they had a pretty good idea of how much a $1.10 would provide. And for the first two years with, I think it was Governor Dean, Governor Dean's people recognized that that $1.10 was gonna raise much more money than the school districts were gonna spend. And the budget when we showed up cut the appropriation from the, that went to the Ed fund to pay for state aid, it just cut it to the bone and it stayed $1.10. And in the first two years, we were using statewide property tax monies to fund the general fund budget, which was contrary to the promises made by those of us that voted for Act 60 to begin with. And when Governor Douglas was elected, he pursued the same practice. We showed up and the appropriation was, it was supposed to go to the Ed fund was diminished because it was extra. And with under your leadership in finance, we crafted that letter, which said based on what the schools appear to be about to spend, it doesn't take $1.10 to fund the schools and we reduced that $1.10. I don't know what it was the first year or so, maybe to $1.7 and then eventually cut into the 90 cents and 80 cents and the general fund transfer continued to be applied. And the ratio between the amount of money to pay for the schools raised from local taxes, property and income and the amount raised from the general fund remained fairly stable. But in the first several years of Act 60, that was the problem that was being solved that the Ed fund was shorted the general fund transfer and the property tax was picking it up. And after that, the letter really only required a sentence or two. This is what the tax rate would be unless you changed the law and that prevented governors from crafting a budget that I'm assumed that used up all that money. So I'm repeating myself and I will stop, thank you. Okay, so this was keeping everybody honest and only setting the tax rate at the level necessary to raise the anticipated amount of revenue. This is after school, finance officers, superintendents, let the department of Ed know what they were going to be proposing, right, in their budgets. Yep, they scouted the budgets and they had a pretty good estimate and I believe that for most of the years following that change, we were able to lower the tax rate and keep the share of state funding and local taxpayers' funding constant. And the paragraphs began to be two or three or four paragraphs where governors began to say what we ought to do, but we were not preempted when we arrived to be sworn in in January and that was very helpful. The money hadn't already been spent somewhere else at least in proposal, all right. Well said. Okay, committee thoughts on this. Madam Chair. Yes, Senator Bray. Thank you. I just wanted to check in a couple of questions. One was I thought I heard you say that we're hiring an expertise at $100 a day and we only see a little bit of a text here. So I don't know if that is elsewhere or if I misspoke. No, it says they would be given a per diem. And I think our standard per diem is, well for the courts, it's 50 bucks. I think we're somewhere under two, around a hundred. Yeah, I think there is a standard per diem which is very low. Right, I know it's still 50. The courts, well when I was on jury duty two years ago it was $50 a day and if you've got a parking ticket that was your look out. I think we might still be at the same level. I think we are. Right, and so to the extent that we can hire expertise for that, for a per diem, I don't know how that would work out like who will get, but I'll leave that one alone. The other one I just wondered about was when we create another working group or whatever we call it, a committee, I always wonder to what degree are the people who are being signed up for the committee agree that this will be a useful construct for them in which to do work as opposed to us thinking it would be a good way to do something. But the people who are being recruited don't necessarily find it a particularly useful way of getting something done. So I don't know how to assess that other than to hear from the operating areas to say, yeah, this would be helpful to us. Sometimes they'll say, yes, this would be good because you'll organize are coming together and the rules of the road will be straight for all of us, et cetera. Other times it's a kind of a make work situation and it's not all that helpful. So I don't know how to distinguish here but I wanted to raise the question. And I think we would all agree, especially going through this year when we've lost our in-house expert on the property tax that it would be good to have some expertise, not, you know, the question is, but to me, there's learning our taxing system also doesn't happen overnight. And so continuity would also be an issue. And I guess the question is, for me, is the highest and best use of the expertise having them do what is designed to be the very straightforward, get out your calculator and add up the numbers and divide? Or would it be better to use that expertise on a less regular basis but to assess the weighting structure which is where we started out? So thoughts there, no thoughts. Okay, we're taking that down. Well, we can go on and just flag that. We are set to have a break. So why don't we, let's see. We got a half hour break, Faith. I think we do. I lost my agenda again. 15 minutes. 15 minutes. Okay, you've got a 15 minute break and then we're gonna come back. And that one we're going to, we're going back to the retirement income tax exemptions in the feminine hygiene products bill. So I will see.


I2K and other disappointing disasters. Verbal warning, I'm going to talk about disasters and people dying. Not just like little disasters, disasters where people die, so if that's not a thing you can handle right after lunch, no shame in leaving, thought I'd warn you ahead of time. So, how many of you were working in software or technology in 1999? The few, the proud, the brave, the grizzled. It was December 1999 and a young CIS admin of my acquaintance had just bet her boss that their systems wouldn't go down. The stakes of this bet were her job. If anything went down, her boss said, she did not need to bother coming back to work. He would wanted her to stay on the servers all night and watch them to make sure they had stayed up, but she was 20-something and wanted to drive from Des Moines, over here, up to Minneapolis, it's four hours away, and go to a party with all of her friends. And she wasn't worried about I2K. So, a lot of you may get pages. This is how it used to look. This was actually the modern version. The old one had, you just saw things along the top of it. Back then, you wore it on your belt until you dropped it in your toilet and then you got a better belt. And when it went off, you had to go find a pay phone so that you could call the person back, because all you got was a page with a number and maybe a little tiny snippet of data about what went wrong. So, you'd go to a club and there'd always be some harass-looking person hanging on a phone outside the club like this, going, it went down? Yeah? No, come in. Sometimes you'd actually have to hang out of cars to talk into pay phones. They were not actually super convenient and they were always located somewhere super busy. And it didn't really matter what you got paged about because you always had to go and fix it in person with your actual fingers because remote administration was in the future and we weren't in the future yet because it wasn't 2000 yet. So, our sis admin had her pager but she knew it wasn't gonna do any good four hours away. It's as good as the end of the world. It didn't matter. But she left anyway because she knew nothing was going to go wrong. And she knew nothing was going to go wrong because she had spent all of 1999 upgrading and hardening systems. And this didn't mean like a trivial deploy. This means like hand wiring in a floppy drive so that you could put the floppy in the floppy drive and upgrade your router. That was what it meant. You had to write people and ask them to send you physical pieces of software. Magazines were full of patches and fixes and alarmism. Like this was an equal distribution. Maybe a little more alarmism than everything else. We spent a lot of time preparing for Y2K. So, our sis admin had patched all our scripts and installed all the software updates and traced back every piece of hardware, software, firmware, weekly wear, coded cables. If we had had DRM then it would be even more nightmarish than it was everything. Traced it all back, made sure it all worked, tested it. She knew nothing was going to go wrong. As we get further from the reality of what it was like to do Y2K mitigation, we've been forgetting. How many of you were in school during Y2K? Right. If you never experienced this, this was not your problem. We get further and further from the millions of dollars and person hours that went into reducing our risk. Utilities did fail, but they failed in like July of 1999 in isolation from the grid because they were testing. And they figured out how to mitigate. And so we did not have a cascade failure of all of our power systems because they knew that it was going to stay up. And probably about half to a quarter of the people you know who were in the industry did not go out and party like it was 1999. They stayed in their server rooms. They stayed in their war rooms. They stayed on conference bridges to make sure that the world did not end. Back then we didn't really have the idea of push updates and we didn't really have the idea of regular updates. Like the longer your system stayed up, the better it was. If you had to take it down to patch it, something had gone wrong. People would count their server uptime in years with like a point of pride. So it was a really different paradigm to take everything down and patch it. The space program, which we think of as the epic collection of labor and produce and material and we got a man to the moon. We think of that as amazing. It's a tiny fraction of how nightmarish it was to do Y2K. So it's easy now for us to say that it was overblown because nothing happened. Obviously it wasn't that a big a deal. Why are we so head up about this? It was just, it wasn't a real thing. Well, it wasn't a real thing because we worked really hard to make it not a real thing. And it's easy to forget the disasters that don't happen. Nobody remembers all the times that they almost backed into somebody or they almost ran over a kid in the road. It's scary for a while and then you move on. All of this Y2K preparation was risk reduction. We knew that there was something that was going to cause a problem and we did everything that we could to prevent it from happening. We do risk reduction in a lot of ways like we do vaccinations so we don't get diseases and we do anti-lock brakes so we don't hit the kid in the road. And we do train gates so people stop parking on train tracks. We understand how to reduce the risk of things. And how do we do this? Like what are the essential elements of reducing risk? It's impossible to avoid all your risk. You can't do it. We make trade-offs with risk all the time and those trade-offs are complicated and sometimes illogical. Air travel is much safer than driving but we will still prefer to drive a vast majority of the time because we feel like we're in control. There are lots of subtle things that we do to reduce risk but rather than becoming overwhelmed at the idea of trying to avoid all the risk in our lives we can figure out what is in our control and reduce the risk there. And we have to decide how much risk we can tolerate in each zone. For instance, I'm a cyclist and I cycle on the roads in the winter in Minneapolis and that's a risk I'm willing to take and people die doing it every year. Cyclists die because they get hit because people don't see them. Well, I try and mitigate that by wearing bright clothes but I also carry a lot of life insurance. Like I've accepted that for this lifestyle I'm just gonna have to deal with that in other ways and if I die my family needs to be protected. That's how I'm securing my zone. In another example, I work directly adding documentation to our APIs and there's a risk that I could break a customer's integration if I do something wrong but we have a standard way of preventing that which is to say somebody goes over my commits before we actually push them and make sure I haven't done something insanely stupid which only happens once in a while but you really want to avoid it. This is so standard that we don't usually think of it as a risk reduction strategy. There are lots of risk reduction strategies that we've just built into our lives. Like feature flags and development branches and test instances and canary launches and deployments and just all sorts of things that we think are just best practices but we'd never trace that back to best practice to avoid risk. Another way to think about reducing your risk is to predict that the state's something can end up in. I love state machines. I am the English major most likely to draw a little heart emoji around a state machine and the formal way to think of all the ways your stuff could end up in is called a finite state machine. What are the transitions and what are the end states and how do they get there? You can draw this all out for almost every piece of software and the more horrible and complicated it is to draw out the more you should be thinking about microservices because once you get a finite state machine that's like this big it's very hard to secure properly. So to use this to assess risk you enumerate all the possible states your system could be in and what would make that happen and then you can address the transitions that would cause that. So here's an example. I want somebody to put a zip code in. I know you can't read this, it's okay. I only want them to put in a five digit zip code. I don't want extras, I don't want unders, I don't want anything invalid. So my state is an analysis of what they entered. If it is alphabet, if it is more than five characters, if it is less than five characters, I reject it. Try again. Enter a real zip code this time, you know what it is. Only five numbers will get through my filter. Those are the only two possible states. So having figured out what my two possible states are I can derive what the risk of somebody doing it wrong is and fix that. And if I forget one, I can add it to the list, be like, oh yeah, you also can't use characters. You've now reduced your risk of invalid shipping and or weird injection attacks. Harm mitigation is the flip side of risk reduction. Harm mitigation doesn't matter until something goes wrong. No matter how much we try to avoid risk, something is going to go wrong. The truth is that bad things do happen all the time. This is why we wear seat belts and life jackets and install fire sprinklers and smoke detectors and doors that open outward. Usually we don't have house fires in the first place but if we do, we'd like to survive them. So the most concrete example of harm mitigation that I'm going to talk about is building codes. You can also think of seat belts as harm mitigation. We hope you don't get in a car accident but if you do, we hope you don't go through the windshield. Like, stuff has already gone wrong. You're already in an accident. How can we make that less bad? Rate arrays are the assumption that some of your inexpensive disks are going to fail. You just have to have the data distributed across enough disks that you can accept that risk. But building codes. Building codes save lives. In 1985, an 8.0 earthquake destroyed Mexico City. At least 100,000 people died. Mexico City is built on a lake bed that was sort of kind of filled in.  It's not an ideal substrate for any kind of seismically active zone. And this was the hospital. Like, that was one of the better built buildings. The whole city of Mexico pretty much looked like this. This is the earthquake that happened this year. That's pretty much the worst picture I could find of what happened. In the intervening 30 years, Mexico City insisted on rebuilding with stronger, better, more robust building codes. Nobody put up seismically unsound buildings because they knew what was going to happen. And they had this really fresh vivid memory of thousands and thousands of people dying in building collapses. So when Mexico got hit by a 7.1 earthquake, the modern building survived well and it looks like the Mexico City death toll will be 228. And this is not because Mexico City is smaller. Its population is 25 million. But they've even installed an early warning earthquake detector that detects the super low frequency vibrations that get there earlier. People in Mexico City have a 10 second warning before they get hit by actual tremors that they can feel. Mexico City knows that they are in a bad place and they are just doing as much harm mitigation as they can. They're going to get hit by earthquakes, but they can mitigate how bad it is. The last example of building codes is even a harder for me. On your right is Grenfell Towers. It's a 24 story residential building. We still don't know how many people died in this fire. We haven't been able to get in and find them all. A lot of things went wrong, but one of the primary things that went wrong was building codes.  The cladding outgassed cyanide when it burned and there was one evacuation stairway for 24 floors. And it was blocked. There were refrigerators, there were mop buckets, there was all sorts of crap in the stairway in a way that would make an American flinch because you don't think about it, but every one of the doors that you hit from here to outside will open outward because we have a trauma about pictures of people piled up in front of doors that open inward. You can maybe think about triangle shirt waste factory pictures and how people were blocked by locked doors. You can think about the ghost ship fire. 50 people were in that building and 36 of them didn't make it out because there were no fire exits. But if you thought about it and yesterday the fire alarm went off, you knew where to go. Even if you weren't sure that there was a fire, the first thing that happened was you hit the doors and they swung open outward. And you hit the next set of doors and they swung open outward. And in an emergency, that little bit of mitigation saves so many lives. This other picture, this nighttime picture is the sadly named a torch tower in Dubai. It had the same kind of cladding, the same aluminum flame cladding. But it had four exit stairways and it's 78 stories tall and everyone got out okay. No deaths, no injuries because of building codes. It's not that these didn't both catch fire in a really dramatic way. It's that building codes mandate you be able to exit. So think about that when you're thinking about what does harm mitigation do? We accept failure and we make it as not bad as possible. So the first step to mitigating harm is acknowledging that we are never going to be able to fully eliminate risk, which is kind of horrible for us. Like as a parent, I would love to be able to wrap my children up and make it so that nothing bad ever happened to them. But it's not really a workable technique. So all I can do is sort of life proof them. Like give them bumpers and let them learn what hurts and what doesn't. Once we accept that there's risk in everything we do, we can identify it and figure it out instead of just hiding from it. So assume that everything is going to fail. If you don't know how your product is going to fail, you are missing something. If you don't have an idea of what your failure modes are, it's gonna be really difficult for you to make that better. For example, failure modes can have a lot of different meanings for different people. If I leave my phone unlocked and my partner sees my chat history, I will get teased because it's silly. If somebody who is an abusive or controlling partner leaves their phone open and their partner sees their chat history, they could end up dead. Like why are you talking to a woman shelter? Are you trying to do something? Are you trying to get away from me? Your threat model needs to include a variety of people for every failure circumstance. In a software as a service setting, what may seem like a harmless bug around graphics could break an important custom analytics module or it could be something that nobody uses at all. Only by using analytics are you going to be able to tell what people are actually doing with your stuff. Testers are the amazing people who never believe in the happy path and we need more of them in our lives. Like it's important that we do our own testing but it is also important that we hire the professionally paranoid because they have seen all the ways that things fail and they are excited about finding more of them. And if that's not what drives your life's purpose, hire a tester to do it for you because those people really exist and they're amazing. So think about what it is that failure means in your product. When we try to mitigate harm, we need to think about what it is we're trying to protect. For example, nuclear power plants are designed to fail safe. In the absence of power or positive control, the control rods drop and the nuclear reactions stop. That is the ideal. Usually it works. Like we hear about the times it doesn't but for the most part, if your control rods can drop at all, they will drop as soon as a power cut happens. On the other hand, time lock safes aren't designed to protect people. They're designed to protect money. You're not supposed to be inside a time lock safe pretty much at any point. And so you can't open it without a combination and a set time. And that protects the money and the person because you can't coerce them into giving a code and have it do any good. So what we're trying to protect there is what's inside the safe and also what's outside the safe. In most computer cases, the thing that we're trying to protect is data or state. When a laptop runs out of power, it shuts down pretty gracefully. It's like, oh, I'm almost out of power. Nning. That's actually pretty graceful. Like it could be so much worse if you've ever had somebody trip over your desktop cable. You know what I'm talking about. That never happens. But laptops also have this cool thing now where if you knock them off a table, they will attempt to save the state of the data that they know about in the moment that they are falling. Like how cool is that? Somebody has figured out that we care so much about our data that even as the computer is falling to its probable screen death, you're still saving whatever you need to to the hard drive or the SSD. So you have to figure out what you care about. I worked at Microsoft BitLocker for a while and it was really interesting because we realized we didn't care about the laptops. Like hardware is easy to replace. Access. Access was our real risk point. The number of executives who leave laptops in taxi cabs is really large. As a person who has not lost their laptop but has lost other things, I sympathize. But we say like that's disposable. It's a $2,000 laptop, we don't care. It's so much less than the data breach of somebody being able to hack into our network with physical access to your computer. So we secured it so that it was easier to accidentally wipe the data than it was to recover it. We're like, we're gonna assume your data's in the cloud. Nothing is worth somebody else getting into our system. That is our risk model. Again, predicting the possible states allows you to understand where you can make a difference by focusing your attention on transitions and states that could be altered to be better. You really need to think about what it is you can do to do continuous improvement on your product to make it more risk tolerant. The easiest path in this for software is the kill switch. We used to do this process by redeploying a known build but that's painfully slow when you're in the middle of an outgoing crisis. Instead, you can always have a transition state of hit the kill switch.  I meant to do that. That was never deployed, nobody saw me here. You want to be able to say, turn something off in a hurry without having to redeploy everything. Failure is inevitable. We are all going to fail. We fail every day. We forget to remember to bring home the milk. We typo something, something happens. We fail all the time. Disasters are a concatenation of failures that add up to a catastrophe. We can avoid disasters. Why is predicting states so complicated in heart? My simplistic examples don't make it seem like it's super difficult to do this but I think even though we've designed for redundancy and failure, our systems are sufficiently complex that we can't understand all of the interactions. We saw this on LinkedIn. Who here remembers when LinkedIn managed to turn on all of their features at once? No? I was the only one who saw that? It was terrible. It wasn't at a time that they'd stopped nagging you for a little while. Well, they got their stuff sorted out. Turning on all of your stuff at once is not a thing anybody could have anticipated because there's no reasonable reason to do that. You have to have that mindset of what could possibly go wrong no weirder than that. In modern software, it's gotten really difficult to say I can follow a message from one location to another. There's a real like internet magic happens here in the middle, especially in software as a service. Like you can't be sure which node it's going through. You can't be sure which channel it's going through. You can only predict that probably it's gonna come out the other end. This makes testing kind of difficult because you can't build with just the nodes, just the round parts of the Tinker Toys and you can't build with just the sticks of the Tinker Toys. You have to have both the software and the channels, microservices and the connections. Multiplying combinations grow super complex super rapidly. Like if you tried to test every possible permutation of a Tinker Toys object, you'd quickly run out of computational power. This is just two objects in space rotating around a central point. Imagine what it's like when you have 60 microservices and they have more than one channel each. You can't test all of that. So you're going to have to test each of them, test your microservice, test your channel and then when something goes wrong, trace it back and find out where it actually fell over. But you have to give up on the idea of 100% test coverage to do that. It's not a thing. I don't think 100% test coverage is really possible for software as a service. Certainly not for any kind of complex microservices ecosystem. So if a disaster is a combination of failures that causes a system to stop working, usually in a way that cause lives or money, every disaster that you can think of from natural disasters to engineering disasters has several contributory factors, right? So in a hurricane, high winds and storm surge debilitate buildings and take off roofs. Okay, that's bad. But it's really bad when the power is also out and people can't get water and they can't dispose of waste. And it's not just one thing that fails. It's not just that your roof is gone. It's also that you don't have clean water or a place to store your medication or a place to sleep that's safe. Disasters are about a multiplicity of failures. 1,200 miles from here, at the end of the navigable Mississippi, there's an interstate bridge. It is not this interstate bridge. This interstate bridge fell into the river. It was pretty unpleasant to be in Minneapolis right then. It was a beautiful spring day and it fell very suddenly. There was almost no warning. And 13 people died, which is a miraculously low number because it's the fourth busiest bridge in Minneapolis, St. Paul. The engineering report says the primary cause of the collapse was the undersized gusset plates at .5 inches thick. Contributing to that design or construction error was the fact that two inches of concrete had been added to the road surface over the years, increasing the static load by 20%. Another factor was the extraordinary weight of construction equipment and material resting on the bridge just above its weakest point at the time of the collapse. That was another 262 tons consisting of water, sand and vehicles. It wasn't just that the gusset plates were too thin, although that was the main cause. They'd stayed up for 50 years by that point. It wasn't just that there was a lot of concrete on it. There'd been a lot of concrete on it for 10 years. It wasn't just that there were construction vehicles on it. It had had three major resurfacing. It's that all of those things came together at one time to contribute to a disaster. And it was by the grace of over-engineering that it stayed up for 50 years. Because if it had been, like, spec'd properly, it would never have fallen down. And if it had been built shoddily, it would have fallen down much sooner. It was only when you got this combination of factors that we exceeded engineering parameters to keep it up. When we fish our own disasters out of our metaphorical river and piece them back together, we find out that it's the same. This cluster had a bad shard, but that didn't matter until the other cluster blipped and threw a bunch of traffic over to the first cluster and then the shard failed to replicate. It's always going to be something like that. So the only thing you can do is build in engineering tolerances that allow you to recover from these problems. We have to give Netflix credit for popularizing this idea, toolifying chaos to head off disaster, causing minor failures, and seeing if anybody notices. And if anybody notices, your engineering tolerance is not big enough. You need a bigger tolerance. You need more servers. Because you set your parameters for Chaos Monkey and it takes servers out. And if it takes out enough servers that somebody notices, then either your parameters are not an accurate portrayal or you need to add more servers. Here are some general concepts that I'm going to go over really quickly, but if you want them all on one page, this is the time to take the picture. Here's things that cause disasters to fizzle into failure. Use microservices and very loose coupling. Chad mentioned this in his talk. Not pets, not even cattle. Think of them as bacteria. The system has to survive the death of any bacterial thing. Like your server doesn't matter. It needs to not matter. It needs to be a replaceable element. Use very loose coupling. Use internal APIs so that if something goes down, it doesn't chain react and cause other things to come down with it. If your APIs continue to ping something that's down, you will in fact have a problem because now you have DDOS yourself. Which is unpleasant and unfortunately common because you're like, why aren't you up? Why aren't you up? Why aren't you up? I'm down. One of the common mistakes that we make in mitigation is that systems can't always talk to each other smoothly. If there's a problem, it sometimes takes out the messaging too. How many of you remember the status page on S3 was inaccurate? Yeah, right? Because one of the elements on the status page for S3 was hosted on S3. Your messaging about failure needs to tolerate being part of the failure. So if you have a status page, make sure you're not hosting it yourselves. Like status page IO will host it for you and say, our site is down and everything's on fire, yo. And then people will stop calling your tech support to find out if your page is down and everything is on fire. Storage is cheap. Make sure you have your data in more than one place because nothing and nowhere is safe. Delta, I am looking at you. Delta had a perfectly nice data center in Atlanta. And they had, when they acquired Northwest Airlines, a perfectly nice data center in Minneapolis. And then the Atlanta one caught on fire, which would have been fine if Minneapolis was up, but they decommissioned it. They had decided that it was not worth the cost to keep a redundant data center up. And what are the odds of our data center catching on fire? Right? Get in more than one place. Test your feature in production. You cannot test any modern software fully in test. It is impossible. We are talking billions of transactions and messages and you just can't do that. Absolutely. Do your unit test and your integration test and your big test and your, you know, have a test server. I'm not saying that. I'm saying for the love of God, you're gonna have to test an actual production where there's actual traffic. No script on earth is going to write you enough messages to make it look like Black Friday. That's just not a thing. So roll out your stuff to a few people at a time, make sure it goes okay, increase your rollout as you go, because that's gonna make it a lot safer to test in production rather than just flipping the switch on. Humans are the slow fallible meat part of almost any system. It's a great design when you're trying to prevent cascade failures of overreaction, but when we automate our world to react, we also need to automate our ability to shut things down.   You don't need to listen to it. And automating our circuit breakers and our kill switches is going to give us a lot more safety because it's better to shut something down than to have that one failure spin up into a giant disaster. If you have a runbook for how to get a system back up, why haven't you automated it? Like you know the system went down, okay, that's good. You don't necessarily know why. And because of the mystical power of how computers work, sometimes turning it off and turning it on again makes it feel better. I would love to tell you that we have solved this problem where we're actually good at diagnosing why things go down, but a large percentage of the time turning it off and turning it on makes it feel better. So if it's already fallen over and turned itself off, let's automate turning it back on by itself because again, we are the slow meat part of the system. There's no reason somebody has to wake up at three in the morning and hit the switch and run the runbook to turn something on. If it falls over again, sure, wake them up, but like try the first time on automated. Test for load and stress and outage. Like I said, you can't fully test modern software in a test environment. You have to test it under load and you have to figure out what your peak load is and then calculate beyond that. Like in the old days, and again, I'm going to date myself here, we had this phenomena called getting slash dotted.  And then people would be angry because they wanted to see this cool thing that slash dotted mentioned and the site was down. And we don't have as much trouble with that in like an elastic capacity world, but we still need to be testing for slash dot level events happening to us because it's easy to mitigate, it's easy to prepare for. Like maybe you need to throttle inbound traffic. I'm not saying you necessarily need to be like staffed up like Amazon at all times, but you need to think about what would happen if you got Amazon level traffic. So if this talk was too long and you read Twitter instead of listening to me, expect failure because it's going to happen. Make your systems less rigid and more nimble. Plan for disasters and expect that they will happen too and degrade your service gracefully so that in an emergency situation you can still offer something instead of a whole lot of 404. I work for a cool company called Launch Darkly. We do feature flags as a service and if you would like a free t-shirt, I am too lazy to haul them around the country, but you can take a picture of this and go to the site and we will send you one. Thank you.


The stages of behavior change are commonly represented by the trans-theoretical model, TTM, which was developed by James Prochaska and Carlo Dicklement. This model describes how individuals move through different stages when making intentional changes in their behavior. The stages are as follows. Pre-contemplation. In this stage, individuals are not considering change and may be unaware or under-aware of the need for change. They may be resistant to the idea of changing their behavior and may not recognize the negative consequences associated with their current behavior. Contemplation. Individuals in this stage are aware of the need for change but may still be ambivalent. They are considering the pros and cons of changing their behavior and weighing the benefits against the costs. Preparation. In the preparation stage, individuals have made a commitment to change and are preparing to take action. They may begin taking small steps toward behavior change, such as gathering information or setting specific goals. Action. The action stage involves implementing specific strategies or plans to bring about the desired behavior change. Individuals are actively modifying their behavior and this stage requires the most time and energy. Maintenance. After successfully making a behavior change, individuals enter the maintenance stage. The focus is on preventing relapse and consolidating the gains made during the action stage. Maintenance involves continuing the new behavior and avoiding a return to the old behavior. Termination. In some versions of the model, termination is considered the final stage where the individual has zero temptation and 100% self-efficacy. It's important to note that individuals may not progress through these stages in a linear fashion and relapse is a common part of the behavior change process. The model recognizes that change is a dynamic process and people may move back and forth between stages. Understanding the stages of behavior change can be valuable for health care professionals, counselors and individuals seeking to make positive changes in their lives. Tailoring interventions and support to the individual stage of change can enhance the effectiveness of efforts to promote behavior change.


So, I'm here to talk about what we're calling the QuantStamp Assurance Protocol. I guess I should introduce myself too. Yes, I'm Steven Stewart. I'm the CTO co-founder of QuantStamp. Prior to that, I'm a computer scientist, and I guess I'm a PhD dropout. I don't like it being it, but it seems to be the cool thing to say. I noticed that Remco from 0x, one profile for him, is his PhD dropout.  But QuantStamp got too busy for me. So last April, I became a PhD dropout. Without studying at the University of Waterloo, working with Professor Derek Rayside, using graphics processors for accelerating the operations of something called a SAT solver. It's a tool used in formally verifying, well, as an example, I mean, there are many applications, but one application in software engineering is for formal verification of software. Of course, the word formal, implying mathematics, mathematical rigor, verification, meaning checking at certain properties holds and being able to prove that they do. And so Richard Ma, myself, we created QuantStamp because I can't keep track of the time. How long was it? Back in 2017. And we had in common an interest in the idea of securing code. Richard's background was in high frequency trading, and he was quite familiar with all the testing and rigor that goes into that. Because if your strategy has a bug, it can have a catastrophic effect. And so the two of us both had this interest, and he also contributed to the Dow. So we have these slides that we always show, but I feel like it's getting dated now. Second here. Everyone remembers the Dow. By the way, these slides are Jan Gorsnes. I've crossed his name out. So I will probably stumble a little bit through these because I don't normally present them. But Jan is a law chain researcher, and he would normally be doing this. So there's lots of motivation for caring about security. I think most people already know about that. And given the situation where once you deploy your code, it's immutable, then as noted here, when you're thinking about contract security, you usually think about pre-deployment options. So you're conventional kind of testing, auditing. So QuantStamp, we actually do code audits of the service. Formal verification, bug bounties. I always have this sort of conflict with the word security. You know, sometimes we're talking about bugs. There can be a bug in your code, but it's not necessarily a security issue. We use these words kind of very loosely at times, even though I think terms like security do require precise definitions. But I've kind of taken a look at some of the literature, and I see a lot of different definitions for computer security. But anyhow, my background is in more informal verification and parallel computing, and it took me some time to get used to the branding of QuantStamp as a security company. Are we doing security or are we doing software engineering? Turns out we're doing security. So my co-founder was right about that. And then there's this notion of continuous security. So begging the question, you deploy your code, how can it be protected against new attacks? Maybe you've already done everything you could possibly do. Your code is just absolutely perfect at that moment in time. Something changes in the future. I don't know, the EVM changes. And someone discovers that you can now do a new re-enter to the exploit. Things like that can happen. So QuantStamp is interested in this idea of continuous security, which is basically internally we call it monitoring, basically observing a smart contract while it's live and alerting, creating alerts if there's unusual behavior. So there's some examples of what could be done there. I'll skip through that because that's not really what I'm looking to talk about. Okay, so security assurance. Assurance, it starts with an A, like quality assurance. This notion of somehow being able to assure, let's say the owner or the stakeholder of a smart contract, if you will, that their code is secure and won't get hacked. And so we have these two actors in the assurance protocol. One is the stakeholder and the other is the security expert. So the stakeholder wants their code to be correct. If something bad happens, they might use funds, digital assets, and the security expert who wants to get paid for her skills, for her expertise, and also has some funds available. So Yon does a nice job of creating these visual slides. So as you can see here, the stakeholder, just as it says, has some amount of money that he or she is willing to compensate for exploited contract. And then we have the security expert who wants to audit the code and get paid for it. So this is the scenario we begin with. The security expert is so confident in his auditing skills that he says, you know what, I'm willing to stake some of my own funds, put my reputation and money on the line, and I'll stake a certain amount to claim that the contract is correct. Why would I do that? And why would somebody, so this is the slide showing, and somebody else went to, so not just a security expert, but anybody in the world, they could just say, yeah, I guess 10 security experts have gone in and said, this contract looks good, maybe I'll go in too. Well, what is going on? We'll see momentarily, but I'm talking about the staking of funds and what would incentivize you to do that while the stakeholder pays for this. So it's kind of like, what would you call it, like a premium kind of payout. So stakeholder pays for the stake periodically, provided that the contract is not exploited, because if it's exploited then the security expert actually didn't, you know, didn't, there's something that he didn't find. And so maybe I lose funds. So that's exactly what it says here. So I'm a stakeholder and I say, okay, my funds or somehow my contract is exploited. Actually more generically, whether it's funds or something else that you might imagine, namely there is some policy or some conditions that are, that do not hold. And as a result, I want to claim these funds. So as it says, from the security expert perspective, these funds are lost if the contract is exploited. Okay, let me clarify. So the security expert and someone else, they're staking funds and they want to do that because they're getting paid by the stakeholder. But as soon as something bad happens to the stakeholder's contract, then the stakeholder claims what the security expert and others have staked. I hope that was clear. And so this timeline simply, it shows a sequence of events that would happen here, contract is deployed, potentially, you'll see it says down here, the QSP network report is published. I actually just realized, I didn't really tell you much about Quantstance. I don't know how much you know, but we're building out the QSP network, which is a way of sourcing compute power to do formal verification or to do code audits, automated code audits. So one potential step here is that an automated audit is done. So it sort of ticks off that checkbox and then you see the top first stage of interviews funds and there's a bunch of details there. Actually it gets quite complicated. We have extremely detailed diagrams on our internal Wiki and too much to go into for this. And I don't know what that slide is for. And so obvious question is, well, what is an exploit? So this question is being posed because we want to know when can the stakeholder claim to have suffered damages and can claim what was staked. So as it says here, it's up to the stakeholder to define the bad behavior. It's up to the stakeholder to wash her back door. So basically the stakeholder at the very beginning, let me see here if we get into that, defines the conditions. I believe we call this policy. And basically this says, so here's for example this function, this is obviously a terminal example, but in the body of this, let's see here, basically it says, oh, if the balance goes to zero, that indicates this has been hacked. I think you're going to probably imagine scenarios where a balance line to zero doesn't mean it was hacked. Maybe there was a deliberate reason line went to zero. This is just a toy example. And actually our team is working on creating a set of predefined policies that people can use. You can probably imagine it can get fairly complicated. I actually was pushing for this idea. Alternatively, you could consult an oracle to verify that some event occurred so that that's been identified as another possibility. I like the idea of being able to actually define the policy as a contract, as is where a contract, and that's defined upfront. And then the security experts who are supposed to be experts reviewing code, they can read the policy. If they're comfortable with it, then they're willing to stake their own funds. You can also imagine, maybe I'm really clever and I somehow write this obfuscated policy, make it really complicated. Actually, I have my own back door and I'm actually trying to get the funds that are staked. There's lots of scenarios. We've talked through dozens of different scenarios that could happen. But I've always said, if you agree to the policy, you agree to the policy. Actually, when I say, in the company of Fire Beware, I kind of push for this. We'll see how this works out. I guess this slide is just emphasizing stakeholder with a smart contract, and the policy, and contract addresses and terms. I apologize. I don't quite know what this slide is, contract addresses and terms. Sorry about that. I don't recall what Yon had in mind there. This is just another high-level representation of the same thing. So it's just a system perspective. We have a lot of these diagrams. So arguably, you'll see that the slide is titled Staking. And arguably, and really what this probably should be just titled Assurance. So this idea of security assurance arguably enables and helps to scale the verification. This could be, we could discuss that, but by providing a marketplace for auditors, so code experts and others to stake collateral as a claim on the contract security, and avoid fraudulent reports. So since checks are automatic and rational actives, won't stake on insecure contracts or policies which can be gained by stakeholders. I think what Yon was getting at here, since checks are automatic and rational actives, won't stake on insecure contracts or policies. So I actually don't remember what he was referring to here for reports, but I think this is about where I'm saying, like I said earlier, you can kind of gain this. You can imagine a stakeholder even gaming it. But a rational actor who studies the code and understands that they're not going to stake collateral. If they know, there's some kind of backdoor or vulnerability. So summary here, basically looping back to the beginning, the idea that post-deployment, we can do some kind of continuous monitoring of a smart contract. I kind of compared this, so a few years ago I worked at a startup company called MacD, and we were using GPUs, in-memory GPU databases to do this really fast millisecond querying of gigabytes of data. Actually, we were talking to big FinTech companies, I'm not sure if I can actually name them. It's been a few years, I probably can, who are interested in fraud detection. So it's kind of similar, basically looking at certain transactions that trigger alarms, certain patterns of behavior that would trigger a security incident. And then the staking, which again is referring to the security assurance idea, just as another post-deployment approach to, not quite securing, but hopefully offering assurance, increasing confidence in the security of that contract. And that's the presentation. Yes? I was wondering, if I was, if this existed, and I saw a very smart expert, if I was watching and I saw a very smart security expert put down, say, one ether on a contract, then could I, a non-security expert, then put down ten ether, and then get paid ten times more than the actual expert? That's a good question. I don't think so, I think the way we've designed this. So there are, I don't have all the details of the protocol. But I'm pretty sure the way there is, it's designed to offer greater benefit to actual security experts and to incentivize them to participate. There's what's called a TCR, a token curated registry, that will maintain the list of, I guess, accredited security experts. But I don't have the exact detailed answer to your question. But I can find it. Is there a minimum staking period, or can the security researcher destroy the stake at every time? Sorry, what was that again? The security expert withdraws his stake every time, or is there a minimum staking period? So it's kind of the withdrawal. There are, there are minimum periods. So actually, we have been developing user interface, I forgot. I still have these to show you. And you'll see things in here. I realize the text is small for you, but so this is defining what we call a pool. So in the stakeholder, so the pool, of course this refers to the pool of state funds for assuring the contract. And then there are all these different details that define how the payment works, the payment periods, all these different things expressed in terms of number of blocks. So there's minimum staking time. Yeah, so there's quite a bit there. So this is supposed to be the stakeholders view. And then there's the staker control as well. And this is a prototype at the moment.  And you see actually in this, I think the withdrawal state, but it looks like to me it's like faded because at that moment they're not eligible to withdraw. So there are a lot of details that have gone into this. I'm afraid I don't have them all committed to memory, but we do handle that idea. And one more question if you allow, what do you expect the ratio between stake and reward basically to be? So when... Stake and what? Stake and reward. So you get reward when you stake, right? Oh, reward. So what do you think what's the ratio? Like when do you have your money back basically? That's a great question too. And so I don't know the answer. What we're doing is we're running simulations. So Sebastian, one of our senior researchers, he's actually in Munich. I was hoping he could come because he knows this inside out. He's currently building this agent reinforcement learning simulation with an objective function to maximize the end balance for the stakers. And we're developing that right now. And then we're also doing a community, a quant-stamp community-involved test trial of our current prototype. And this is going to help us to get more precise answers to your question. I hope that makes sense. So we're still very much, we're still very much a work in progress. Do we have some more questions? The way you define whether there was an exploit or not, if you try to encode this in smart contracts, it seems like it will be very, very complex smart contract. And I'm not even sure that it's possible to check whether, like, check every situation. But even for every situation that you can think of, you will create a smart contract that will verify. And it seems like this new smart contract which will verify whether there was an exploit or not, it also needs an audit. So this smart contract can be even more complex. Who will audit it? If I'm willing to save my own funds, I'm a security expert, for example. It's not on me. I would say fire viewers, tell our engineers. Yeah, as a security expert, you're right. But as the owner of the product, or user of the product, if you actually do the audit for the users of the product. And these users cannot be, like, sure that this product is safe just because these security experts, they know that they are doing what they want. So security experts are safe because you cannot, like, you cannot find exploit that will allow security experts to lose money. Incentivizing experts to put time, put their own funds and hopefully get paid is a way to create greater confidence. That's why we use the word assurance. Greater confidence in the security of the smart contract. I'm not saying this is perfect. And I obviously, like, the policy could be very complicated. It could be as easy. It could be as simple as if the balance goes below whatever, then, you know, condition that could be that simple or could be very complicated. That's possible as well. And I agree. You would probably want to have professionals are the policy as well. Because maybe you have a nefarious malicious security expert who sees a flaw in the policy and they don't tell you because they actually want to exploit it. Yeah, exactly. Yeah. Yeah. Good question. So do you believe that the system is better than paying security experts directly? Oh, I don't have the data to indicate that. This needs to be, you know, tested and actually used in practice to find it, to get an answer to that question. And also who's supposed to, like, write this smart contract that verifies whether there was an exploit? Is it, who's supposed to use it? Is that supposed to write it or use it? So this complicated contract and who's supposed to write it? Someone who owns the product? Somebody who knows how to write smart contracts. Whoever wrote the smart contract being secured, potentially. I'm not sure, actually, I'm not sure where you're going with the question. Yeah, I mean, whether you provide the service of writing this kind of contract. Who provides the service? We don't, no, we don't provide, we don't offer smart contracts as a service. We don't do that. Okay, thanks. We do, we do everything, but we don't write smart. Perhaps a silly question. One that follows from that is like, could there then not be a flaw in the agreement itself? Yeah, there could be. Yeah. By everywhere. We spend a lot of time thinking about ways we could gain this. There are definitely tricky ways. The policy is a very important part. You obviously have to get it right. But hey, you could use the security assurance protocol to stake on the policy. So this is going to be our product. It is, yeah, it will be, it will be in a test state for the foreseeable future. And we hope to see it become a product that we share with the community. So it won't be, it won't be like that, something that you sell because it's just going to be something dependent? No, yeah, no, it's not a product we'll be selling. It's a, we'll be designed to use our token interacting with our automated audits. So although it doesn't have to be, someone could certainly fork the code and use it in a different way. But it's something that, you know, one of our aims is to build utility for our token for our token holders. It's a very important responsibility that we have. So there are various R&D projects that we work on to that end. This is one of the more mature projects, but still, still early. Anybody else? Yeah, I had one more small question. So imagine a one security expert already denoted and then comes a second and finds a flow there. Do you, do you plan to penalize the first security expert already at that time because they didn't find the flow or you will have to wait until the contract is exploited? Punishing, so punishing, somehow. I'll have to think about that soon. Anybody else? Well, I really appreciate your questions and feel free to reach out if you have any more questions or ideas to help us out. KoanStamp.com. Thanks a lot. Thank you.