I thought I called me into order. We're going to do smoke off for us, please. We have Tom DeVee. Yep. Are the exhortment. Yes. Tommy Trancey. Black pepper. Yes. Kerry Snow. Yes. And staff, we have Maladon. Lisa Gowder. Here. Thank you for that. Here. Here. We want the number two approval of minutes from November 14, 2023. Do I have a motion? Motion to approve the minutes. Any discussion? All right. Let's vote. All those in favor say aye. Aye. Any opposed? Any objections? None. Okay. Motion passes. Number three. We don't have anything listed on the agenda, but we do have a couple things that we wanted to mention. For instance, this is the last one. Okay. We have a motion. And a second to approve the minutes. Any discussion? All right. Let's vote. All those in favor say aye. Aye. Any other comments? Any other comments? Thank you. And the second question. For instance, this is the last meeting of 2023. You do have on the calendar so far, the same schedule for 2024, the second Tuesday of the month at nine. It just want to confirm everyone was good with that. And then ask the question, the reason that we move these to rotating at the properties is so that the public could come. That is not necessarily then the actuality. up. So we wanted to ask the group we could hold these at the Civic Center in a conference room and have a hybrid option all the time or we could keep it at the properties and still do hybrid at Hawk when we need it. It's just a coordination challenge. We don't have the set up every year. I mean they would have to go somewhere anyway so yeah. Yeah, I'm all for Civic Center as well to make it easier for anybody to apply virtually and also for the public. There's an option for it. I just want it available to them and maybe be here for them to also find instructions. Well the other thing is it's always at the same place to keep your people to remember that too. Rather than keeping track of them. And you know there's somebody from another property comes here and they have to figure out how to get in. And we're just going to bring back the group and let's show them the site. We're just guessing about the next meeting, what it is, 1.4 having a Civic Center. Rather than putting it on properties if you have a student. I think it'd be different too if we weren't doing the copy of the conversations. So that's all for people who you know make it and share it and put it on one. You can go online or buy one of our meetings too. I didn't get that in place. So she'll send out updated meeting invites with the team's name for anybody that wants to join her. Number five development and product updates. So the big news is, it's going to be a little domain on Thursday. So I've been sleeping quite hard since. Congratulations. All in all, closing, they're always crazy, but this one was definitely our students yet Katie had a set up so much was done in advance. And it was really like working through it all rather than cramming it all at the end. So it was really thanks to everybody and work with and everyone was it just not very smooth. So we're kind of in a downtime, we're letting our contractor order stuff. And then we will start right on January 16 start work January 16, our first move out. The relocation team has been working with all of the residents are doing their one on ones with them right now. And we're working out ways to give everybody as much flexibility as possible like moving into a unit instead of a hotel if we have it and it works out. Meaning if it's, you don't want them to move into one that's not that renovated. So if they have one that's open that we can do that. We went and did the project based voucher lottery there last week. So there's 18 project based vouchers going on to those units, which we did an open lottery just for residents. And so that is going to be pretty, pretty great for some of those residents that is just long term security. And it helps the property operationally. And so they have not yet been notified right of the selections. So we'll be finding that out this week. So did you have a good turnout from the people applying enough to get the 18? Oh, yes, we had more than that. So there's probably about 15 getting a denial that they were not accepted. It was totally random. So everything on main as the official name is excited to get started. Excited for this little lull over the holidays and then get started. For other items for ascent at hover crossing right there. We were awarded. So I think that we've mentioned that the city from its archa allocation has set aside so hard 500 from $5,000 for the early childhood education center. We need 3 million. So there's 525 in the city. We applied for 750,000 from where they cause. We were awarded, but 150,000, but it's still something. So we're up to 700. And then we still have dola strong communities for an outstanding and they're supposed to be notifying. Any day. So we're hoping to, to score on that. And we have working with the long community foundation still to see if they can help they want to help. But it's does their financial mechanisms that they have access to work with a life tech project. That's what we're trying to sort out. So the project is a go. We were awarded tax credits for that, which is first go around. Quite awesome. Believe it. So it is very clear that from the list of awardees, just remember that chapter was oversubscribed by about four times. My tech, which a shows that there's just really not enough resources out there for what people are even ready to do. And so we need prop one, two, three to really kick in, which it seems to be kind of trickling and slow and not opening up new programs right now. But I'm hoping for January. And then we also, the Boulder County voters approved the affordable housing tax too. So when those funds are available in 2025, hoping that that stuff can all backfill this lifetime. But still doing these projects without tax credits is all still a huge self letting it basically with those other resources is still a big deal. That it is clear from the awardees that if you don't have some sort of unique service model or like there were several with early childhood education centers attached. Or the only other ones that didn't have one of those two things were like the phase 505. They're just closing out, you know, really wrapping it up with about other than that, if you didn't have something unique, you were not selected really was clear. So going forward, thinking about our projected projects down the road, we're going to have to come up with some really good stuff. Or I expect just wait and keep trying and trying until it works. So that it's great news, but also gives me a big strategy, thoughts about our strategy going forward, just trying to get really, really creative. So on a second, the idea is to now design as a full force now, it is moving forward. We will fund the CCE one way or the other. We're just piecing it together. It will happen. And then the idea is to close in about July of next summer and start construction, which will take about two years. So what do you think of DOH? Right now we need, we need, what did I say, we're up to 700. We need $2.3 million. Uh huh, yeah. Right, we applied for, I thought to be applied for for the song communities. I'd have to look it up. I can report to it at the end of the meeting. And then we know that the Loan With Community Foundation wants to give a million dollars. But they, they are brand new and in terms of set up kind of a program like this and they are really needing it to be a revolving loan fund to keep it going. But that just doesn't work for WhyTech. It needs to be basically a forgivable loan or a grant of some sort. Having to actually pay it back is just not realistic. So we're seeing what we can work with them. We're going to sort of back the chain and see what we can figure out. So do you actually have a Vindra on hand for early childhood? Yes, it's a lot of Plum Center. We have an MOU with them establishing what we want. I know nothing is fully formalized, but it's at least an MOU so that everyone knows what we're working towards. But yeah, they serve, they're, they've got multiple locations in Longmont. They pretty much only serve C-CAP or Head Start Families. They're very established and they've got a good relationship with the city and it's been going really great so far. And it's, we only have room for three classrooms and so it's most likely going to be focused at the youngest children because, you know, four and five year olds can, there's more options out there, but infants to three year olds or maybe infants ones and twos are harder to find. And so we're thinking about, we're seeing if we can focus it there. It's interesting, it's a two-story building which is a challenge with infants because you have to make sure they're on the ground floor or else they can't have an elevator because they have to crib them out in a fire and any sort of emergency. So it's all of those things. So our, our like tech developers are getting a good taste of childcare licensing right now. So, but it's still in the works and as other funding sources come out we'll go over them too. Alright, I think so too. With what's out there, piecing it together and having the like tech awarded and having $700 already in the bucket helps those other sources want to just help you finish. So that's the scoop of those. What's it like when you have that funding kind of back closing? Back closing. Do what they should be doing. We have not, and that's really just from the housing team out there looking. We haven't even looped in human services yet. Yes, all of these, there's other groups to talk to that we just kind of went through the ones that we first go, what else do we have? So our other properties, we've got the Royal mobile home park that's just land banked at this point for a future project. We have, we have our first land donation from the city's inclusionary housing program proposed. It's going to go to council in January to see if council accepts it. And so the city will just like Royal, the city will hang on to that for a period of time, but then the idea will be to eventually transfer that to LH and do something there too. So we're just going to see if that comes to fruition in January. It is about a half an acre on right behind the Bingo alley. So it is an area that already has there's thistle and in between are operating on that street. That is a challenging section of the city with folks. But I think what we would look to do is something more like it might be a town home product or something for families not necessarily single occupancy just to not have too much concentration right there. So we'll see but it's a very developable site. It's an easy site to build on, which is important. So that is proposed and we'll see what happens. So we're land banking, which is really great. If council approves this as satisfying their inclusionary housing requirements, then they will donate it to the city. And then with the idea of transferring it over for development. I feel like I'm missing. Oh yeah, we sold the CPWD building on November 30th right on the day that we planned on it. And like for business days before village on being close, which was a little bit awkward, but it was completed. We did. We updated all their ADA compliance issues before we sold it, including redoing the parking lot. And we actually have to go back and do a fix there. We've got to make one bit of an accessible space, but we did everything because the VCA are compliance agreement with HUD listed items for that. And we just because the people they serve, it was the right thing to do. We had funding available through the CD grant. And then we could also close out that property completely on our VCA here at the end of this year. So that happened the we were told that the veterans community project is not currently in a position to buy 1228 being first a number of reasons. They're not necessarily if they took on Briarwood, they're not necessarily ready to take on permanent housing. They are really in the transitional housing segment of work for their, their clients. And also the building might be too big. They, you know, with hybrid work and everything, they might need something smaller in the long run. So, so that is on hold, if not off the table at this point, which is okay. We'll figure out a another plan for now. They're going to continue leasing until they figure out what's going on there. They got hit with construction costs and rocketing would hit their tiny village pretty hard. So they really, they just want to focus on getting that completed. And it was one more. Oh, Zinnia. Zinnia is going vertical. And that's our latest update, right? Their element did another very similar project to Zinnia in Boulder. And they, I think they're just about fully leased up. Are they going to release up there? They're going to release up there, aren't they? But they're learning a lot of lessons that we're going to get to benefit from once releasing for Zinnia, which we'll start here probably in like the spring. Start prepping, release after that. Crispin too, I need to get an update. They are switching over to an income averaging project. And they've been working on getting all the approvals to do that with the investor team and Chapa and everybody. And that is going fine. But I need to get an update from them on that fourth billy because it is not that vertical yet. And we've been, they're prepping for, they're starting lease up as well. So they're, so they're going to phase in lease up. So they're going to do about that. Fourth building is still in construction. I'm worried about their downward adjuster on credit since over the eight. But I need to get an update on the construction side. We haven't been able to touch base with them yet. So they have laid foundation and they started wood framing the infrastructure and foundation. They started working June and they just started vertical at the end of November. So it was a good five months or so. They had to move utilities. They had to move a lot of utilities. And any update on the service provider? Recovery cafe. So we have a meeting set up with them for next week with Harold to go over their plan. We think that it is very, they're getting a lot of positive reaction to purchasing a property on Main Street. In which case they would build their, their main office and then do a satellite service at this week. And so the meeting next week is to talk about that service model. And when it started and everybody put something together and we're going to start nailing down with that service model site. What about the potential partnership for that first and main? So the first and main at the transit station one. So it looks like that is going to be, they're going to satisfy their inclusionary housing requirements on that site. Which they're looking, they're hoping for onsite provision of it and not being low, but we'll see. It's not going to be LHA on that site at this point. But we do have the Royal site, which is about two blocks away. And so we'll be focusing there. They're going to focus. It is very, very early. It is a massive, massive project. Like 400 units. It's huge. So there's a, the residential is only on both sides. There's a legislation for a state tax credit or TODs, which could be there, I'm not sure for us. Yeah. Well, if it were to be for any TOD, but that would be a year, if it were 100 or something like that. It would not be. Not them, but us. Us. Right, for Royal. That would be awesome. That's what I'm hoping for. When we start digging into Royal, we really look at TOD ease work. So sort of, I'm just going to look at a new market tax credit too, just to see, yeah, what might be a potential for that one. Is that the property where they have a big substitution? Royal? Not at Royal. There's a substation behind the cheese. The cheese importers, and that is the Miha site. The Miha site would be right in front of the substation, which we have not heard anything on very recently. The transit-oriented development is, I mean, first in the game. So that, that forms 21. So the transit-oriented development is on the north side of that street. So there's going to be a lot going on in that whole area, which is why we're not rushing on Royal, because A, we've got some things we can sort out for a bit. But B, you want to see what ends up really happening with that. And then if they're looking at master developer arrangements to see if, exactly, like, how could we tap onto something that's already, I mean, if it's that big, it's going to be big time development. And maybe they would want to do that the whole area broadly. So we're just watching and paying attention at this point. You mentioned the townhome behalf paper site need to be in townhomes. Would that be for sale then? No. Well, I mean, we haven't gotten far enough to see anything. But if it goes to LHA, then it would be rental townhomes. The only reason why we say townhomes is just to, for families and to mix up the unit type that's on that block, just to make sure that we've gotten the mix of incomes and size of households and such. Because that is a, this site, if it was 2023, that site would not actually qualify for the inclusionary housing. Because the way we built it into code is we don't want to concentrate over concentrate. And so we base that on QCT. So in 2023, it's in a QCT. And so therefore they would not have been eligible to use that site for inclusionary housing donation. But in 2024, it's really pushed my line frankly. But in 2024 they, I mean, it means all the requirements.  Don't want to concentrate, especially in an area where we're already having a lot of neighborhood problems with unhoused folks, just very much all in that area. But then on the other side, we don't have a neighbor issue to resist. It's a very developable site. It's near transit. It's near job opportunity. There's a balance. I mean, I don't want to over concentrate either, but I also want to either get it or we don't get it. We can try and put it in other neighborhoods. This site is incredibly unique, but it is completely de-concentrated from at least income restricted, except for this area. But it has no neighbors to, to raise a stake. And so in my mind, there's value to getting something actually built without having a big public uproar. Yeah. So that's just where all the factors. Does LHA ever done fore sale? That's something I think that we've talked about and rent to own as a way to transition into that. But LHA's never done strictly fore sale. And if so, then it might just do the city, if that was the idea. But that would break inclusionary housing rules because they're supposed to get the units. What if it's to another portal provider? Is this what we use? I mean, we could certainly do any of that. I don't know about this. We're way earlier. But I think if we were going to give it to another rental provider, it would stay with LHA. FISL has done affordable fore sale because they're involved with LUSA. So we could maybe do that too. But there's all options to make in the next year. Yeah, I think we would just say it's going to have a real option. Sure. Anything else on a middle income housing authority? I think it's been quiet. So we submitted the city submitted our letter on November 15 saying the city would not be towed to a project that looked like this. And the developer is supposed to resubmit with those parameters and then see if the Miha board approves it. We don't expect the Miha board to even meet until January at least. So we haven't heard anything yet. So I am just keeping in touch with the developer just to make sure to keep chatting with us. If there's anything you need to talk through any of this. But we're basically waiting until January to make sure the Miha board is going to accept a project like this. So what we said, the basics of what we said are in exchange for not receiving tax revenues on 80% units, we would want a fee paid to LHA to cover the unique services that we provide, which is access to our service coordinators, access to our public safety relationships, and all of the things that the city LHA partnership can bring to a project. And we want some 70% units because the way the middle income housing authority was set up does not perfectly fit long months. And if we're going to do this here, middle income to us is 70% units. So that may not fly, but we try. And we have data to back it up. So we asked for that. We asked for some sort of benefit to LHA in exchange as a swap, basically for losing tax revenue on what is market units here. But right at first refusal, if the state ever sells these properties, like the Miha program, because it's so new, just they decide they're not going to do this anymore, that the LHA can have first right. There are things like that, because what we included in this proposal. I'd say that the city would veto, and really the only reason they wouldn't veto, because it's still great to have a development on that site and bring people downtown. But it's no property tax on what is market rate here. So that's the trade-off, that's why we're really negotiating more on some of those aspects. All right, that's all I have. Number six, items for input from the LHA Board of Commissioners. Hey, it's Netflix. Hot off the press this morning, 8.2. So you have it wrapped in your packet. This is a, with some revisions that came in. This is where I put it. I printed two. It has some of the revisions. So you've printed the ones with your comments, and I've turned the ones without, so it's pretty good. Something we want. The ones with the comments are in the agenda, and this is the one you sent this morning. Perfect. Because we're just going to swap memory with our revisions. We met with Christie on Friday and discussed some of the revisions that we see highlighted in the packet. And the additions that we made. Number one, being the, getting in the unit to replace the batteries every six months when necessary. We're really hoping that it's longer than that. The next highlighted residence she'll live in by management immediately. The left arm of the sector. And it's getting noises, her arms. So I insured Christie that it would not make a noise, but she wanted to put that in her just in case. Just the CYA. And then Lisa had a great comment to add regarding the data she had just tested and actually made it. And then at the bottom you can see where everyone has to sign in and initial. So pretty straightforward. Addendum. Any questions or additions? So our plan is, what's our plan for using these units? Assuming the batteries are out. So right now we are testing, so when we get back up a little bit, we got a new detector with a new platform. We need new, new everything as far as technologies moving forward. We put that in a hot unit at the suites and it's running off alkaline double-a batteries. And I've had to go first. So what's great about the new data set is that I can see the battery pulled to join it out more as the other ones. You couldn't see anything. So what's interesting about this is Harold and I talked, as I said, hey, this is the second time. We were told that it would be about 14 months or 10,000 cycles of sensing. But due to our understanding after meeting with Dan is that we really think that it's draining the battery because it's trying to connect. There's a connection issue with the carrier. So the ultimate plan is that we want to send our SIM cards to them. So our SIM cards are in there for the LTE network, which we believe is where the new platform is going to be a solution. So on that note, Harold decided to move forward with anything we're still on a holding paddle because of this issue that we need to meet with Dan. And it's been pushed up, I think, until the 20th to meet with New Zealand to talk to him about what we're seeing here. I've been communicating with him each and every step of the way trying to just let him know, hey, this is what we're seeing. But Harold was to purchase 28 of these to test at the city level in the bathrooms with a memorial building at the same time. Basically every public restroom, rec center, library. So we're hoping, we're spelling because we're hoping that this connection piece, if you were a reason we're having the battery issue. Because if we have 28 of them in the city facilities, I mentioned to Harold that I'm going to pull my hair out. That's going to be a little bit difficult to manage. You're speaking to that one. Yeah, right. So he does have that fight with risk helping on this project. And the other piece of this is that we met with a remediation company out of the public area that has moved to northern Colorado. And they're rock stars, meaning that we were in with them. They are just very adamant about their process and how they're getting 0.5 or below with just heat using heat and water versus doubling everything. And the woman that she's got online with us, she's got a bubble. She's like a lead and she's very knowledgeable. So we met with them and Harold had to walk them for bins currently in the bathrooms to clean them before we put the math detectors up. And not getting a baseline of knowing what anything might bring. So that's in words as well. We're still waiting on that bit to come back. But pretty amazing process for when, you know, they're just describing it versus tearing everything out. And the success that they've had down in the public housing authority, they work very, very closely with them. So we're going to test them out with one unit at the seats because of our bins. So we got our bin for clean or bin for clean? Oh, you did. So that means we should be getting them soon too. And it means basically the same price is what we cycle had their cleaners did. Do they have much of the waiting list? Are they going to get those in the break? Well, since they're starting to branch up here, they think they can get us break with them. And they get it. That unit specifically was really low. So Sarah, are we the first ones to use these math detectors or other places use them because of the problems we're having? Several places have used them and they disclose that they have a couple of Colorado vendors. And I believe Jefferson County, they haven't disclosed exactly who. And I actually have comments from, they're not a housing authority, but it's pretty similar housing authority in New Zealand, that they wrote great comments and they've had no problems. So I don't, I really believe it is a carrier connectivity issue. So Sarah, we, so when we buy these units for them, do we have to also release their IP to process the data? So are we going to have like an ongoing thing with that as well? Yes. Well, the ongoing cost, and it depends on how many we buy, but basically Harold negotiated some, especially with the city side, he negotiated some ongoing costs due to what we will end up getting. So, yes, my question, I was looking over the draft and I, I mean, these are pretty much similar. If I'm wrong, it's in this, I'm just wondering. Um, but you know, we talk a lot about like the notice issue, which this is, this is great. You know, that everybody's going to see this and notice of what's going on. But I was just thinking if I was a resident reading this, I think they'd be a little concerned if there wasn't like an actual list of what chemicals we were like looking or testing for. And I just feel like, gosh, you know, I'm not, I'm not making math.  And then I, you know, sort of thought like the chemicals used in manufacturing. I mean, it's why people cook math, right? And they mess. They're so easy to get. Like the red P, the red phosphorus, like they're in fireworks and there are many reasons we don't want fireworks. I get that, but they're also on the tips of matches, you know, like people buy massive amount of matches to come up. Um, or iodine.  And I guess I just don't have a clear enough. I thought, obviously, pseudo pseudo. Now that all those cold medications are going to be off the market for essentially being ineffective. There's all sorts of information saying that pseudo is going to come back in a big way. Though it is being regulated, but more people are going to rely on it. So I guess my concern is with these sort of common household things that people use to make that, you know, are the detectors testing for those. That'd be my first question. And then my second question is, you know, it just seems like they'd be really hard to enforce. Like what the residents agree to number four, it says you can't use or add with any chemical, right? That's used in making math. And again, that covers whole mitigation of matches. So I guess those were kind of my two concerns. It just comes back to, I just want people to feel like safe and comfortable and they're not worrying. Oh shoot, you know, I don't want to get busted for lighting a match. Two clarifications. First, this is not going to be attached to everyone's lease yet because it's only going into clean units. So basically new, new tenants will have this attached. We can't do this for people in progress on their tenancy. So then secondly, it's a little tricky because the chemical that the detector is testing for is proprietary. Right, I read their website. It's very, it's very sparse and you have to start applying for three hours right now in New Zealand and internationally. Those are really difficult if you're really careful to cover it. But if you're a company because if you give away two minutes of information, that really requests that. And then your copyright, your trademark, whatever you're going for. So we could put some chemicals that are known to be in it. I don't know how to address that actually and be specific enough. I would say that that is not an issue due to the fact. Like if we were to do an inspection, following back up, we haven't seen anyone making math except in one part. And if they're doing the one part, they're going to have still minimal chemicals. So that is because of the methods, almost all the math now is coming in from Mexico. I'd say 98%. So we do not see the issue of people making it here. And I think it's discretion. Like if I saw, it's all about what you're looking at. If I went into an inspection and saw some kitchen pots, some chemicals, some batteries, some matches. Like I haven't seen that in years. We have not, in the city of Guam, not seen that in years. We've seen some one pots in a backpack, in a car. But I think it's literally like, I love your idea of listing the chemicals in it for people to be aware. But I don't think unless they're, unless we go in and see enormous amount of these things. I mean, that makes a lot of sense to me, but it would be discretion. Of course it'd be silly if you were going in and like, are those matches by your camera? But when you have a document where people are excited and like, I still want people to think like, but it does say in there, you know, X, Y, and Z and I can get evicted or I'm not sure if I'm doing things correctly. And this, you know, it's going to get me to some of those that I'd be worried about. I wouldn't want to mess up. Like I wouldn't want to sit on the detector and create more work for the manager for L and J.  I guess I read, and I gave your point there, but I, as a resident, if I was reading this, the last couple of sentences of that second paragraph, where it talks about if there is an alarm notification, then there's further testing and they'll verify it, they'll bring in a professional company. So, as a resident, yeah, I would imagine somehow that set it off. I can rely on them actually doing further testing rather than just proving, you know, because it talks about further testing. That's how I read it. Yeah, that's a good point. I was telling him he's not read that. Well, that's a good point. I mean, when I actually say, I'm like, this says you won't cook enough. Okay. No, I don't. It's the lawyerly background. I just get, you know, like, I worry about signing things. I think people need to worry more about signing things. Yeah. So, this is like separate of that. I mean, it might be, like, I added a second paragraph, and I want to say if, you know, you have those three positive tests, it will lead to an eviction. It could. It doesn't always. The last two, the last paragraph, and then the five. So, we'll try and set it off. Well, I think I think most people are probably, I mean, a lot of people, just basically be judgment-proof in this situation, but it's just saying that they really defy a whole harmless, which means that they technically do the LHA to be able to recover from them for all costs associated with, like, the cleaning, with litigation, with the attorney's fees. I think they're specifically listed in here. Court costs, you know, it just doesn't, it doesn't happen a lot, but I think it's important language to have in there to, first of all, to get as serious as it is and if there is a way for, you know, the LHA to be able to recover. That's the idea. That's probably the way I'll face it, because it's not a lawyer. But I know we're not. It's just a little confusing. One of the reasons is the lawyer's fees. I don't want to be eviction court. You can't write into a contract like that. I can see the lawyer's fees from you and she can't see them from you. It's only a little bit. Yeah. It's different for different courts and different areas of law. I think because this is part of the agreement, sort of a fundamental lease agreement, you can have that language and you have to have the city to make sure, but I appreciate your therapy with it. And the Fairhouse Christie wrote this. And really, this is damages that LHA would incur because of your tenancy. Not necessarily just eviction fees. No, I understand that. The thing that caught my attention, was that you wanted to remodel so you could put it on time. That's what I was thinking the same thing. Just to make sure, because that's now, so new and fresh. You probably can't recover from the litigation. And you might do right. I mean, I don't know what that is. Yeah, for sure. So it's worth, we can just ask Christy the question, like making sure that this, I mean, I trust that Christy keeps all this new stuff in mind since we've been talking to her about it so much, but let's just make sure we can answer to this really clearly. So if somebody does test positive and go through the real world as in the other stuff, if you might have victim, what is the plan for these individuals? I'm going to kick them out. Are we going to give them resources? Are they clean? Is there something available? Do we prefer them to? Yes, but that's not necessarily the property manager position. So yes, as a city, we have been doing that outside of Lisa. We understand that, but it's just like, I just want to keep these people out of the street because they're creating that problem with the city as well, right? That has been the crux of this whole issue. You keep people house, but then you are putting their neighbors and the property management and the property itself at risk. How do you? So there are resources and we are going to utilize Recovery Cafe once they're in to try and make referrals, to see outside of the LHA balance we're now moving into city. So we can, and at that point, I mean, to be quite honest with you, at that point, I mean, we have a property here in my heart right now where they're literally moving these folks in and I don't know what to tell them because I know that they're moving in people that are using that right now and that's their choice of learning that they're the problem and then they end up being the problem. So it's just to your point when you're using meth, you're most likely using fentanyl right now. So it's the ups and the downs and it's really, to be quite honest with you, people have to hit their rock bottom and sometimes it's moving out to the street and being homeless is their rock bottom. I mean, and so the reality of it. We had some evictions for meth and it was a whole team effort. I looped Sarah in. There's kids involved. We looped in way before. We have resources for them. It's something we don't handle but we make sure that there is something beyond us to help navigate the next steps. And that's where our connection and ever before due to our partnership. So I'd like to suggest that there being a court of evictions would be the type of reasons for to get some food for any of the court. Not individual fees is necessary. We've only had one meth eviction and that is because I'm sorry? We've got the one way of court online before which is pretty interesting. Especially with drugs. I'd say we'll be interested in the court of evictions to find out why we've got payments on. We have a lot of red payments but I can't really go through why the situation and how many notices we send the background just doesn't see kind of what we're going through and I'll put a lot of people moving in to make us feel like a general feeling of or knowledge of you know they maybe using meth if you already have them they might be people who are going into either police places or just be friends with a person it seems to me that it has been massive uptick or at least an uptick and people who are being thrown out so I might just put a lot of helpful information as we go forward and sort of quarterly and recall that we've only had one eviction for actual meth and when we don't put that into the eviction they can go and have it be housed anywhere else and we are passing the buck on to now somebody who may not have the resources that MJ has to manage this so that's the problem too that that's an interesting point because one of the things we do in the initial court other places try to finish the landlord so that's the way to do the eviction so it's not on people's records because you have it outside the court and the people who don't have stuff on the record they're homeless and they're back to these problems that's why when we go to court we sit there and there's some that we are up hard we can't suppress and we cannot suppress you know I'm very hard on that if we have any suspicion we know there's drug use or anything that would have set another landlord negatively we try not to suppress but like I just logic what more people will be like found out or I mean that's the whole point of doing this is to make sure that we are covering our books with insurance costs and making sure people aren't smoking up and hopefully being what we're hoping for is that it's preventative if someone cares enough about their housing that would prevent you from doing it by the way we do not know exactly what's going on right now but when was the last time we had a positive meth test having the conversation transparently and on the table at all time or a residence to know that we're not just ignoring this and pushing it under the rug that it plays a role and talking about it talking conversations talking about meth detectors having them out now they're in web two we've had them at three different properties now they're in the lobbies very possible which we that's the which we figure out faster and then we have more people from there maybe they will just keep it secret keep it secret though and it also their conversation but I mean Landlord's looking at their criteria and I've just recently heard too that people were moving away from looking at any of their civil background which is quite shocking to me and they're just doing criminal which is another huge change in laws in the last five years right you've made it harder to look at it yes I mean it's it's a result of the Landlord like going to that territory it's disallowed so that's why you set your criteria Susan and I have met about this and I think we finally have a list to bring up to the folks that are interested on a set of criteria that is legal it's it could work for people like I'm interested because I mean all the things that we've passed on in this I don't even know how to fit in part it is it's almost impossible what's that so so there's a reading that's been serving enough for my LHA to send somebody to go in just for that person's safety or whatever like who's going in and like do you envision sort of figuring out with the residents time to come in and do additional testing or you know obviously I just don't want to somebody just walking in the apartment where people are smoking that like that'd be scary so just sort of how you guys envision that going forward so obviously we're not sending like police into the group so like how does LHA going to do that if we have reasonable suspicion to believe in LHA for the ECHO like maybe an officer's in the unit they saw some paraphernalia maybe an officer arrested them outside the property and they found paraphernalia and drugs on them at that point they would post a notice to go into the unit and that is nothing to do with me and we would do our preliminary tests as we happen and then go off that reading I'm going to be a manager to do that currently wearing PPE and if it becomes a positive then we bring in the industrial hygienist we have that suspicion I have it's two staff members one stays at the door and we have Narcan and everything else too so gloves, booties full cover suits whatever we need so unless you want to wear it alone there's a unit or two we might have needed them it would have been good we have that option that's the problem when we have come across paraphernalia in the unit we had no feeling that that was going to happen and then you're going in and turning the unit and it's on the drawers so that was the last little current one the one that were waiting to be cleaned resident was in the paper she got hit by a car deceased we've got possessions from the daughter we went into the unit and the daughter opened the door and found paraphernalia so that gave us suspicion to test and were you wearing PPE when you went in you just put it on one second well they have gloves on because of the roaches right but yeah it was a mess everyone I've had seniors he was a 78 year old it was an eviction for hoarding and we started cleaning out and we found 10 plus they had a call Sarah they weren't even showing up to that one and had a call for her to come pick it up we had families parents with young kids 20 to 30 it's all over the board no racial or ethnic patterns to be found for each there's no socio-economic differentiation to everyone going back to turning the units is there a mobile testing unit that you guys would want to throw in the room for a day before you actually go in so we have any suspicion we have it's a hoard swaps that tests the whole unit and that's what we do if we have any suspicion they had criminal activity and we met it just $40 and it just gives us a piece of mind or knows us how we to tell us how we need to proceed but in that situation where you get in there and you have no idea and you pull in the rope and you're barbie exposed everybody who's been in the room is it better just to make it I don't know what the cost is so maybe it's cost-effective but you just make that part of your process I think wearing gloves and boobies should be part of the process I don't have 10 eyes yet or 50 arms so I mean and I'm kind of saying that jokingly we as a society still have not determined like minus the the child's looking a wall that has meth on it there's a problem but for the adults that are going into these units that are exposed there's no medical or scientific anything saying that oh that there's just being in the presence isn't correct same with it I mean looking at the fentanyl piece too like we can go into a unit and someone just smoke a fentanyl that is not going to put you down it's touching it touching it will put you down so so so the community managers when you think about property management you don't think about always not maybe this group this group probably does more but the general public don't think about the rest of the community manager faces every day there's some very very traumatic situations that they witness or are inservant into and it's on par with this is what Harold says all the time it's like on par with public safety they need to be trained in things that public safety needs to be trained to encounter so when it comes to that it's that it's similar insurance coverage that we have with on the city side we also have peer support and we bring in counseling services if there is a traumatic event which could be one of many things it could be coming across some that deceased or a period of time it has been stuff that you can't ever unsee in your whole life or a shooting or something just like any community resource worker in any community so we do have the same type of coverage that the city has for anything like that and all we can really do is be prepared to use PPE, have our training there this is part of the job and the tough one my staff is required to go through quite a few trainings that the city puts on everything from the escalation drugs, the police department puts on there's a whole laundry list like even driving with you know before we just take you back off the city's training so anything that would apply to our position that's taught by a police officer police are involved in, we're taking final thing on this list, when it goes live I hope the mold and moisture goes to the bottom and then the other this is kind of silly but should number four be number one the president shall not manufacture used instead of they should chamber with the detective and I think you should pull out resident as responsible to take actions of their guests and make that one bold too because there's a lot of times when people get a lot loose at BCH before they've been evicted because like their daughter is staying there and losing an elderly parent who is not a drug user is not on the streets and then just on the third line the day should be any number four I also hate reading text like this can we fix the text where it looks like it's like a pdf converted to word and then the spaces are all wider I took this off of what she says and it had a change to word to pdf to print it without the contents so you could just run a deep unparmed every day and redo it I know it's a pain in the ass but that's a stupid property so the spacing learns that's a lot of work so the recommendation is to put them above the door I mean it would be I think property dependent because if it's a studio you'd probably want to put it above the doorway to the bathroom again the we need to look at where people are using it most often bathrooms, bedrooms not likely out in the living space unless it's a studio if the suites is a little different so I think it would be unit specific I think earlier you and Harold said that they didn't think about stuff going out of the hallway so that would be my concern and I think it would be a false false just because somebody walks by you know how close it is to that hallway what I'm seeing even if we were to get a small sample the gas sensor has to be literally taking a sample at that time and it wouldn't be significant enough to trip us it has to be a 60 or above the other day we had a 10 it's Green Creek and I was like that's kind of interesting so Lisa and I moved to the camera so we can kind of see and go back and I can see who walked through there at 6.30 right from that time frame but they're still in the air but that wouldn't trigger and I don't know if you would see it but a 10 is like nothing to really be like that person who was with someone and it got on their clothing you know like 50 to 60 years what we are concerned with and that's what we're going to be saying it's 50 or 60 that's a little correct and then we can if it buzzes but you know what I mean if it goes off we go in we can do the swap and if nothing comes up then we can say maybe it was a walk they had somebody you know on their clothing and I get so anytime it's tampered with or moved I get an email so that's what basically triggers the so called along it's silent it's an email and you can put as many people on the email which you want right now it's just me the only thing I really want to know is are we taking the batteries out moving it around so I don't know how much information I've been able to share with you but is it pretty much what they're looking for is just like literally just the exhalation from the smoke from like it's not really looking for anything else I mean it could possibly but not for what we're measuring and then when we're testing using the test kits and swabs how the unit I mean how sensitive it is it's pretty accurate if somebody smokes in a unit it won't pick that up it's kind of like a range like it's when it gets darker the more there is kind of like a pregnancy test almost the darker it is the more and then we bring in the hygienist so but we can look at the test and be like okay that's nothing that's not going to test high enough but based on all the ones we've done the areas we've tested and comparing them to the hygienist test we kind of can tell by the number okay this is going to be a very light clean so if we have low reading we're sort of over in the other to confirm the number make sure it's low so our vacant unit plan assuming we make the order of the machines then what properties are we focusing on the place I know is one because we're going to attempt to if we have well this is what we talked about but if there is contingency left over in the construction budget we could use some of that to purchase the initial machines but the problem is we're going to have to move back in starting as early as March 1st so if we have them by March 1st we can start putting in the clean units and then adding this addendum on when people come back we could just add the addendum in the interim and because everything in the unit would be new and we would test and verify that everything is negative and then so we could do that but that means we're ready to go March 1st for the first round so we've got 12-ish units going March 1st that was an idea though that we had it using those places and then using vacancies we're just going to call it the village that's what I'm going to try and say for now and then others it's just going to be ad hoc additions when we have unit turns so that'll be across the properties we're not planning on focusing on any one property right well in Julia she runs the demographic we're seeing it all properties so it's not like one property we're going to focus on what's the cost for it for the alarms 600 but if we buy them they literally go up we'll have them go up 5 or 550 and then ongoing cost depends on how many you have it's a 28 I think it's like a $3,000 on the road for the 28th of the city and that that's put some reduction in terms of the differentiation $3 for 28 units for one year and that's for IT management the data fees problem solving what you do at the cost and what we pay for one unit basically what we have to do all the testing, the cleaning the econ just even the unit right now just needs a cleaning by the time we're done with that it's low level we're going to be out 12 to 15,000 and that's us or is that insurance no insurance and if we found any insurance they pretty much said that because of all the claims we have even going with the pollution more likely it would be because we have to we're done with they did say it took a way out of that coverage what have they reinstated if you hit a certain percentage of unit speed monitor or something I don't know most don't get it so it sounds like we were pretty fortunate to have it but they have to look at their respectors so whoever they're actually insuring at the time the population with our insurance company is all having abilities that's who they insure they could be seeing a risk across the board and so it wouldn't just be us it's their whole risk so even if we can show them once we've got the detectors in place and our claims go down they might still be using an algorithm that doesn't take that into account we tried to have a conversation with them we want to do this but we don't want you to be alarmed I'm not intended but they just were like we just don't know we just don't know enough and we knew that they were worried that if we just started testing units randomly that we would have a spike and they don't want to be around for that part so that was a very successful conversation it seems to be an industry-wide problem that we'd have to solve it's not just us so we will certainly try again once we know that our claims have been reduced or we've got this pretty locked in but if we're monitoring it we might be self insured at that point why go out and see other insurance companies? right well technically it'll only cover one because you get a cap of $100,000 so that's one thing so one tear down but if you're actively monitoring that's the goal is to minimize that back to the reporting software that the manufacturer gives to us the city contracts definitely they're on it they're on it they're on it they're on it they have a license and I'm going across every year is there any prescription on if they can do it there? the contracts the city contracts these days are pretty robust about data sharing and making sure that you like we had to test we had to confirm that the pieces that go into these detectors were not manufactured in certain places in the world for example so the city side of contracting is pretty on top of that and there should be no identify all data it's not tied to the person feel good with all the the numbers what's the number of days I thought I saw it says within blank hours of positive test results that's a blank the attorney let the blank for many reasons to test and that's what we did like if it's an extension if it's a book we need in there 48 hours it could be that we have certain circumstances that we might want to do 24 we should be consistent then across at least the property I was thinking that if we're saying within then it is a maximum that we could just put something in and then we're still covered we did it yeah we have to give them a notice as well in order to get into the unit do you have to give them a notice to enter a unit? yes so if you're in the unit and you see math pipes what do you do with them? if you're in the unit for a already done I'm going to talk about the alarm alarm goes off do we need to give notice to them whatever the notice is for the weeks maximum 3 days but we don't have to tell them the alarm went off we couldn't say that we can give the proper notice we don't have to tell them why we're going in it's just a what would you call it a 24 hour research to see what your clean evidence there is there is chemically on the open market you can use to help lessen it yes hmm sorry sorry what are the pensioners generous January 16 this will go to court yes yes some green right and I can send that that's good number 7 really quality of life anybody hearing on this that's down it's going to number 8 the operation so so in November about 95% occupied we're renting but not at a fast pace right now with the weather and a lot of people don't want to move this time of year with the holidays and everything so we are trying to get people lined up on the back burner Austin Meadows neighborhood I certainly have three that stand for that those all require some type of little bit of rebuild one is the one that we've been working on for almost two years it's coming along it's going to be drywall about to work with everything else so hopefully that one's online back right around the first of the year actually hopefully all of you are back online right around the first of the year senior we have five vacants but I got three rented they actually rented all of theirs that were vacant so we're hoping to end the year pretty good it's just trying to work around people's holidays that go by them all don't worry don't worry as I say that it's coming to play the sooner we can if there just you so the two units you may just do in-house for Austin neighborhood because they're not too much and what the contractors want you can take care of it you can do it in-house faster I'm just I remember last year we were worried about investors were worried about our rent ready rates for their tax credit because you can't have a tax credit all year so technically if you've met one they've been clean so they are they are habitable with what needs to be done yeah so they're not the only way that's down down is that B2 still at A&M yes I know what you're asking I just know we went round and round and asked you yes so still have a few units down for month we have 731-2 I believe that one should be almost done that one should be back online by the end of the year as well 733-0 we do about 15 minutes now so the prior ones will be done by the end of the year being in that unit it's CRC Sarah and I met for one lady she worked for LHA way back in the 2000s she would have the Metro Denver apartment association orientation we went to so we had a bit at our end is it clean or clean though public health and safety just a note we didn't do property updates just because everything they reported last time we hadn't done much so knock on the wood things went good senior properties have been super quiet and the suites just steady so the residents are up and down in their days having good ones and bad ones but I think overall we're really trying to find out if any of you started to use this ROI everyone's spending a lot of time giving so we're waiting to hear back about that we didn't hear anything she said I was waiting on Andy so everyone has signed this and kind of verified it so I think it's back in MSB's lap and we'll circle back with that but really getting those ROIs when needed will help all of us strategize to assist these folks when they're spiraling does everyone know what you were saying when was the ROI that was the least mentioned and I apologize for not getting yearly cost of service in detail for you I have not had a chance to do that but my plan if you are interested I can have that for January so you can kind of do a recap of the last couple of years and dive into a little bit of what we've been seeing more and less of I think that would be good because I know you've shown the calls for service when you have to follow up through your copy of conversation and how many of those were you so absolutely I think I would show that your involvement in diving into the data and kind of when people think of an overall call for service being super high well that's not really the case and we were able to really articulate that to even public safety staff regarding an issue recently looking at calls for service on some businesses in diving into the data and seeing what calls are actually going on so I'll get that ready for January I was just talking about how clinicians do they added a 20.4 when we started posting those yeah so human services is taking the lead on that and I just got enough data I was going to see what it was I think that they finished the job descriptions they needed to re-benchmark the clinician one because they just knew the salary needed to be looked at and that changed it needs to still go through HR for benchmarking okay yep I just I will follow up it looks like she sent our job descriptions to HR one for benchmarking and I think we're going to just be finalizing these job descriptions and the pay rates here in the next couple weeks and then probably at this rate because HR and benchmarking takes time so we're going to be posting in January as what I anticipated that's the resource position that we're hiring for the senior sites so this is the one that's not the clinician it's the position that we have had and we are focusing off of the suites and on to just the typical properties to be accessible for more residents so that one it's all in LHA wheel cord working with senior services to help hire balance update from the number nine other business I just would like to say that the landscaping at Aspen senior looks very nice and so I understand the weather and everything now but what are the plans for getting the rest of the rocks out of the other places I have a staff member who's getting bits we're a little right on the thing that HR is going to pay for and then weather permitting we've actually had quite a few maintenance out with injuries in COVID so we've been running on a limited staff now the plan is hopefully maybe after the first of the year we can get some nice weather to get to that I know the weather weather and we need to lift the rocks they're large that's it it's right it's during nine at 10.22 next week


The present study updates a previous evaluation of the cost-effectiveness of the herpes zoster, HZ, vaccine from a Swiss third-party pair perspective, using updated vaccine prices, a different age cohort, latest clinical data and burden of illness data. The results show that vaccination of 15% of the Swiss population aged 65-79 backslash ZA0Y would result in an additional cost of 17,720,087 Swiss francs and gain 594 quality adjusted life years, QALYs, with an incremental cost-effectiveness ratio, ICER, of 29,814 Swiss francs per QALY gained. Sensitivity analyses showed that the results were most sensitive to epidemiological inputs, utility values, discount rates, duration of vaccine efficacy, and vaccine price. Probabilistic sensitivity analyses indicated a more than 99% chance that the ICER was below 40,000 Swiss francs per QALY. The updated study supports the value of an HZ vaccination strategy targeting the Swiss population aged 65-79 backslash ZA0Y.


So I'm Adam spires. I'm a senior software engineer at Sousa working on Sousa open-stat cloud And my colleague introduced himself. Yeah. Hi. So maybe some of you have expected Marc Houdara to be on stage today, but unfortunately he couldn't join the Omstech summit. So I will replace him. My name is Boris Mac I'm like mark part of the cloud infrastructure engineering and architecture team with an SAP So today we will talk about the customer-vendor relationship between Sousa and SAP and how we successfully established DevOps model in this relationship Yeah, and so this is a quick agenda of what we're going to talk about just a setting the scene Regarding the open stack strategies of the two companies Then we'll look at what a typical vendor customer relationship Looks like and how that caused some issues for us and how we Felt that we needed to go beyond that and develop a new model So then we'll go into how we develop that new solution and what the benefits that we've got out of that so far are and We'll talk about what we're planning for the future and then we'll wrap it up Okay Yeah, let's talk about Omstech at SAP as well as on the Sousa side. So I will Start With the summary why we choose Omstech at SAP as our platform As you can see on the diagram the SAP IoT platform is running on the so-called SAP cloud platform on the other hand the SAP cloud platform is using cloud foundry as its platform as a service layer Also, SAP wanted to be vendor neutral and then avoid any vendor lock-in. So these two points the cloud foundry platform as well as the vendor neutral To be vendor neutral and let us to the decision that Omstech would be the right infrastructure layer as the fundament of the whole stack and Quick few words about the product that I work on which is Sousa open-stack cloud And that is one of the products the one of the cloud products that SAP has selected to build as the Infrastructure as a service platform and they build everything else on top of so it's an enterprise distribution of open stack it's fully free open source software and In addition to packaging An enterprise polished version of open stack it includes some additional deployment management capabilities So let's talk about me. Let's talk a bit more about a Customer and vendor relationship How it was applied That's okay, and how it was applied, but I was applied between SAP and Sousa So as you can see on the diagram We have an internal customer within SAP which is consuming the infrastructure as a service platform, so this customer or stakeholder is talking to the SAP infrastructure dev team for its requirements the dev team is trying to Set up these requirements and establish them in the production environment and so we have a connection between the dev team and The ops team within SAP the ops team, of course is responsible of running the whole stack and Operating it in case of any problems and failures the operations teams can contact the vendor in this case Susie L1 and to support layers So if the Susie L1 L2 support is not able to fix the problem They need to do some internal escalation to the Susie cloud engineering and Finally, if it is not a problem within the Susie cloud product bus But maybe a problem on the Susie enterprise Linux operating system the Susie Cloud engineering team needs another escalation to the Susie's less engineering team But that's how it should be in an ideal world let's come to a more real-world example because we have not only Stakeholder talking to to the dev and ops team We have also the platform operations and as you can see the communication that's much more Difficult because the the platform operations is also talking to the dev team It's talking to the ops team in case of some problems and errors the SAP dev team Is not talking to the ops team and the ops team to the to the vendor, but the dev team will directly Speak to the vendor to speed up communication to simplify communication and It's hard to get the big picture and get an idea of who is working on on what topic but Do you think that that's already the Did a full picture of course not because we have also management involved So in case of a real escalation management always want to be involved In the in the communication and communication flows so This means of course some extra communication on the one one hand this can Can hinder because It's it's communication flows on tops and as you can see it's really Confusing communication flows now of the other side and on the other hand This is sometimes required to get the right and priority of the Of the error of the resources on the vendor side so maybe You remember this picture we had before and you are wondering why the beautiful Simplicity of the support diagram feel familiar to you. So let let me remind you where you may have seen this before Okay, just talking Let's see if you haven't seen it before it's the open stack architecture diagram Which is one of the it's what it's the the image that is like it has to go on every open stack presentation everywhere It's like as a rule And it's yeah, beautifully simple Okay But what are the issues with the standard support even if everything is working like expected? Yeah, so the first issue as we've seen like the information can take multiple hops to get from The the source to where it needs to go if somebody needs to finance some information So you get this inherent latency in the system of communication and Similarly that makes it really hard for people to get visibility into what's really going on if they have to Ask somebody who needs to ask somebody else who needs to ask somebody else and so on so the the they can the all the different communication channels and the the hops between them really contribute to this lack of visibility and Similarly this has an impact on the ability to respond quickly when things happen because it can take time for the some critical message to get to the right person in the in the right team and And so that that's a kind of short-term issue in terms of agility, but there's also the long-term issue as well of if the the customer wants to Feedback to the the vendor and kind of Give real-world expertise and guidance around the direction the product should go Then that the efficiency of communication is important on the long term as well Yeah, so let's have some real-world example or should I better say it's a catastrophe So it wasn't a beautiful Saturday morning. We had a well-prepared maintenance window Everybody was feeling like let's get starting early finish early have a nice weekend enjoy the weekend but Of course we during the maintenance we run into some massive network problems So the VMs stopped to communicating to yet each other and we had no clue why this happened Yeah, because you just wanted to apply some home stack patches, which doesn't relate to any networking feature At the end with we were able to solve the problem But for this we need to start a restart and all Nova services and OBS services on all of our hypervisors So Yeah, I will continue So let's get more to the details of this outage. Yeah, as I already mentioned it was Saturday morning. We were well-prepared for the maintenance Everybody felt comfortable because we had also Suze on on hand. So Adam was connected to this maintenance bridge and We were like, yep, everything will went well However And Adam has had no console access to the environment, but there was a screen sharing session available So we started applying our packages and updates through on the configuration management. Everything was fine and then suddenly the VMs started and Loose their connectivity as I mentioned before we had no clue Why this happened because we didn't touch anything on the network layer and just applied some Open-stack patches. So what to do now, of course, we had to travel through the problem We had to extend the maintenance window So luckily several hours later We were able to stabilize the cloud again and everything was running smooth again, but Due to the limited visibility what operations did to fix the problem and Nobody really could could answer why this happened and we had we didn't had any root cause and Of course, as you can you mentioned, we had to extend it to the maintenance windows for hours our customers really Happy about that situation Yeah, so I was one of the lucky people Involved in the the whole saga from beginning to end so Like Boris said I was in the maintenance window And then on Monday morning the the root cause analysis Started in earnest on top of what we'd already Gathered on Saturday so we started collecting debug and this was really quite a laborious task because there were huge numbers of log files and We needed to get them over to the suzer side so that we could analyze them in depth and run some of our analysis tools on them and At first we weren't sure exactly which logs we needed So we collected a bunch. We did some initial initial analysis It was clear that it was the problem was related to open v-switch in some way and we we had a sort of initial Conclusion that we shared with SAP a couple of days later And we kept analyzing We were still trying to get to the bottom of things had an internal call did more analysis Eventually we got to by day 12 the place where we thought okay. We've got a pretty good Understanding of what's going on. It's not like a smoking gun. They were a hundred percent confident But it does look like we found what seems to be very likely the culprit But of course we were not satisfied with that so we kept digging and eventually when we'd Brought in all the possible experts and spent as much time on it as we could we finally came up with something with We were happy. Okay. We really understand what's going on now so it was it was quite a long period and the In case you're curious, you know what briefly what was the the cause of all this? So a while before someone in the SAP dev team had Needed for very valid reasons to test a new feature. They didn't want to test in production just in a you know other environment And this new feature relied on neutrons L2 population feature, which you don't need to know what that is if you it's not really important for the Purposes of this, but it's a it's an option in neutron that you can enable or disable and the code change to enable it Accidentally got into the ptf package that we built for production which included a bunch of other fixes that SAP needed For their production cloud, so it got accidentally enabled On some of the nodes that we updated during the maintenance window But not all of them and this was a big that really the source of the problem So why did it take us so long to figure it out? Well, firstly neutron L2, especially with urban beast, which is just a really complicated beast and there's there's no way around that There's you know hundreds thousands of log messages that you really have to be an expert to understand and And the the fact that we had accidentally enabled this feature on some of the nodes But not others meant that we were effectively using an invalid config and there was no validation in open stack to catch that invalid config and It was probably some situation that maybe you know No one else in the open stack community had made that same mistake before So the resulting behavior was not only undefined but undocumented and you couldn't Google for it Or really figure out You know that recognize it as as bit as coming from this this cause the other mistake that with hindsight from I guess as soon as I we made Was to not involve SAP enough in the in the ongoing investigations Because they did have you know some great knowledge about exactly what happened during the maintenance window that could have potentially Led to us figuring things out a bit quicker But that it wasn't the the main cause of the the difficulty and Of course While this was all going on we all had other responsibilities that were competing for our time with what was going on So we learned that the organizational structure was not set up for Succeeding in the way we want to And the communication while this was going on was all over the place like the diagram you saw earlier We did waste time collecting logs because we collected some Early on and then realized that we didn't have the right logs And then we collected those and then we realized that they were still not the right ones because the logs that we actually needed Were had been rotated because they were from an earlier time period So we have to collect again and so on and this is clearly not the most efficient way of debugging So remote access helps a lot with that because you can just get the logs that you need straight away and And similarly if you have remote access, especially in a joint vendor customer situation, then you can do screen-sharing type tools through the same terminal session using like screen or Tmux or something like that and We also learned that the the process that we had for building these custom packages For customers who need fixes Earlier than they normally get released through our maintenance update process was not rigorous enough that you know, they're manually built and it's hard to keep track of of all the fixes that you're building into that ptf and We didn't have in rigorous enough testing around those particular packages even though the rest of the products and all are Officially released maintenance updates do you get very rigorous testing? This was a bit of an Achilles heel And finally, of course open stack is just a complex thing as we all know, so There's not much we can do about that in there in the short term at least Yeah, so Like Adam mentioned we had a lot of lessons learned But what to do now and we decided to have a two-day joint meeting together with SAP and SUSE folks to See how we can can improve the process and the situation here So yeah, what was the outcome of this? Of course, it's a new DevOps model approach together with SUSE So we apply this new DevOps model and Yeah, as you can see it looks quite different to the To the drawing we had before so we really try to avoid all this cross communication flows we had before So as you can see we have two main tracks one Still for for development the other one for operations, but we have now in the middle this DevOps approach So let's begin from the top We still have the stakeholder and the platform operation guys But they are now on this on the same level and they Need to sync each other if they have new requirements Second we installed a new team, which is called in our case the infrastructure architecture team As a new layer Between the stakeholder and and death So the architecture team will Validate the requirements and if they are not valid give it back to the stakeholder They will finalize and prioritize Also the requirements and coming from the customer So the architecture team will hand over this these tasks to the dev team and As you can see the dev team has now direct access to some of these user cloud engineering resources So they can even work together and on some of the topics if you have bigger topics We are both involved So and for this dedicated suzer team it is much more easy to involve the right resources Within suzer if the other resources from the vendor are required Rolling out new features. This will happen in this DevOps part. So not Development is rolling out new features and or handing over to operation It's working again Features big so it's it's a joint approach approach here with deaf and ops people and even if it is bigger rollouts of Of some Features which which are Not it not it not that easy To roll out the dev ops team can can easily involve the suzer team for this rollout in the production environment and so also We have closed With this approach the gap between the deaf and the operation team So if the operation team has some problem during the running the production and the daily work, they can easily Connect to the development colleagues for further help on the right track, we have still this The daily operations flow where the platform operations if they see some incidents or have some incidents They can still open a ticket on with the operation guys and operation guys will will run Yeah, we'll run the production and operate the production environment okay, and on the development side like on the technical side we decided On a new approach that was to do effectively a friendly fork of suzer open stack cloud There was specific for SAP that had the short term fixes and enhancements the SAP needed Urgently so we could deliver them to SAP normally Faster than we normally deliver through our standard update process But we didn't want to maintain a long term Continually diverging fork. So we agreed that everything should be Fed back into the the mainline product and of course that that allows our engineering to scale in the normal way rather than having yet another Product effectively to test and maintain and a whole extra code base and We use agile practices within suzer as standard So we wanted to keep that going in this context So that means things like daily stand-ups obviously testing and see I in general and a shared Sort of product backlog for this This new you know product in the sense that it's a new code base stream delivered to a single customer effectively so the way that we Decided to host that was a new github organization with shared access from from both sides that has forks of the product repositories that That we expect to deviate from from our standard product and each repository has an issue tracker in it which is of course shared to both sides and We also use this github project feature where it supports a single Kanban board type of view of all of the issues from all of these repositories in a single view and We developed a branching and tagging strategy for the git repositories Which is that that's not rendering properly Okay, that's a shame if you if you at the end we have Again the URL for the slides so if you if you follow that and you view the slides on your own Laptop, hopefully you will get the proper view for this. I don't know what why it's not displaying but there's a git branching Diagram that accompanies the comments, but effectively have a mainline Stream that we already have for excuse open stack cloud and then we branch off that for each cloud each cloud the SAP is deploying whether that's production or test and So each branch effectively for represents the queue of changes that we want to deliver to that cloud and then we tag Every time we deploy to that cloud We we make a tag on that branch so that we can go back later and and see exactly which code has gone into that cloud So it's typically one tag per maintenance window But it could be multiple if if we forget something in at the beginning of the maintenance window and then make a correction later on And we merge the changes back into the into the product like I said before So we get from this improved communication because we're using github As the focal point for the communication previously we were tracking internally in suzo We have a bugzilla instance that we were tracking The the technical side of these issues But it had limited visibility to SAP basically by default bugs are private and we wanted to share them with SAP For visibility on their side, so we had to CC all SAP onto all those bugs So they have access to the historical information, but it's it's you know for stuff going forward We're using github We have a shared slack channel that we're on you know all the time throughout the day So we can quickly message anyone in either company and we have the daily stand-ups Like I mentioned before Yes, and so all as Adam already mentioned so the it was really key to enable remote access for the render and So we we gave through to the remote access to the to our development as well as to the production system Hmm, of course, we needed to take care about the right access level for the susa engineers So they should only have read access to the to some resources And not full access but this improved working together a lot as the susa engineers can now examine Mine and directly the the production cloud and also we don't waste any longer time in collecting locks or let's say the right locks Anymore there's now Also more in-depth examination possible for the engineers and finally Yeah, I didn't already mentioned we couldn't work together more if Officiously because we could use tools together like screen sharing Or like Sharing the console on a on a screen session yeah From my personal perspective that that was a huge boost in Making making my life easier helping out sap Having direct access So on the ci front We decided we wanted Essentially to make sure we could make code changes Faster get them deployed to the sap clouds quicker, but without increasing any risk So we wanted to obviously test everything before we deployed it But just not test it as a fresh deployment, but but test it as an upgrade path from a cloud Like sap already has deployed. So we're really focusing on what is the impact of of upgrading Rolling this change into an existing cloud that's At a certain state already So the way we did that we we could reuse a lot of the existing internal ci that we have at susa For our products. We have a bunch of build and test tools and methodology and Fortunately a lot of sap's existing infrastructure was very accommodating for that. There's a lot of commonality between the two environments Of course, we wanted to focus on sap's hardware and software stack and their particular open stack configuration Susa opens that cloud is a very Flexible product in terms of how you deploy your open stack cloud But sap have decided to go in a particular Root in a particular set of configurations. So obviously that's the one that this ci Needs needs to focus on so we have daily builds Various forms and then we also have gating for pull requests So every time a pull request is submitted the ci runs on it and we can see how it performs So the components we use for that obviously github again As soon as we have this thing called the open build service, which is a free open source project that anyone can use to build packages For multiple distributions actually not just for susa distributions. There's a many different distribution supported by it It's a great place to host your package building Processes and it's certainly the the core one of the core components That we use at susa to build susa opens that cloud And it was the the ideal Component to to use in this environment as well for building custom sap packages for testing in the ci And the whole thing was orchestrated using Jenkins, which is I'm sure you're all familiar with So, uh, how how have we done now? It's important to point out. This is still a relatively early approach to or relatively young approach to this Change in in organizational and collaboration model. Um, so it was only around february time frame that we started so It'll be interesting to see how it develops over the next six or months and and beyond But we've had some positive results already And so a couple of examples, um, there was another uh, a recent outage Where some packet loss was reported on network nodes in in the morning one morning and Obviously with susa got visibility of that very quickly because now we have the slack channel So sap people can just message susa people straight away So obviously, you know, we took this very seriously immediately reacted And then a short while later, we also discussed it in our daily stand-up and as luck would happen we had um a guy on the on the stand-up who Before the organizational changes. Um, he would not have had any direct involvement with sap is kind of specialist automation engineer But due to these organizational changes he attends the stand-ups And uh, he heard the description the problem from sap and immediately recognized the the issue was something that he's been looking at for the last Six months. It's an ongoing open v-switch Issue and he knew a workaround for it straight away. So so we went from Pretty much a critical situation that we were all very worried about to Suddenly feeling, um, maybe I shouldn't say relaxed, but certainly a lot more comfortable with because we already Um, had a good understanding of the of the issue And how to mitigate it Um, the second example was we had um on a Tuesday evening, uh, an unexpected restart of the message queue and, um When we started to look at this in earnest, um The uh for for engineers including myself From my team we're all able to directly access the production cloud that was being Affected and when we took a closer look and we produced some statistics from the metrics that get gathered And you probably can't see any of that at all. Um But I don't know there's one there's one square graph in particular Which is in the first column and it's the Well, it's the one in the middle basically, um, I don't know whether you can see it But there's a very sharp increase in in in a red line which represents the load average of that particular node each of these Five nodes is a is a control node And we we saw that the load average on one was just going up sky high to about 140 150 At which point the the node just completely fell over And that was obviously a very Bad thing to happen and so we looked more closely at the statistics And we found another spike Um, which again is in the first column. I'm not sure if you can see it But it is the um fourth from bottom and it's a spike in udp v6 traffic And as soon as we saw that then we Um lightbulb started going off in people's heads and we effectively realized it's correlated to nfs activity and there was an issue with one of the nfs servers Which explained that and What we were pleased with in in that incident that is that we were able to With this kind of combined firepower of four engineers all logged on at the same time We very quickly honed in on the right thing In in a speed that we have maybe not managed to to do Previously so that was definitely a nice benefit Other than outages um that there are some other benefits So working i've i've been working on a new feature recently for example around api rate limiting And it was really helpful for me during my kind of research Phase of trying to figure out what's the best approach to this It was really helpful to as i was looking at different technologies available and different configuration options To be able to message my sap colleagues who are the stakeholders for this feature and say hey, you know Do you need this particular approach or do you need this one? And to have that kind of agile approach to The the design rather than having sap you know write up some lengthy Specification of the feature that they need and in advance and then pass that over to suzer and then analyze that It just felt like a much more efficient way of collaborating Similarly with short-term support requests, which Traditionally go through this this system Which i as an engineer don't have direct access to and for long-term support requests for new features Again, we have a a system that is is a kind of more heavyweight process So yeah, it just all felt more efficient So far just a short outlook what we are working in in the future on together with suzer and sap so So we want Oh, sorry. This was your part still. Oh, okay. Yeah. Well, so yeah, I mentioned that I was working on the the rate limiting and so the reason for that is that the The customers of sap Are tending to use the open stack apis From a cloud consumer perspective very heavily. So we obviously need to to mitigate against the potential for one customer Having an unwanted impact through overloading certain api services And we're also working on other things like improving neutron l3ha and And other bottlenecks in the control plane And it's it's handy to be again to be able to discuss those directly with the stakeholders on a daily basis So now what's planned for the future? Um, of course, we want to improve the monitoring at the moment we plan to have an elk stack available in within the suzer product so we can really as a customer now influence the the suzer guys Um to point them in the direction we want to see the product um Yeah, we we will work together on a centralized management because we have the idea to to deploy more Open stack clouds um in in multiple disease with multiple availability zones and multiple In a multi region um deployment so yeah, really for our operations and To have the visibility in one point over all our cloud installations We want to have decentralized management and um, finally, of course We we want to improve um the ability of the cloud overall so uh to wrap it up the um Traditional approach to uh vendors and customers working together. Um, of course it works in in many situations but it it's not the only way to do things and um, it it sometimes can be a healthy thing to to try considering other approaches um, and The the obvious sort of rebuttal to this approach is like well, that's great, but you know, it's it's expensive for Uh, you know suzer to dedicate engineers to a customer It's expensive for the sap to you know dedicate teams and have have this sort of It like it does take extra effort And and that is true, but It's not an all-or-nothing thing. You can you know steal some of the some of the approaches we've taken. I think apply in any situation No matter how how big or small the the relationship for example, you can You know as as we're all working in in open source and open stack believes in the four opens as a community It you know having for example an open issue tracker makes a lot more sense In terms of collaboration than a closed one And things like ci any any customer can potentially build ci that then integrates with the the vendor ci and adds extra voting or non-voting input to to changes in the code And and of course anyone can you know fork a code stream and submit changes back again because Working in an open source environment So and last but at least we think it's a win win situation for everybody for sap for suzy for open stack Because sap as I mentioned can influence the roadmap of the product suzer has The ability to work directly with the customer to improve the product On the customer needs and don't and also have access to a production cloud not only to two small labs And yes as we mentioned all our fixes will go back upstream to the open stack community So I think we are the end and yes, unfortunately, no time for questions, I guess But it's worth to mention that we are hiring so if you want to be part of a great sap or suzy team Yeah, just talk to us after After the session