If that's largely made up of small cultivators, the resources that would be required to make sure that we can adequately have a good compliance and enforcement program as part of our larger consumer protection program, I think once we kind of get a better ballpark on the market size and of what we're expecting, Kara can take that even more into consideration when he's looking to get us recommendations from our resource perspective, but it's certainly part of the conversation. Great. Yeah, I just want to like the staffing up of that to me seems like, you know, it could be another kind of another big hurdle that we have to overcome. And, you know, as long as everyone's on the same page and we all recognize those, I think it's it's tough because, you know, he's not in the market structure subcommittee, so which is kind of, you know, the compliance enforcement is the other side of that. Yeah. One of the other, you know, questions that I've posed to that subcommittee is, I think I said this last week at our Friday kind of wrap up overview of the week. You know, right now the hemp the hemp program tries to do random inspections of 20% of their license holders is that and I asked this subcommittee from an adult youth perspective, understanding we have lab testing and hopefully a culture of compliance. What's what's a good number from from that perspective to aim for? And again, that just that lends itself to your point on making sure we have an adequate, you know, compliance team and making sure that we have the resources that we need to do this effectively. Yeah. Great. Anything else on this on these two subcommittees? Any direction for Kyle that we need? No. Okay. I will say, I will say, we just just sorry, Friday morning after a long week, trying to just remember everything. One of the other things we did discuss in the sustainability subcommittee is active 50 jurisdiction. It's something the board's aware of and wants to make sure that we have a good understanding of because this is technically a commercial product, what triggers there would be for growers at various different tiering or sizing of an operation. So we've done a lot of outreach to other agency partners to kind of get an understanding and we're hopeful that the board can to the best that we're able to with active 50 jurisdiction anticipate where triggers might happen. Julie, just do you mind and I don't want to put you on the spot, but I think it does intersect with our work next week on compliance enforcement. Just thinking about the municipal survey that you did. Do you want to give just a really brief update on where things stand with that and what you did? Sure. So I sent out a municipal survey with questions that went from, you know, do you intend to put this on your ballot at the next meeting day? Have you already voted on it? And then what types of, you know, resources that municipalities think they'll need in order to license and have establishments in their town. We have almost 50 respondents out of 246 incorporated towns that we sent it to. I'm hoping to get a few more before the survey closes on Saturday. And then the next step in that process is to pull together a group of respondents for a roundtable. And I'm hoping that some of the early work that happens on Monday of next week in market structure, I think in particular is going to talk about some local issues. We'll sort of inform that roundtable and give us something to present to that group of people and provide some sort of feedback on to just kind of get a sense of where municipalities are at and what their concerns are. Awesome. That's great. Yeah. And bouncing off of that, you know, and the compliance compliance and enforcement committee local local or model ordinances and fees is a part of that committee as well. Our subcommittee member with that expertise had family emergencies and was able to unable to join us next or last this week. But next week, at least from a model ordinance perspective, that should be on our radar as a subcommittee. We've asked NACV to to help comb through various other model ordinances in Dillon states around the country with a with a cannabis marketplace for for that subcommittee to happen. It's kind of a starting point. Yeah. Yeah. Great. Well, if there's nothing else on those two, Julie, would you mind giving us an update on the social equity subcommittee and then the public health subcommittee? Sure. If it's okay, I'll talk about public health first. A lot presented in this particular subcommittee this week, they reviewed standard symbols and warnings from other states and some samples for Vermont, one that was developed with ISO standards in mind. Next week, they'll look at those on products, like sample products to see so that the committee can conceptualize what those might look like and whether or not they stand out, they'll talk about the placement on the product, how large the font is and the type of font and that type of thing next week. So I think they're pretty close to making some decisions to send back to us. In that regard, they also talked about kind of connecting to the sustainability, the packaging issue that you brought up in the last meeting Kyle actually came up in public comment as well. So the the subcommittee moderators made a note of that to discuss ongoing in the future. So that's the and they also reviewed some some of the language that's used in other states for warnings and information that's given out at the POS systems. And then also talked about doing a one pager for establishment so that they understand what the marketing and advertising requirements are and have some sort of flyer that they can look at that's easy to understand. They talked about it in terms of a roadmap. So you froze up a little bit, I'm sorry about that. Sorry. Do you feel like do you feel like as a board member who's sitting in on those that the advisory committee members have sufficient, you know, evidence and the information they need to make an informed decision or recommendation for public health? Absolutely. I think they do. Yeah, they're looking at a variety of different samples. They're looking at language from other states. Yes. Great. And then for social equity, there is a lot of progress in this group as well. So they continue to review the language that was proposed in H414 as sort of the base for social equity applicant definition. They reviewed incarceration rates and traffic stop rates. They discussed the impact and harm of cannabis prohibition. And then ultimately began to draft NACB anyway began to draft a definition for social equity applicant that includes living in an opportunity zone as defined by the census. Someone who is black, indigenous or a person of color impacted by cannabis prohibition, meaning they themselves were impacted or a family member. And there was some discussion, I think still to be had about what qualifies as a family member. There was discussion about relationships and then in local parenthesis kind of relationships as well. And then they talked about lived in Vermont for one year as one of the one of the qualifications. And they also began a discussion about what types of documentation would be required as part of a social equity application. And I think there's more to be discussed with that. I think there's some thought that some documentation might be available that just simply might not. So for folks who have been in the foster care system, for example, they may or may not have access to very accurate documentation. And that's not a fault of Department of Children and families, they work with the information that they're given, right? So I think there's more discussion to be had on that. Their next steps, in addition to sort of finalizing that draft language for social equity applicant is to consider the fee initiatives and how to use the business development fund. So those are the next things that they're going to be discussing. So on those kind of criteria, it sounds to me like those are either or, or, you know, there's a list of criteria. Are you thinking about them being kind of weighted differently? Or is that part of the conversation? So I think just getting those points down was the last conversation. I unfortunately don't have the slide from yesterday, so I'm not clear on whether or not those are and or ors. And that's something I'll need to ask. Do you have? In the slide, it was an oral conjunction. Okay. So I'm sure that that could be up for discussion, but it was an oral conjunction on the slide. Yeah. Okay. Well, I would like to just hear that committee discuss whether certain criteria should be weighted more heavily than other, you know, and what that should look like. I just I don't know the answer to that. And I don't know what other states are doing, but that that is something I'd be curious to hear as we're kind of reviewing them and determining. But then again, you know, if we have unlimited licensure, I don't know if that's the case just to be clear, but it might not matter if they're weighted differently. Right. I'll be happy to bring that up. I agree. And is there any discussion? Has there been a kind of social equity cooperative model? At the very end of the meeting, Gina touched on that, but not enough for me to report back on it at this point. I think that's going to come up in a future conversation. Yeah, I believe delivery came up to curier license types for specifically allowing certain groups of folks early access to licensing types came up at the very end. This particular meeting was really focused on the definition. Yeah, it was exciting to see some a definition on on paper. It was very recognizing that that might be not be its final form, but yeah, it was I sat in on that committee. It was it was great. Yeah, that is very helpful to have something to dig into. Yeah, I just be curious to see how things move progress in that. Of course, you know, as we have heard as a board over and over again, you know, access to land access to capital are the biggest barriers for everyone. And mostly the folks that have been disproportionately impacted by prohibition. And so, you know, the delivery license type to me seems like an obvious path, low barriers to entry there. The cannabis cooperative social equity cooperative seems like a possibility. I know Massachusetts has a cooperative license type, but I don't think they've actually started any cooperatives yet. I think that's they have the ability and they have some applicants, but they haven't issued one. So, you know, maybe you can direct Gina to that model or talk to Jen. Jen Flanigan about it. Absolutely. All right. Anything else on social equity? Nope. All right. I'll give an update on the market structure. I'll start with market structure and pause and then I can move to medicinal. And so market structure has met twice since our last Friday meeting. On Monday, we really dug into the rough estimates for the total demand. And that demand is made up of kind of our in-state, our Vermont resident demand, our bordering state, the people that would kind of travel to Vermont to purchase from kind of our neighboring states, and then just tourist demand, the people that are coming here for other reasons that might purchase here. And essentially the demand was there kind of a rough back then napkin poundage was calculated during that meeting based upon the kind of 350 to 400 square feet of canopy. And it translated essentially into a little bit more than 55,000 pounds of cannabis annual. We did, when that calculation was made during our public comment, got some concerns from members of the public that this actually underestimated the demand. And that also that 400,000 square feet of canopy would actually produce in excess of 55,000 pounds. But we really that on that Monday meeting dug into with our consultants in the subcommittee, the questions that are specific to the cultivator licenses. And that really is, should we be setting tiers based on square feet of canopy, or should we be doing plant count or some sort of hybrid of both? Whether we should allow cultivators to apply for any tier of cultivation, or whether you have to start with a smaller tier and then graduate to larger tiers, whether we should have a provisional license. And this would be kind of we as a board would grant a cultivator a provisional or some sort of conditional license based upon basic information, like a business plan. And then once we've granted that that would enable a cultivator to go out and secure a location, start getting their local permits in order, their statewide permits in order, get their finances set up. And then they could then come back to return to the board with a kind of full application, and we could convert their provisional license into a full license and they would pay the kind of fee. You know, I think there's a lot of the subcommittee, I think all of them thought that this was a good idea. They saw the benefits at least. It really helps test what's being referred to, of course, is the entrepreneurial demands. You know, if we really want to rely on small and craft cultivators, are there going to be enough of them out there to meet this kind of 400,000 target of canopy? And it also would allow us to kind of get some of the stuff that takes a little bit longer on our end, the review of applications, the background checks, you know, we could get all that stuff in order. So it would be kind of a staggering of some of the more administrative processes around approval of licenses. And of course, it benefits small cultivators that, you know, might or small and craft cultivators that might, you know, not want to start paying rent at a location, for instance, if they're not sure that they're actually going to qualify for a license. So it might kind of help set that, you know, get people kind of the assurances that they need that, you know, that they will inevitably get a license if they can get their kind of ducks in a row, essentially. So I think there was, I don't know if there was a specific vote, I'd have to go back and check the minutes on whether or not to make this recommendation. But I think I saw at least, you know, the three subcommittee members interested in this. On Thursday, I think that was it on Monday, but I could be wrong. But on Thursday, though, the kind of moderators looked at some of the other types of licenses, the retailers, the product manufacturers, laboratory, wholesaler and integrated. And they were based on some of the public comments they've been receiving. They tried to, they're trying to get creative with some of the tiering of those. They didn't speak about specific, you know, canopy sizes or plan counts for kind of you know, the ones above small cultivator. But they did think about how, or they did think about how we could get creative with our tiering of retail licenses.  We see in Massachusetts and Colorado. But, you know, they were thinking, you know, we might want to also consider a different tier of, for instance, a nursery license that sells seeds and clones. They talked about potentially having a limited location retailer. And this is really kind of, you have a broader store, like a general store, and it has a section within that store that you know, would be kind of shielded from view of kind of youth. It would be a separate portion of a general store that could retail cannabis and cannabis products. They talked about a direct to consumer, you know, kind of the craft cultivator, small cultivator direct to consumer sales, potentially a farmer's market sale. And then they talked also about non store front delivery, either the kind of courier service, or there's a lot of different models on delivery. But, you know, they thought that this could be potentially shoehorned into a kind of creative retail license, a specific retail license. And of course, I think a few members brought up that this has an intersection with social equity as well. And then they also talked that there could be a tier of retail licensing that would be for special events, like a temporary sales permit for special events. With respect to these retail ideas, these tiers, there was it was a mixed bag, honestly, from our from our subcommittee members. I think that, you know, until you have these kind of enforcement and compliance issues worked out, you know, a lot of our subcommittee members have experience in the alcohol industry and alcohol regulations. And there are a lot to think about when you have kind of direct consumer sales or special event permits. And so well, I don't think that they actually took a formal vote on any of these issues. I think they did want to kind of hear more about what the security and the compliance and testing and enforcement committees are going to come back with before they wanted to really dig into whether or not we should have these. But that being said, we're required as a board to make recommendations on at the very least delivery and special event. And so and I think direct to consumer, I can't quite remember. So I kind of directed them to whether, you know, let's put the compliance and enforcement aside, what would this look like and what should the fee structure be for these things? Because I think, you know, unless our recommendation out of the gate is just no, don't do this. I think we need to it's incumbent upon us to kind of, you know, phase out for the legislature what this could look like and what the fees might be for them. Anyone should we pause there for a sec? Any questions? I think I think that all sounds great. I'm glad that folks are getting creative from a from a retail licensing tier perspective and trying to take, you know, what Vermont values and how Vermont functions and trying to come up with license types that kind of reflect that, especially for some more rural areas and rural participants in this marketplace. I should I should mention and sorry, I didn't when I was overviewing the compliance and enforcement committee. We had talked about elevating security as a priority because I heard that from the the market structure committee. I think I have, I don't know if farm gate sales or direct to consumer is is statutorily, you know, we have to give a statutory recommendation on it. But I'm interested in it for sure. But I want to see what, you know, the compliance and enforcement subcommittee would say about security and how that could be handled, you know, with with consumer protection in mind. But overall, it sounds great. I have a question about the limited location license or limited. That wouldn't just be limited in terms of the amount that sold, but it would also be potentially what sold, right? So it could be limited like topical products only versus consumable products. Is that right? Is that that concept? I think that's right. I don't I, you know, I think that the the concern that came in is whether if we have this license type, you know, whether whether this should be a subcategory of a retailer license, I think the concern that we heard was that why would anyone have a more traditional storefront, the brick and mortar if this was available? And I think that, you know, part of what the consultants are trying to do is ensure that there is economic viability to all of our license types. And so, you know, I think that they were thinking that there would be some sort of limited capacity on the storage and sales if there was going to be this kind of limited license retail option. But I think that this was really just they've heard a lot of folks and I don't know from who, but they've heard a lot of people in their outreach. I think we have to as well that, you know, we it's not going to be viable for a lot of people to kind of upfront the cost of putting a brick and mortar store, you know, in every town. So we got to get creative here. And this is one way to kind of reduce the upfront costs. But I don't know if they've really dug into the to your question, Julie. I guess when I'm thinking about it, I think about the traditional brick and mortar cannabis only stores tend to have a lot of product knowledge and the people in those stores tend to have a lot of product knowledge. So maybe on the compliance and enforcement side, if there is a limited capacity in a general store, for example, how do we then also communicate that same level of product knowledge to somebody who's maybe purchasing for the first time. It's a good point. Yeah. Yeah, again, another intersection between the multiple committees. And so that's why I think it's going to be helpful for us to really tie all this together when we have our recommendations with the full advisory committee and amongst ourselves. So there is they also discussed having tiers of licenses for the product manufacturer. You know, the three that they talked about is having one product manufacturer that does solve them basic extraction, which has potentially a more costly slash public public safety impacts, one separate product manufacturer that doesn't have that there might be a kind of less expensive tier. And then one specific to small cultivators that would be somewhat limited for like pre-rolls, they said. But I guess one of the big takeaways and this is through no fault of our consultants, because we haven't really fully fleshed out our own kind of operating expenses, but there's no recommendation at this time around what this basic fee structure is going to look like. And I think that there's going to have to be on that a menu of options for us because it really depends on a lot of other collateral or corollary issues to be tied into it. So any any questions on that before I move to medical? I have a question on the production side. Is there any discussion about like cottage level licenses in that structure that you were just talking about? Something that people can do on a limited basis, perhaps out of their homes. On the product manufacturing side. I all I heard was the kind of small cultivator license that would be really somewhat limited. I think that we could get creative there with that. So any direction for this committee that you want to hear? Do you want them just to keep chugging along? I like I like how you phrased things earlier. I know that that committee wants to hear more about compliance, enforcement, security, some of the maybe even some of the issues that Julie just raised. But to just go ahead and try and make a recommendation recognized from a license type fee perspective. And just, you know, making sure that we have that box checked while we decide if it's something that can work in practicality from a security perspective, I think is a smart approach. Yeah, that's great. Oh, yeah. This is where, of course, we need to like try and tie these two committees together. And there there's not a ton of overlap on, you know, amongst the subcommittee members on this, but yeah, I think our consultants can do the work of kind of like bridging these two committees and the work they're doing. On the medical side, I think this committee seems to be somewhat wrapping up. I mean, they've they've talked a lot about purchase caps. I think currently it's two ounces within a 30 day window. I think that, you know, there's some division about essentially, listen, in the adult rec side, I don't think we would have that purchase cap. Maybe we would. But I think, you know, that's kind of a low purchase cap, especially when you're talking about people's medicine. And so increasing that, you know, I think there is a kind of a question about whether you set it at a kind of 90 day supply, whether you set it just increase it to like three ounces or whether you tie it to the rec market. You know, my advice to them or it was kind of tie it to the rec market, but also think about what's legal and illegal to possess on your person at any one time, because I don't want any patients to have to be detained and try and prove that they're a medical patient just because they have, you know, three ounces or four ounces of cannabis on their person at a given time. So I just to me, I suggested you should either set it to whatever we do for the adult use or set it to what the kind of the two ounce limit, the personal possession limit. And again, that personal possession limit does not include what you have stored at your house in a locked container. It's really just what you have on your person or in your vehicle. So, you know, anyway, we'll see what they come back with. I think they landed on. I know Meg Deely wanted to tie it to the adult use rec program, whatever we decide there. And I think Jim Romanoff, the medical for syndrome relief oversight wanted to do increased to three ounces. But anyway, they talked about increasing plant counts for for patients and caregivers. I think they landed on six mature 12 immature they talked about eliminating the three month relationship that you need to have with your health care provider before that person can kind of qualify you for the program. They talked about expanding the qualifying conditions to whatever a physician feels like is appropriate. That to me has a little bit of a complication just in the way that the program is set up, because right now physicians aren't actually recommending cannabis. They are a testing that you have one of the qualifying conditions. So, you know, if you have PTSD that's they are verifying that you have PTSD or they're verifying that you have terminal cancer. So they're not actually if you change it to whatever a doctor thinks is OK. They're actually putting a doctor in a position that they're not currently in. But I think there's certainly ways around that. But anyway, that's the kind of the conversation there. There's a lot of discussion around around reciprocity with other states. So if you have a medical card in Maine, can currently you're not allowed to come to Vermont and purchase. And there's a question about the patients having to designate a dispensary. And to me, I thought these are kind of easy questions. Just allow reciprocity. It kind of the dispensaries have bank accounts product. And so from a supply side, there might be concerns around reciprocity and not having designated dispensaries. But I think once we have an adult use system that I think those probably can go by the wayside. Well, but so there is a lot of rest evers and so caregiver is somewhat of an umbrella term in the medical worlds. And it means it's it's people who can be designated growers and kind of go to the dispensary on a patient's behalf and purchase and administer. And so when we say so a lot of discussion was involved with the do we expand the patient to caregiver ratio, which is currently set at one to one. And the two advisory committee members said absolutely expand it for the administering and purchasing of purchasing the administering and purchasing the kind of like caregiving aspects, but don't expand it for the growing the designated growing. So I know that we've had very strong recommendations from some of the other folks in the medical world about expanding that designated grower. You know, my thinking on this is I actually don't have a problem with that, but I really do think that we start to run into potential quality control issues if you had, you know, say a one to five ratio for the designated grower. You know, at what point do we need some sort of third party testing, you know, because currently what it is, I think there's four hundred or give or take caregivers and the vast, vast majority of them are nuclear family members or very close friends. You know, there's a there's a relationship there. But if you when you start to kind of attenuate that relationship, do we as a board need to start thinking about quality control measures for the patients for the benefit of the patients? So anyway, there is I don't know if they came up with a final recommendation on either of those issues, but I do think that that whatever they decide, we need to really kind of like dig into all of those issues. And then, you know, for me, I just I don't think the elephant in the room for me on this, which has not been fully kind of confronted is that when you have kind of a profit motive and you have a huge market for adult use and you have a very limited market with some very specialized products that are prohibited in the adult use world that are expensive to create, are the patients going to get the kind of attention that they deserve? And so that to me, what is the essentially comes down to what's the long term viability of the medical program? And of course, the easy thing to do would be allow patients to just purchase tax free at the adult use markets. But that doesn't that doesn't really address the concern that if someone needs a specialized product that's prohibited in the adult use market, how are they going to get that how they get that product? So I think that's the question that I need answered and think about most often with this subcommittee, but you know, I think they did have a lot of work on the kind of the specific recommendations they needed to make. So anyway, that's that's where that committee is at. I think from what I understood, they might have kind of some final recommendations for the board or for the full advisory committee relatively soon. And and maybe they can just turn to that secondary question. Any thoughts on that one? I'm not sure I caught all of what you said. So I apologize that you cut in and out a little bit. But on the caregiver piece, I am concerned about the the subcommittee's conversation there. I think one to one is not necessarily realistic. And, you know, if if you're caring for one parent and then the other parent, especially if we're expanding the qualifications and the conditions that are allowed, it's possible that one person might have two parents or two people in their family in that nuclear family that you're talking about that are providing for or growing. I hear the concern about at a certain point, there needs to be some sort of testing and quality control. But I think one to one is is not realistic. And I'd like to hear the subcommittee talk about that more. OK, I agree with I agree with Julie. I think we again, I agree with you to Mr. Chairperson, if we go too high on plan count, what does that mean from a from a quality or excuse me from a from a patient count? What does that mean from a quality control perspective? And maybe that's a conversation we have down the road. But I would like to see the subcommittee consider the public comments that we hear from caregivers and growers and dig in a little bit more into what's the real concern about raising that or elevating that patient count, because it's something that I'm interested in. Great. Yeah, I'm with you on that. Just because I'm cutting in and out, I feel like Julie, would you mind managing the public comment period? Yeah. I have one other thing on the medical as well. I think among employers, what types of employee employment protections are available to people who who have medical cannabis cards is confusing? And if there's a recommendation from the subcommittee on that, as it relates to, you know, reasonable accommodations or anything like that, I think that would be helpful if if we're able to provide guidance to employers. Did we lose him altogether? Did we lose you pepper? Why don't we move on to the next the next item on our agenda then, which is comment from the executive director and general counsel. I really just have one thing to add to the conversation, which is a reminder that we have about most of our subcommittees are meeting about four more times before the date that our report is due. So we've been obviously doing a whole lot of work up until this point, but I would just encourage members of the public to tune in, especially these next couple of weeks, because decision points are going to be happening probably at every meeting. So now is the time to go on to our website and lodge your public comment or come to the meeting, the subcommittee meetings, if you can, or make a public comment at the board meeting, because things are going to start happening even more quickly over the next couple of weeks. All I have. I don't have anything for there to add. All right. Public comment. So I think the way that we've been doing this is if you want to raise your virtual hand, we'll take the folks who have their virtual hand raised first and then folks on the phone and then folks in the room. Is that how we've been doing it? Sure. We've been doing folks in the room first. So I apologize. Would you like to make a public comment? Sure. My name is Nicholas Shoreman. I'm with Vermont Normal. I'm going to talk about something that I mean, I got here a little late for that. It wasn't discussed while I was here talking about caps on THC for for medical products, both concentrates and for flower. So I just came back from a two week long trip in California and in Oregon and in Nevada, three people in very robust markets. And while I was there, I visited dispensaries in each of the states. And I really spoke to the people who were selling the candidates who are at the forefront of this market, you know, not those fulfilling the orders, but really those who are talking to people at their at the moment of decision when they're right. So how are they deciding what to buy and what is kind of the deciding factor? And it always comes down to THC percentage, whether or not that's a good thing for the market.  So really the one factor that people are buying off of is that one number, which, you know, you can argue is hard to compare across multiple products, but that is what people are buying on. So I think that's something important for the board to consider that these caps are very restrictive for a lot of these craft growers because it sets a it sets a ceiling for the level of the level of expression that is able to be, you know, expressed by these growers really and like to be able to separate the really good ones from the mid-tier ones. And it was fascinating to see that that was the number one factor across all board, no matter where I went, no matter who I talked to, it doesn't matter color, what it looks like, what it tastes like, what it smells like, it always comes down to the THC percentage and the price, but mostly it's that number of 30%, whether it's 17%, 31%, and I think that's something to just keep in mind and consider moving forward. Thank you. Thank you. And then I can't see who to pass. Dave Silverman is first. Dave Silverman. Thanks. I'm really excited by the by the progress being made by the, especially the Market Structure Committee. The discussion on retail tiers is great stuff and really glad to hear it. A word of caution on the discussion on medical. You know, you discussed sort of, James discussed three different things that I think in combination could create an unintended consequence. So if you eliminate the three month relationship and expand the qualified conditions to anything, which are things that I support and you allow dispensary, allow patients to buy tax-free at adult use shops, you couldn't inadvertently create a situation like what we saw early in the pandemic in Massachusetts when mass closed its rec shops, but kept open its medical dispensaries. And so a bunch of consumers who are adult use consumers are not really medical patients went and got medical cards so that they could continue to buy during the shutdown. But as a result they now the rec shops open back up, now they're buying tax-free. And I don't think that's the kind of subsidy that anybody really intends here, nor do I think it's going to be particularly necessary if in Vermont what happens is the same as happened as has happened everywhere else where prices begin to come down once the regulated market is fully matured. So I'm just wanting you to keep an eye on that and the potential for tax revenues to be too low if all of those things come together. And when I say too low I mean just that you're unnecessarily subsidizing the purchase by some people versus others as to just whether or not they were able to go and find somebody to write them a recommendation without a true relationship. So just to keep that in mind thank you. Thank you. Thank you Dave. Tito. Hello today I would just like to talk about the medical program. I believe that the most intimate and effective relationship a patient can have is with their personal caregiving grower. And on the marijuana for symptom relief oversight committee meetings it feels as though that group would like to end the relationship that caregiving growers have with patients. And it's also my belief that interpreting the law that's saying that the marijuana for symptom relief oversight committee has to have three seats dedicated to caregivers. It feels as though they're trying to change the definition of caregiver just so that those three seats are not filled by caregiving growers. And so I'd like to reach out to anybody listening to this right now. If you are a caregiver please please take an hour and a half out of your day on Wednesday and attend this meeting. The marijuana for symptom relief oversight committee it's at one o'clock on Wednesday. They do make it hard sometimes by not posting the agenda or links. If you have trouble finding it or getting on there please reach out to me or Jeff from Vermont growers or Jesse Lynn and we can help you get on there. And please have your voice heard and let's keep the medical program alive because it feels like it's on the doorstep of being over. So that's it and thank you all. Thank you. Catherine Breyers. Hi thank you guys. I'm a regional planner with the Bennington County Regional Commission down in Southwest Vermont. And board member Halberg mentioned that there's a municipal survey out and I wasn't sure if I understood that is that going to be closing this Saturday or next Saturday? It closes this Saturday. And I can connect with you about who in Brattleboro has completed it. Okay yeah because I was just wondering sometimes when we email our member towns we can help give a boost to responses but but with it closing tomorrow that that wouldn't be that helpful. But if there's a way you know if it weren't closing until next week we could potentially try and push it out if there's a low response rate in Bennington County. Okay thank you. Thanks. Any other public comment? There's two phones and it's star six to unmute. Yes it's star six on mute. If there's folks on the phone that want to make a public comment it's star six to unmute. Okay. It looks like pepper's back. Yeah I apologize for that. I hope I wasn't too jumbled. But if there's no other public comment I would entertain a motion to adjourn. I'll move to adjourn. Second. All in favor? Aye. Aye. Okay so we're back at it on Monday with subcommittees. Thank you.


So we're at the grill at the Knights of Columbus swim club And we're talking to Chris right right and Chris. Um, what have you done with your summer? I've worked here. I've played lacrosse. I've played basketball Pungent with friends traveled a little bit. Oh, where'd you travel to? Cape Cod? Okay So did you save a lot of money from working here? Yeah, actually I have What are you going to do with that money? Probably save it for college Maybe buy a moped or a bike or something.  And how old are you Chris? I'm 17 about to be 18. Wow, okay So you're going to be thinking about college Yeah, do you have anything planned for the last couple weeks? I'm actually going to the beach Next week and then unfortunately back to school. What's your favorite thing about this summer? Um Not being in school The ability to do whatever I want and actually over the summer I got my EMT certification Wow, well, that's a big deal. Can you tell us what EMT stands for? Emergency medical technician so the people that ride in the ambulance Cool. Yeah. Well, good luck to you. That's going to be great. Thank you


My name is Mary Lim Thiam Swan and currently I am a judge of the Federal Court, which is the apex court of this country, Malaysia. My first degree is a Bachelor of Laws Honours from the United Kingdom. I returned to Malaysia to practice at the Attorney General's Chambers and after about ten, eleven years I felt the time had come for me to recharge, to rejuvenate and I looked around and I thought it was time for me to go back to school. Australia was always very different in its approach and so I deliberately chose Australia because of its difference. But I had to pick somewhere which was at the same time convenient for me because by that time I had started a family so I picked Perth because of its location and its proximity to Malaysia, same hours as here in Malaysia. So if I pick up the phone and speak to my husband I don't have to apologise for having woken him up or anything like that. The reason why I chose Australia also is because the master's degree then was not exam-based. Unlike how it was run over in the UK and other jurisdictions, it was very thought-based. You have to write theses, you have to write papers, you have to do your own research. And that was an extension of what I had been doing in practice. I'm so thankful that after ten, eleven years in the Aegis Chambers I actually had the presence of mind to make that, now when I look back, very much a correct decision. Similarities between Malaysia and Australia, it is so cosmopolitan like Malaysia. The culture is colourful, diverse, vibrant and it's so attractive. I went there because it was different and I came back finding myself different. It impacted on me and gave me confidence in my work. I believe I have a lot more maturity and it endures until today.  The sun, the sea, the trees and the eucalyptus trees, the gum trees. The smell now is so nostalgic for me. Australia came through for me and I'm very thankful. I like to encourage my children, your children to take that same journey. Do it. Go to Australia. If the opportunity presents itself, do it. I will grab it.


Hey everybody welcome back here. We are on the stage to my right. You're left. It's not get confused Jared. Psych Lawrence Jared How's it going? I'm pretty good. Cool. Cool. Now you gave a speech on this very stage that we're sitting on this morning What if you could give everyone a preview out there? What did you touch on? Basically, I got new the specifics of sexual escalation. I covered the intergame mindsets and three top Intergame problems guys have I got into the outer game issues exactly what to do step by step to go from Needing it to go comfortable. They're touching to get it to chasing your touch and then I gave some fun little tactics Everybody to use here tonight at home one of my students actually use them and actually pull last night So I'm very proud of him if you're watching yeah No names, no names, but is that is that is that kind of how you see like it starts inside your brain And then you're able to act that out or is it the inverse how I basically came up with everything a lot of trauma Okay, basically it started with me to try anything and then massively failing and then looking back at it and go Yeah, should not have done that let me try doing this thing up right and then see patterns I saw a lot of patterns with each one of my sets and like What did I do that worked or what I do that just blew up in my face a lot and when to use them And it's just a lot of trial and error great great now so many speakers this year Yeah, so many wide range of topics. We're glad you got to speak on that What do you like would you say the most about the 21 convention? All right, the thing I really love the most about the 21 convention is it's not so much a Pick up the type convention is the other stuff. It's out there You get like it really is more like men's self-improvement And to see guys you topics that are just great for men all around from fitness to business to get it better with women It's just all comprehensive. It's all Anything a guy would want to learn that there is life is found here and that's what I like about this convention more than the other one Yeah, it sounds great. Would you say that you endorse the convention? Yeah, I wouldn't say Absolutely towards the convention. Yes. All right, cool. Well, Jared. Thanks for chatting with us four times speaker here I'm keep coming forward time speakers amazing. All right, and we're gonna have to put you in the rafters and retire you I want to haul a fade thing We'll give you a scroll and a diploma. I'm Jared psych Lawrence. I'm Peter Murphy year you Next time cheers


Good morning. Good afternoon. Good evening. Welcome to another edition of ask and open shift admin I am Chris short executive producer of open shift TV I'm joined by the the main, you know host of the show Andrew Sullivan But we're also having our special guest the one teammate. I know that Bragg's about his knowledge of DNS Christian Hernandez. How are you today? He is the DNS man yeah, it's Well, you couldn't have an episode about DNS without without you. I know yeah without me jumping on yeah, I'd be I'd be almost offended if you did right I am happy that you are here today with us in person instead of just participating.  Just not not actually in person So, hello, welcome everybody to the ask an open shift administrator office hour stream So this is one of the office hour series of streams that we have here on open shift TV And the goal here is really for you all for our audience the people who are listening and watching in To ask us anything at any time about whatever it is that is at the top of your mind So please don't hesitate to do that You know we more than welcome those questions again, whether or not it relates to whatever we happen to be talking about today We've already got a question Great. That's that's not a question, but like Interested in seeing the episode. So yeah, sorry speaking of which I didn't even I didn't look at the staged YouTube channels to see if anybody like last week remember we had a question four days early So yes, we did not have that I don't think I Haven't seen it at least I mean, I always have the reaching chat open. So I feel like I would have seen that Had it have by kind of deal, right? So, all right. Well, thank you everybody. Welcome. And again, please don't hesitate to ask those questions at any point in time So today's topic as or if you didn't see any of the social media or other things leading into today We're gonna be talking primarily about DNS and also a little bit about cluster networking but primarily about DNS and the genesis for this is Christian and I spend What what feels like a substantial amount of time? I mean relatively a lot of time. Yeah, yeah talking about DNS and DHCP and DHCP we we think is starting to get out there rate the requirements the configuration how it affects things and all that but DNS is still largely a black box to folks You know, there's folks. Yeah. Yeah, there's some of us might have read the RFC. Yeah, well Understanding how it works and knowing how it works in OpenShift and Kubernetes is not the same thing Exactly. And that's what we often see as a problem. So today we wanted to kind of look at, you know maybe demonstrate a little and and explore DNS inside of OpenShift and Keep in mind that a lot of this will also apply to Kubernetes as a whole So it doesn't necessarily Matter if you're using OpenShift or not Hopefully we'll be able to answer some of those questions. DNS is a universal standard Yep, and it's it's funny because you know, I was an infrastructure administrator for a long long time and It was it was always the networks fault until virtualization came along and then it was storage as fault, right? And now that we've kind of worked past all of those right most folks are using either hybrid or all flash storage now Suddenly it's back to being the networks fault and a lot of times it's DNS. Yeah, great. Most of the time is DNS I have a nice haiku. I could post you. I was gonna say where's the haiku about that? By now everyone knows that DNS haiku It should be it's posted on people's walls, I would imagine. Yeah, I Very seriously yeah Yeah, I very seriously figured out how to put nor was looking into how to put up one of those little Sidebar things like the news that just had an image of that haiku Oh, nice. Yeah, I think There's there's some software that that does that Oh, yeah, it's like a virtual camera and you can post things on it. Yeah, there's there's a few of them So before we get started before I hand over to you Christian and start asking you a lot of dumb questions There was a few things that I wanted to talk about So for those who are regular watchers regular Attendees of our stream, you know that I like to cover a few things that I have seen either internally or externally that I believe are Relevance and also important things that maybe you haven't heard yet. So Let me find the right tab here Just always these days always the tabs I've actually had very few open today. I just have a lot of windows All right, that's the one I'm looking for So first and foremost a bit of shameless self-promotion We have been doing a weekly blog post after each one of our shows they come out on Fridays So this one happens to be from last week if you miss anything from last week Not only do we have a link to the stream itself But we also go through and try and put as many of the questions as many of the things that we can find in There so if you want to write quickly skim because I know, you know One of the big things about doing a video stream is there is no Transcript there's no way to do like a control f and find a topic of interest So I try to go through I try to identify the questions and pull them out and then link to where they're answered inside of the Inside of the video. So nice. These are better than our release notes. It's That's a lot of work, yeah Come on this there we go. Yeah, that things little window gets in my way. And I yeah Yeah, I'm always trying to figure out how to move it and I never just needs to like Like I have three monitors and it needs to just like go down right, you know, just get out. Yeah. Yeah So I will paste a link to that into the chat. The second thing I wanted to talk about is This is a bug that's been in place for a while. I had actually you can see here It was first reported back in September late September I actually had not heard or encountered this I don't like to bring up bugs very frequently because you know, they happen all the time So usually until it's something that I feel is coming up or more I guess serious or coming up more frequently than any generic old bug. I tend not to bring them up. So This one Happens because I saw no less than three customers this week that encountered the scenario where Their vSphere IPI node had gone offline So for whatever reason the worker node was offline, right? Maybe it was turned off maybe the physical node had had you know, powered off failed for whatever reason some yeah, and the IPI process is effectively reaped that node, right? It deleted the node because it was going to provision a new one to Recapture in that that capacity. Yeah, the problem is that virtual node the VM had some pvcs attached Yeah, and so the pod or the node was running it had some pods those pods were using pvcs The node failed for whatever reason. So it still had those dmdk's attached to it. Okay, and When IPI came along and destroyed the node so too did the pvcs Yeah, so that's that's not good, you know data loss is never a good day on at any point in time ever So a couple of things to be aware of here. So first and I'm scrolling down. Yes, there's a number of missing comments in here I apologize. I'm not logged into bugzilla right now because there's customer information associated with these So I don't want to show that that information to the worlds, but there is a fix for this I believe it was just committed yesterday evening or maybe early this morning Or PR just rather so it should be fixed in a coming release right hopefully sooner rather than later But for now just be aware of you might want to set that if you're doing auto scaling if you're doing Or using IPI in the auto scaler together to reclaim those nodes or to otherwise restore those nodes You may want to configure the auto scaler with some sort of delay So that way it will have at least five minutes So that way in the event of node failure the Kubernetes scheduler will say hey I need to I need to you know, clean these up and I need to move them Yeah, give the the scheduler enough time for those for the workloads on the on the node to move to another node Before it it reaps that that BM. Yeah. Yeah, exactly So another one and man, I always feel I feel bad bringing up two bugs That makes it seem like a bad week even though, you know, it's well, you know what you can actually feel a whole show of just like looking over But how depressing would that be So so this one I've seen come up a number of times Internally especially I've seen it come up a few times externally as well essentially we have uncovered a Inconsistence bug for Coros virtual machines deployed to vSphere using OpenShift SDN So we think this has something to do with the VMX net drivers that are in the Rel 8.3 kernel the one used by Coros 4.7 and it's effectively this manifests itself as some packet loss So if you're if you have upgraded to 4.7 or if you've deployed a new 4.7 cluster on vSphere And you're starting to see, you know, one two three five percent packet loss This could be the result of that. There is a workaround. I don't know similar to the other one I'm not logged in because there's some customer information here. There is a work around It doesn't look like it's in the public comments. If anybody needs that just let me know Andrew dot Sullivan at redhead.com. I'll share that with you effectively. It's setting or turning off the VX LAN offload for the Network adapters inside of the virtual machine. So it's just a couple of sys controls that you execute You can test it by OCD bugging or you can use machine config to set them in a reoccurring manner Okay, getting out of the Things are broken.  Hey, we love failure. We love, you know Helping folks, you know Fix their problems on the show channel I try to, you know, again, it's it's uh I try to preempt or try to make you aware of things so that way You don't accidentally stumble into a bad a bad situation. Yeah Um, so the next one is a good one, uh, you know, I talked about sizing in a couple of different ways With three or four different aspects, whether it's sizing the cluster sizing the nodes sizing storage Right all of those things across a handful of shows here on the open shift admin hour ask an open shift admin hour So recently you see march 16th Simon the lord published a blog post here that walks through application sizing principles So a really good read, you know, share this one with your your application developer application admin peers that kind of walks Through all of the different relevant settings different capabilities things like that to help you size those clusters Both how the application is allocated resources and how those resources are then reflected inside of the cluster And when I do the show notes for this one, I'll uh, I'll try and link those other episodes inside of there If I remember to make a note to myself Yeah, also your episode on, um resource limits is really good also to link there because that has um, oh yeah going from You know going from a vm to to a container is very different because the the developer is used to getting 8 cpu 16 gigs of ram because of like scale right because oh, I need to be able to now that's different now You're essentially constraining them. What do you mean? I want to get you know, half a cpu cycle it's like well, it doesn't really work that way so um that that episode your episode of uh of uh of resources resources limits and ranges is good to um, I got I got the link right here. Yeah recommend those users watch that as well Drop it in chat here for everybody. There you go. All right. So the last topic I wanted to talk about before we get started here Is one and we were joking before the show started that uh, I'm just going to kick this hornet's nest um, because it's it's a bit of a sensitive topic um, I think internally and I know that there are some people who are passionate about it externally as well And that is reference architectures Oh, you said about a word I know I I'm sure if any of our our marketing peers are are watching right now They all just had like their stomach not up because they're afraid of what I was just going to say You know like when you said it prior to the show I was like oh boy So reference architectures, um for a little bit of background here So anybody who has been keeping up with open shift for the last roughly two to three years You'll remember that we used to publish reference architectures more or less every version or every two versions for various I see somebody asked for the sizing link. It's uh, it's the second one up the sizing applications right above the last chat Yeah, scroll it scroll up a little bit that came from the city tube. I wonder if I'll post it over there just in case Okay So we used to publish reference architectures red hat published reference architectures regularly And we stopped doing that for a couple of different reasons Uh, so the big one was quite simply resources Right constantly trying to and having you know devoting not just the people resources But the physical resources and all of the things associated with going through and generating those documents Uh was you know quite frankly Hard And it really was and invariably it always led to you know, if we used one partners You know a set of resources instead of another partners, right? Sometimes there would be you know hurt feelings and stuff like that Um, so it it it became A bit overwhelming and in number of different ways So the decision was made rather than creating reference architectures Which the perspective was and I would be very welcome to anybody's Anybody's additional thoughts or perspective on this But the perspective was that most folks read a reference architecture not to literally implement it as written but rather to Glean right rather to determine what information is important what information is relevant What decision points do I need to know to deploy an open shift cluster in my infrastructure? so it was really um For lack of a better term right it was it was a guide Right. It was a a consolidated source of documentation for here's one way of producing an open shift cluster So the docs team took upon themselves to take all of that Knowledge and put it directly into the documents So if we look at Can I type here? No, my browser is beach balling on me Oh, oh it great out everything. Oh it great out. There we go. There we go. All right back So if we look at the documentation here We and we scroll down to for example this scalability and performance section You can see that and this section has grown with each each release They are doing exactly that they're taking as much of that information as they have as they can get and putting it directly into the documents right recommended host practices is a great one and Allowing you Our audience right the people who want to ingest those reference architectures To really understand the rationale behind those decisions right not just oh, I see that You know in the reference architecture. They said, you know, this value to x well Why why was it set to x sometimes we explain that sometimes we didn't so through the documentation The goal is to provide all of that background all of that information So you can make the right decision for your infrastructure So with that in mind, however, we didn't completely abandon reference architectures right it seems that way and Some of you may be thinking well, I have seen some reference architectures come out and you are absolutely correct We now rely on our partner ecosystem for those So i'm gonna post this blog post into the chat here So this one which is coming up on a year old now So I assume Dave will probably update it and I've known Dave for a few years now. He works over on our Partner team right they do a lot of the reviews of these types of things and the goal is Him and his team work with for example, if we scroll down here, right? Cisco and delimc and hitachi and hpe and so on and so forth everybody and those partners Create the reference architecture, which is then reviewed and contributed to by red hat So if we were to look at and i'm gonna pull up one of these, right? This is the hua packard enterprise the hpe site for open shift reference architectures. I'm also gonna paste this in here So if you are using hpe hardware and let's say you want to deploy open shift onto your dl Whatever servers right dl 380. I don't even know what the current gen is So we can click on this 4.6 gl And dl, excuse me and it walks through all of the different components Of their reference architecture. This is all created by our partners. Yep. So they still exist They're still there. They're just not created not published by red hat red hat. Yeah, it's it's at the vendor that you're trying to Put open shift on and and that intuitively that intuitively makes sense because being a software vendor red hat like we We one really can't recommend hardware Because there's just like a ton of hardware that we're supported on like we Like andrew was saying we're we're just we don't have enough people to go out and buy every single hardware And you know vet everything every possible combination. So we we lean heavily on our um on our providers like Like hpe and del del emc to do these right and we can review them that seems to be a better Better use of our time rather than trying to build a team with an impossible task Yeah, exactly and together we are stronger, right? So yes, exactly I just thought of that that simpson's episode. Yeah, I was thinking of the exact same thing The one where they they mimicked lord of the flies. Yes, exactly. Exactly. That's So I happened to click on hpe over here You know you see net app and sysco have one for flex pod net app has one for their you know net app hci There's there's you know, literally dozens of partners that do these Now you will see one exception to this and that is open stack Because red hatch is you know red hat open stack We create a reference architecture and actually it's one of the uh or two of the folks on our peer team the field product Where they field product managers? He'll yeah f f pms. Yes. Yeah, so yeah, august Simon ellie resox and ham and company, right? So they they create that open shift on open stack reference architecture and that's because it's a whole, you know red hat stack there Um, just like if you were to you know similarly and I think I've said this before C&I plugins csi plugins that come from partners. We don't document those inside of the open shift documentation We rely on the partners for those The exceptions being if it's you know red hat virtualization or red hat open stack platform Those are documented by red hat because it's a holy red hat solution And that's that's all I've got these these other tabs are for uh our topic at hand. So I will I will now stop pontificating about things unrelated to dns and uh Essentially, I'm I'm gonna start off. I'm I'm gonna play the role that I was born to play of dumb guy. Um, and Wait, that's my job We can share So so christian, um, you and I started this conversation last week or maybe the week before And really it came out of some confusion around how does dns resolution happen for open shift? And it turns out that this is a little more complex than you might expect So can you elaborate on that and talk about you know dns in open shifts and the difference between node-based dns and pod-based dns? Yeah, so, um So where do I start right? Um, you know what I mean with with uh With with uh with with dns. So I um Originally I was gonna start like in the beginning. What's dns, but I imagine that At at some level since this is ask an admin. Um, there's some fundamental understanding of what dns is, right? So dns and Kind of sort of what it Um, how it works from a high level.  So um So dns in open shift is Um, kind of a layered Design right so originally dns in kubernetes. So, uh, I think I'm just gonna start there dns in kubernetes And what it's used for then I'll work my way up So dns is kubernetes is used for service discovery Right, so that's this is um It turns out dnn that that problem of service discovery was solved a long time ago with dns Right, there's been a lot of software that's tried to do service discovery You know people writing service discovery and in the end it just turns out like dns was the um So what was the answer pause there for a moment. What do you mean by service discovery? so by by service discovery meaning that As an application or as a service inside of open shift kubernetes in general It can reference service by a name for example, I have a front end web give a Stupidly dumb. Yeah to two tier application, right? I have you know, no j s front end and some sort of database back in I want to be able to just say database in my application and it should just Work, right? I shouldn't be as a developer or even as like an admin or as an end user I shouldn't have to care Or memorize ip addresses or what the ip addresses are or how to connect to that So it should be as simple as just saying my sql, right or database or whatever name you decide to deploy your database as so That's the idea of service discovery. I can reference something by name and it automatic automatically knows the ip address so And essentially, you know Funnily enough dns is a perfect software to do that Because that's just what it's been doing there's And that's what originally It was the original intent right for kubernetes service service discovery so in kubernetes, that's kind of like the the bare you know The the the kernel the the core of the the use of dns inside of openshift. Yeah, so pods are deployed that represent some micro service right front ends back ends database whatever that happens to be And then they have quite literally a kubernetes service in front of it that says point to these pods And it it discovers those ip's using this for front facing right external facing external being still within the cluster, but dns name so that Other components can say connect me to front end and the service translates that into The set of pods the set of ip's for the pods that represent that front end microservice Correct. Yeah, so it you can um You know just reference things inside the cluster in general By a name, right? So it's essentially you can even connect to other pods You know in by name, right? So everything is managed and everything has a name And that's managed by The internal dns server in kubernetes, right? So before that was based on cube dns. They've actually upgraded A while back to core dns. Yes, right, which is originally it was sky dns Or sky dns. Yeah, and it was yeah dns. Yeah, I remember compiling sky dns And the very first time I deployed kubernetes in like the point eight days Yeah, yeah, exactly. It was like sky dns It was it was also bare bare bare bones dead dead simple, right? And so I like, um, let me Where am I here? I was gonna share my screen a little bit and kind of go over the docs A little bit so that's I can only find there we go Share screen. Mm-hmm. Make sure it's the right screen wherever you threw it. Now you need it Yeah, wherever. Yeah, exactly. Now I need this little there we go. Can I oh, I can move it I'll move it over here. I'll let that there you go way way over here on the right side way out of the way Yeah, so The dns, right, so I will Um, what's what's funny about searching for dns? On the open shit docs is that it's like not the first one There's the api for networking, right? So, um, it's under networking. That's that's how I remember how to get to it And then there is understanding the dns operator. So, um, open shift deploys Core dns be an operator, right and so Let me make this a little bigger Because there we go I can make it bigger if you like. Is that too big? No. Okay. 110 seems good. Um So we can I you can actually need a bigger. Let us know, please. Yeah, just let us know, right? Yeah. Um I think you can essentially copy pasta this Let me Make sure I'm at the right cluster Be sear. Okay. I upgraded this, um Last night, okay, so it looks like the upgrade went. Okay, not that it shouldn't but you never know. Um It's very reassuring of you. Yeah, right, right. Um, thanks. So if you go, yeah, exactly See, I say as a dns. I didn't say open shift. Um, so So, uh, this is um, it shows the dns operator and Just like all things in open shift, it's controlled by an operator. So we have a dns operator here And uh, you can see the version here And that's all good. So, um, what's really cool about this here when you describe here It'll give you information about your cluster and so, um The important information is this, uh, this information here. So the dns service for Um, for kubernetes runs on the uh, the service address, right? So the service address Um From a high level I guess now I could I could play dummy to you, andrew. Yeah is an overlay network For the most part. It's an overlay network. Um, that sits on top of your regular network, right? A software defined network It's possible But this this will get I don't want to rabbit hole too much But it's possible to have kubernetes running without an sdn If you ever did uh, kelsey's high towers, um Kubernetes the hard way he shows you how to do that. Um, it is hard It is hard because anytime you add a node you have to add the routing table to all nodes and It just makes sense just to use the software defined network. So, um, this is an overlay network So when you see this this is my service overlay network and my cluster domain is cluster dot local Which is the default so cluster dot local is the domain For kubernetes the internal domain for kubernetes. So let's let's take a step back for a moment. So The core dns service inside of open shift is for pod resolution or more specifically service name resolution for pods Correct service name resolution for pods. So this cluster dot local is meant to be only for internal resolution and is has no burying or no Importance with for example the name of your cluster, right?  No, we're just we're at the pod service. We're at the service level Because that's where we're dns, uh, core dns really shines, right? It's the um Uh at the service level. So and then just uh Just a quick aside that cluster ip that you see there the 172 dot 30 So that ip address will come out of what you define as the service ip range in your install dash config dot yaml So if you change that from the default You could see a different ip there and that's perfectly fine Yeah, yeah, so it it it gleams that information from the range you give it and it'll it'll take that information And then it will create this ip address. So um, and this is the dns ip for your pods and that's the cluster domain. So uh one also quick thing before actually just drill down into the pods is the The cluster dot local is um, you shouldn't name your Your cluster, you know open shift dot cluster dot local like that's you'll you'll you'll have you'll have really bad bad time Anything local anything that essentially anything dot local. So That actually came up With some with some customers because they've had like dot local domains. So kubernetes has essentially ruined that for everyone So we have a question christian So sonics, uh, so how does this talk to the router? How does the service talk to the router? I'm assuming core dns and uh, and the dns operator There you go. Yeah, so, um, I'm actually I'm actually going to get to that workflow. So, um, I imagine you mean Yeah, um So but in general the service, um, the the service Layer is a non-routable layer, but that doesn't mean things can't get in and out So but I'll I'll talk about that workflow in a bit. So So yeah, so then this is the, um, the ip address for The the dns service inside the the service name, right?  There you go. Yeah I've never seen that dollar interesting Anyone knows jq. Let me know what that dollar does. Um Um, so, uh, that's that same that's the the service network range. So as, um, As Sully was talking about this is the range and that's and that matches this this ip here. Let me clear it. Um, I always like to clear because you know, people don't like looking at the bottom I know I hate it when someone's sharing their screen and the they're like, can you clear?  Um, yeah Yeah, so, um So before, uh I wanted to go through this doc line by line, but I think I'm gonna skip around a little bit because it just makes more sense, right? So, um the core dns has a um Has a configuration And so, um, and that's this is stored in a config map, right? And by the way, andrew and chris I can ramble all in on so please give me time checks So I can get to the all the points I want to get to um And so, uh, so this is the configuration file All right, so if you do that, um, oh, I hate this managed fields. I can't wait to Kubernetes um In 4.7 in the GUI it automatically collapses those in yeah in 1.21 I guess which would be open ship 4.8. It'll It'll take take out the managed fields. You have to specifically ask for them. So um So this is uh, the configuration file for core dns. It's pretty simple. Um It first gives the, um The the domains it's it's it's the authority for and then, um, it even gives an ad arpa for ipv6 as well. So, um That's you know, something some our customers ask for is ipv6 support. So, um It'll send information, you know to Prometheus. So, you know, it has that plugin already set and then it has this line here called forward.resolve.conf. So let's let's explore that a bit. So, um So the pods all the pods have their resolve.conf set to let me describe that again To this ip address, right? So that's um, you know, uh dot 10, right? 172.30, zero dot 10. So let's get let's get some pods. Um I know I have an app running here. Uh, there we go. Test this my sequel. Uh oc dash end test rsh this guy And then do bash because for some reason sh everyone likes sh command not found clear. Okay. Well bear with me Can I do this? Yes, I can. Okay. And so, um That's about the second. That was Yeah, control. I'm like that was control. By the way. Yes, I didn't see that. Um, if I do a cat etsyresolve.conf It has a search field, right? So this search field breaks down Pretty easily. First it it does the um the namespace Right. So the namespace take is is a part of the dns. Um, so it says, you know, anytime you do a look up look up First look up the name from this domain Then this domain in this domain and then finally this domain, right? Um, last one is your external, correct? Yeah, this one is my is external or my my internal but to open up externals. Um, domain It um, so if I you know, if I try to do do I have dig? No, in this look up Do I have ping? This is a very small container. Um, yeah, so When it is hosts on there. No, this is a very small container. I think I think this is alpine That it does a search, right? If I do it like an NS look up a foobar.test.isv.local Oh, sorry, if I do a foobar, it'll automatically tack this on look for that then tack this on look for that and then it'll go down that side Um, then it has the name server for this pod to that the dns the internal dns server inside of OpenShift. So, um It says 172.30. So it'll look up. This is its main dns server and it's the only dns server this pod has so, um It'll only ask the the internal dns server. So what happens if This pod tries to reach outside the cluster Right. So what if I tried to do a ping of like dns1.ocp4.cloud.chx, right? Or if I do a ping of google.com How does core dns, um handle that request, right? And so, um, the answer is wasn't that other Uh, where's that yaml? The answer is in the next line, right? So, um It said, okay. Well forward everything that I don't know about to the host resolve.conf file Um, and in the host resolve.conf file look for everything sequentially meaning Check the first second third fourth one two three four five six. Yeah, exactly. And so, um, so in the pod It sends everything to that dns server. So that's first and foremost That's the only config in the pod and then the core dns server will then say, okay. Well, I don't know this I'm just gonna forward this onto my host resolve.conf file. So let's take a look at what that looks like Uh, oh, this is a windows server. Interesting. Okay. Oh, see our The debug this is this is what this is a server that I hack on for windows containers, by the way. So, um Hopefully no blue screens here So this will um, so now we're we're basically remote shelling into the, um The uh the node here. Yeah that one node. So I do, uh Uh ch root Christian just to uh, I answered in chat already. Um, so uh another question How do you how do you change the order of name servers after the cluster has been installed? Um And hopefully our answer is online Yeah What did you answer and I'll say yes, so I would I would um, so, uh Changing the order I would put it in Um, like in the machine config or if you're using dhcp, just change the order in dhcp. Um, if you have A particular name server you want to connect to for a certain domain I would use a core dns's plugin feature and like the only plugin feature That we support right now is forwarding and I'll go over how forwarding works in a bit, so Yeah, and so it's in line with what I said. I just didn't do the uh machine config part. Yeah Yeah, well the machine config if you want to just actually change the order on disk Or dhcp, right? Like if your dhcp is handing out your name servers and just change the order there Yeah, um and next time the lease happens. It'll it'll flip them. And so yeah So here I'm in I'm in the node and if I do a cat etsyresolve.conf Um notice it it's uh, I'm using nm state, right? Um And so Here it'll choose this name server then this name server then this name server sequentially, right? um, you'll notice that the first entry is um My ip address, right? So I could do ip adder, right? Uh Is it or is it that's a lot of Yeah, well if you an open ship all the pods. Yeah, it's all yeah all the pods, right? Yeah, it's all the uh virtual interfaces like this one that's making you do a fine. It's there line 11. There you go. Oh, it's interface 11 Let's do that. Yeah. Hey, what do you have? So yeah, there you go. So um So this is for um the the host file, right? So it'll it'll look in the host file first Then it'll do dns one dns two um in that file. So that's how that that um that configuration works If if I go back here, um, it says forward everything I don't know about to the result.cont Um, and then it'll use those name servers. Um As well. So very cool. There is um A few things I want to um, so here Right, it tells you operator status You know, you just describe it and just kind of see, um You know, it was degraded because I did a earlier because I was doing an upgrade But that's and then the the logs here in the dns operator That'll give you information about the logs. So one thing to note, um We have a question christian. Um, so from fahad For vSphere ipi does the dhcp server have to provide dns records for the nodes? And I think what that's asking is Does dhcp need to do dynamic dns updates with the dns server? So, uh, I'm going to say soft. Yes, it has to There is a way to provide dhcp For only the ip addresses and not dns That gets a little Laborious, I would say and I would I would recommend using the dns forwarder For configurations like that and I'll I'll explain that In a bit so the dns forwarder It's probably the catch off for a lot of these use cases And I'll I'll add on to your responses saying I would I don't think it's required, but it is strongly encouraged So and that is specific to on-prem ipi So the on-prem ipis use mdns For their local node resolution So inside of the cluster it will resolve those ip addresses without an issue because it's using mdns For that purpose however if you wanted to do external resolution like you saw christian a moment ago do an ssh I think maybe it was a debug But if you wanted to ssh to that node or you needed to have like you're exposing a node port and you wanted to Reference that by name, you know dns name of the node instead of ip address Then you would need to have external dns have those node names, which is where that dhcp updating dns would be recommended But as far as I know with on-prem ipi so v sphere rev etc They they don't need node names and external dns I'll also add on that if you do have dns names and they're wrong or they are different The nodes will actually take those names and use them instead So reverse dns will override the name that's given to And i was trying to check christian we got 15 minutes. Yeah, okay. Cool. Take care. So thank you so here The the dns service Here runs as a staple set Not staple set daemon set. I think yes. So daemon set meaning you know, it'll I have actually I actually have Seven nodes or one of them is a windows node, but the staple set Selector is set to os litics, right? So if I do demon demon set Sorry demon set staple. I was working with something else with staple set and I have staple set on the mind And so yeah, so that I have I have six. So there's The daemon set does you know one pod for each each host that's added with this label Um, and it's running in every cluster, right? And you notice that there is, um, three containers per so let's do a describe on that Uh, describe Yeah, what are you describing? You should know You should know what I you should know what I want. It's just intuitively pick that up, right? Like yeah Yeah, I should just know what I want I could read my mind, right There was um I used to work with a guy who would make, um, a lot of typos and you I would always hear him tell the computer, right? Because we always yell out our computers Right, do do what I uh, do what I meant. Not what I said I have the intelligence no wait Do as I say not as I do, right? Yeah, yeah So, uh, the the container that I wanted to call out inside the pod is the uh dns node resolver and, um, its file is essentially its job is essentially to manipulate the etsy host file, right? So if I do a, um What is that rsh, uh, and I think that's on the pod host No, it's not here. So, um, its job is to basically manipulate the etsy host file on the pod and so, um You know notice this has like the ip Of the pod. So this is my ip Meet my ip meaning the the pods ip address and all the possible names um, that's associated with that and that's what the um the job of this, uh What's in your resolve.com For that on the pod. Yeah Yeah, I know we looked at that a moment ago, but just to compare Yeah, so it has, um But the dns server, right? So yeah, it doesn't have to go and look for the um The name for like the pod itself. Yeah, right because it has that entry in etsy host Uh for anything else it uses that dns and if it doesn't find that then it just forwards it on to um The etsy resolve.conf on the node. So, um, the last remaining five minutes Let's uh take a look at I've been promising the dns forwarding Yes, right. So, um core dns The uh, if you look it up a core dns, uh Let's look that up. There it is Uh, it says I was reading this the other day It was like a like a y or something like that. Uh, yeah plugins, right? So it's it's a um Uh, it has the ability to do plugins, right? So it's essentially the core dns was made to be extensible um, and one of the things so just beyond Name resolution one of the things is that uh forwarding Is written as a plugin. Um, and you do that by editing the the default configuration So let's take a look at that here Without the dollar Who put the dollar there that copied that tell docs, okay? Um So here essentially, um Yeah, it's essentially here. Let's let's do this. Um, okay, so it's not there spec We're deleting you spec and adding these here. So, um Here you're plugging in dns saying that okay, so anytime someone looks up food.com instead of going through that That whole chain, right? So let's kind of recap the chain the pod looks at its etsy host file locally then it looks at um the its own resolve.conf if it can find it it will then forward it to the Resolve.conf file on the node itself so Instead of doing that the the pod will then uh the the d sorry the pod the core dns When the pod asked core dns, it'll say well for any domain food.com I'm just going to forward that request to these dns servers. So this is um You know, I ran in my day, I ran, um I don't know cluster dns server sprawled up for for my company sprawled across all over uh, north america. So Um, you know, sometimes you do dns delegation. Sometimes you have For whatever reason tech debt You dumb reasons Or when we say dollar sign reasons, you have dns server that's kind of isolated from the other ones that they don't know about each other Um, so if you have you know stuff out there that you want to reach out to you may have to ask that dns server Um, and this is essentially an array, right? So I say food server the zones, right? I can you know put baz, right anything under baz Um and use this dns server What if I'm cool about core dns is that you can specify a port you can't do that in bind Um bind everything's on 53 Well bind you can listen but um the client can't ask on other ports, so um And then the bar server You have bar example.com forward it to these uh dns server. So In this configuration Then once you know if Then it kind of injects itself in the middle if if core dns can't find the the name instead of forwarding it to result.com It'll try these servers And then from there if it doesn't know then it'll forward it to the result.com locally. So, um, this is a way to You know add different dns servers in In your cluster, right? So as opposed to like, you know the core upstream one From your result.com file. So um, so I have a question for you christian as well as there is a uh a question from Amen. Amen. Um, so I'll I'll ask mine first because I think it'll be a quick one. Um, which is Core dns, right the the open shift dns service also applies to static pods that are deployed. So for example the etz d pods on the control plane Correct. Yeah, so that um the the static pods also get uh Because they take part of that sdn, right? So they're they'll they'll get that same configuration Yep, so the and the the five second version is effectively for pod resolution So if you're in a pod and you do a dig or an ns lookup or something that It first looks at its host file It then looks at its result.com which will point it to the core dns pod which is It's a service. Um, that's going to be running. There's a pod on every one of the nodes in the cluster And that will then look across all of the other Services and the forwarders that are defined in this dns config that you have up here before finally Looking to the resolve.conf at the node level So if it is trying to resolve something external to the cluster You need to make sure that that resolve.conf on the node is configured correctly, right? Correct. Correct. And then also, um Um, one last kind of thing to interject there the host is, um The the nodes host Etsy host file also does come into play in that same layer as the resolve.conf file, right? So it'll um You know, we're looking at the the host resolve.conf file that that Etsy hosts there also Um comes into play so So the the question from from iman, uh, can I install open shift ipi across Multiple network zones for example one for control plane and one for workers So you you know the answer to that I do um the answer The answer is for ipi Today, unfortunately, no because of how keep alive D works the way we have it configured is that it requires layer 2 adjacency um so um For it for it to work, right? It is possible Uh to configure keep alive d to work across dot gone layer 3 Um, but that's not how we configure it and that's beyond the scope of of support currently Yep. Yeah, I have nothing to add that you've seen me answer that like 400 times and I know I know you've answered it too. So It should be a bot, right? Um Yeah, it should be Yeah, I think you should have that regex. Yeah So fahad, uh another question can open shift span multiple esxi clusters on the same vcenter when using vsphere ipi That's an Andrew question That is on the roadmap, uh, so I know that Technically if you were to look today at the documentation for like the industry in entry provisioner and all that other stuff You can create the vsphere.conf and have it understand multiple clusters and all that Uh, so we've had a number of customers that are asking about this We did some internal testing. We found that it's mostly works With 4.6. I think they were like 95 success, but they found a few things They created issues It is there is a a jira issue for it. I'll have to dig up the jira issue And if I can do that the next four minutes, I'll post it in So not not yet, but multi dr s cluster clusters Is something that is on the roadmap cool That sounds cool so Another question for you. Yeah, so set up dns forwarding on my home lab, which works great If the upstream dns server becomes unavailable the authentication cluster operator goes into a degraded state Any reason for that? um, so I So the authentication cluster operator So I'm gonna so first and foremost I'm gonna need more information like that or like digging to the why that would be so Before I go into my long-winded answer for my short-winded answer is I don't know But I've had issues. So usually the authentication cluster operator being degraded is a symptom of something else, right? And so I've always found some weirdness with the router So if you have like a router operator, I would check the look at the the router logs. I would look at the authentication logs I would look to see There might be a breakdown of the dns chain at that point um the cluster The authentication cluster operator probably has the upstream quote-unquote upstream dns name like oauth dots, you know You know cluster name dot example dot com And that's probably in the pod and since it can't look that up. It's probably breaking down I'm just guessing at that point Is what's happening Yeah, no such host. Yeah, see look there. So the so the if you're watching this and not looking at the the chat If you look at the chances It says no such it's no such host um That's that's your external lookup and I think that's that's what's failing the health checks for the external lookup Yeah, I think the the authentication operator relies on the quote-unquote well known endpoint um To certify that it's talking to the right thing and I think that connects externally through a node port Yeah, so it looks up the node name Which it's relying on that external right those aren't stored in core dns Those would be in either the upstream dns or in that mdns responder depending on your installation type So I think that's why that's happening Yeah, I did so we're just assuming. Yeah, so oh we got two minutes. Nice. Yeah less than technically All right, um, so monster frames, um facing an issue I asked him to join discord by the way and ask in there Yeah, or I don't think we can tackle open stack right now No, I'm I am not an open stack expert by anybody's the stretch of anybody's imagination. So, uh, please Chris doesn't have a cycle to yeah, so Please follow up on either discord or you're welcome to send me a message andrew.sullivan.redhat.com or If you want to send me a message on twitter Practical andrew and uh, I'm afraid to say that I believe my direct messages are publicly open So you can you can reach out that way mine are definitely open for short two s's Feel free to DM me on twitter too. Yeah, and we'll we'll connect you with the right people We'll get an answer for that so don't don't hesitate to reach out Um, so chris, uh, christian my apologies, uh, any last minutes closing thoughts in the last 45 seconds or so Yeah, so it's it's really, um, I posted link to the docs. Um, all I can say is just, um, if you're having trouble with dns Just follow the chain and see where you're falling down. Um, so it uh many times You know, uh, people blame dns when it's not dns where dns is a symptom Um, so it's good to always troubleshoot with the osi model We had one of those even at one of those this morning Yeah, um So thank you christian really appreciate you coming on today. Um, as always, it's a pleasure to have you appreciate you sharing the knowledge Thank you. Um So for our audience, thank you for joining us today As I said, please don't hesitate to reach out at any time with any of your questions Uh, you're welcome to contact me directly via email andrew.sullivan.redhat.com social media practical andrew on twitter Uh christian, do you have social media or other? Yeah. Yeah, so unfortunately, I have um a weird social media handle Not too weird christian h 814 so, um My first name in last initial 814 so now you guys know my birthday and I expect presents Go ahead and there you go. Send me a tweet. Uh dm dms are open. It's 2002, right that you were born Yes, yes So And as always chris, please, uh, please take us home Yeah, uh, thank you all uh coming up on the channel here in mere seconds. We'll be an open shift comments briefing talking about Uh building multi cloud provider platform gubernettis. So stay tuned. Thank you all