or Spanish for the second semester. And also they're waiting for the results of their AP tests. Got it, Kim. Ed Maria. OK. The 600 is pretty high. This is a piggyback on, well, I think this is an excellent program. Let me say that. But to piggyback on what Kim had asked about, to me, I would think that over the years, it will be far more improvement as far as the numbers of students who receive the seal by literacy. So if we look in 2012, there's been a constant increase throughout the years. So I'm wondering what was done differently then that maybe we're not doing now or areas in growth as far as how we as a district can support these students to ensure that they get the opportunity to receive the seal by literacy. Right. And I think that would be at the high school level with the counselors and with the students. I know that there was a change in that for some of the years, it was the KC. And now that the level of English proficiency is higher that the students have to demonstrate on the SBAC. So there have been a few changes over the years. But I agree. We have many, many students who come into our school speaking languages other than English and to really encourage them to become multilingual. Right. And so what were your recommendations be as far as if it's promoting? I think it's promoting. I think it's having a stronger push. It's something that we've also started doing with the language ambassadors, right? So now the students in elementary have that. And they say, well, we saw those seniors getting those medals. So we also hope that that's going to be an incentive for them. The superintendent wanted to chime in here just one second. So I was going to piggyback on the language ambassadors. And if you saw how many students in fifth grade received their medal for the pathway, for already being on the pathway. Also, we are going to be bringing back later part of the A through G change of trying to change the requirements for our students is looking at using their assets that they're bringing. So right now a student does not have to take foreign language. They can choose between taking one year of language other than English and then one year of VAPA. And we're really looking at changing that and looking at increasing that so that the students are in fact taking a language other than English. And it's not really just for the Seal of Biliteracy, of course, it's because we know in the global community that bilingualism and multilingualism is important. But that will definitely spearhead it. So not only do we have kids that are at the lower grades that are having aspirations towards that, but we're also setting up the system to where they can take those classes and then get themselves to AP Spanish. Because right now we're not. The third thing I'd say is with the creation of the second dual immersion school, we're looking towards really creating that true pathway so that we have biliterate students so that they're taking not only AP language, but also AP Lit, which will help to really get their AP scores up and then increase the number of students as well. Thank you. Well, after Michelle said kind of what I was hoping to hear is we really need to differentiate the experience. If we really want to build a successful biliteracy program, the Seal of Biliteracy program, we really need to make these kids feel special. So I think the Seal is great in the senior year of high school, but it really starts in fifth grade, right? That's when we differentiate that experience for those kids so kids want it. They aspire to it. That's when you're really going to start to see the growth of the numbers. What I'm so glad you didn't say, because I was worried I was going to hear from somebody, is we need to keep those standards high. We need to have kids shoot for the highest standards they can. And when you can differentiate the experience and yet still have very high standards, that's when you're really going to see a difference. So thank you for your work. In my personal opinion, that's how you do it. Start with kids in the fifth grade. And it's just it's starting with TK and kindergarten students. And that's even better. And that's even better. Great. OK, great. Thank you so much. We appreciate it. So that was just a report and discussion. I had no action, so we'll move on. Next is the PVSD employee recognition. And this is, I would think, Dr. Colleen. Again. President DeRose, board trustees, Dr. Rodriguez, I would like to call upon Pam Shanks, our HR director, to come back to share the activity plan for the upcoming employee recognition. Good evening. I'm here to present the item for employee recognition that we do annually in our district. Recognizing and honoring employees of the district is one of the board's fundamental goals. Employees who have worked for a remarkable number of years for the district are to be committed for their service. And we will be honoring them again this year by recognizing employees who have reached different milestones with their employment with the district, starting with 10 years, 15, 20, and so forth. During the week of May 21st, we're going to be honoring our employees for their years of service with the lapel pen and certificates that will be sent out to their site so that their site co-workers and administrators can honor them for their years of service. Additionally, we'll be holding a special recognition for those who have served for 40 years in our district. And we have about seven people who have served for that number of years in our district. And I just want to make a note also that five of those seven are classified employees. So it really shows the dedication of our classified staff in our community. And so the reception will be on Wednesday, May 23rd. There's actually the time is wrong in the board backup. It's actually going to be at 3.30 in the boardroom. It's not 5 o'clock as it's noted in there. So if you can just make a note of that change, you will also be sent that information as well. So you'll get the correct information. But it will be May 23rd in the boardroom. And you're all welcome to attend. So is it going to be here or at the district office? At the district office. OK. So after that celebration, then the board members would come here for the board meeting. Yeah, it'll be from probably about an hour or so. So 3.30 to 4.30. OK, perfect. And that's where we will also be honoring our retirees this year as well. Thank you. Thank you very much. That's a great celebration every year. So thanks for doing that. And I'm always so impressed with the number of long term employees that we have here. That's just amazing to me. So it says a lot about this district. Yes. So for the numbers this year, we're going to be honoring about 300 employees that have hit those milestones, the 10, 15, 20, et cetera. So. Right. Are there speakers to this item? None. OK. And are there any more comments from the board? OK, great. We look forward to it. OK. Our next item is our wellness policy. Mark Brewer, Assistant Superintendent for Secondary. Good evening, President DeRose, members of the board, and Dr. Rodriguez. It's my pleasure to present to you this evening the wellness policy update. We're required every three years to update the policy. Included in your packet is the work that was done by Linda Liu. She did most of the editing and gathered input from the Consolidated Health Committee that works through the district, as well as administrators, and then updated the policy we shared with you tonight. Areas in red with a strike to line through them, which are areas that were removed from the policy. And then areas that were added and updated in the policy are noted in blue on the document. So we wanted to present that to you and let you have time to review that and ask any questions about the process or input. I'd invite Linda to come up to the podium in case you have specific questions for her. And also we've kind of transitioned Joe into this process as well. Moving forward, so three years from now he will work to coordinate the update. So we just wanted to present it to you and get your feedback or answer any questions you might have at this point. OK. Are there any speakers? No. No, I don't see cards. OK. And are there any questions, comments from the board? I see none. Yeah, I have one. OK. You've got to say yes or no. I was nodding. I can't tell unless you don't say something. Go ahead, Jeff. And then Karen and Kim. We had a couple years ago. And I don't know if this plays into this. But we had a couple years ago. I had a discussion. I remember it as heated with your predecessor, I believe, about cupcakes in school and birthday cakes and all that. And Susan's nodding her head so it was heated. I'm not wrong. And the other piece of this is that there's a lot. I remember a lot of fundraising on campus with food. People sell cupcakes, whatever they may sell. It seems to me that we are still under those constraints. First, I'd like to introduce myself. My name is Linda Liu. I'm the Director of Food and Nutrition Services. Linda, how are you? Thank you. Was I directed to the wrong person? No, no. Linda can answer me. OK, Linda. You can directly do that in the fundraising component. But it's a good question. Thank you. And I believe I was at that board meeting where you had the discussion. It was heated, wasn't it? No, I wasn't the director at the time. Yes. So we have, I would say, in speaking with the principals and getting feedback about the wellness policy, that some of the biggest questions revolved around the two things you just brought up right now. So I would say classroom celebrations and fundraising. So a lot of that feedback was taken into some of the revisions in the wellness policy now. What I did was, because the wellness policy rules are passed from USDA through CDE in the nutrition services division, which is where most of my training is coming on wellness policy final rules. So all of those rules I'm receiving through trainings and all of that federal and state-required rules are in the language of the wellness policy. And I did explain to everyone that took part in the feedback on the wellness policy that things that are given to students are not part of those rules. And so what we decide about some of the rules regarding classroom celebrations, meaning things that we have parents bring in or what teachers may provide free to students, is a local policy. And so actually there was a decision to loosen some of the language. And that was part of the updates to this policy. However, it is still, there's most of the language is there. But it's been loosened to, I think, give principals and teachers the ability to maybe kind of allow certain events, but still not have their school overrun with cupcakes on a daily basis. We've moved to the middle, is what I'm hearing you say. And I think that's good. Because I do think you don't want to go overboard. I think the other, but I think that's good to hear. I know there's some concerns around peanut allergies, which I don't remember as a kid, and some other things. So we need to make sure that all of our students are safe. But at the same time, if a parent wants to bring cupcakes on somebody's birthday, I don't want to make that into a federal offense over. So thank you. Thank you for listening to what's going on and giving the local sites some flexibility. So just as an example, if you look at page 6F, which talks about food rewards, that shows you what was put in previously and what we took out. For example, it used to just say school staff will be instructed to not use food as a reward. And now it's as a general rule should not. We also changed it limited to one time per month to one to two times a month. And then we also, instead of high sugar items are not allowed, it's are discouraged. So you'll see that that's just a concrete example of how we tried to meet in the middle, but still do what's in the best health interest of our students. And I would say even though you're feeling relieved that maybe some of the language is loose, and I also get that feedback from certain parties that they're very upset about it. So I think to them, I say, to them, I say for school sites that want to support the stricter version, they definitely have all the power to do that. And then the sites that would have an issue implementing this right off the bat have a little bit of leeway and can kind of work that in. So I think it's a good starting point for everyone to, like you said, meet in the middle. I think the flexibilities are really important. You're right, if there are, if a school site really does, really has a problem with some of these issues, and if they want to be stricter, that's fine. But some school sites want the flexibility, maybe they don't have some of the issues that are trying to be addressed here, then give them flexibility. I think that's a perfect solution. Okay, and Carrie? I heard that they were gonna try to, I don't know why I heard this, but they were gonna try to do it so that children's birthdays, they would do them, all the children that had birthdays in one month, they would maybe do it on one day of the month, as opposed to all the children's birthday, bringing the cakes, the cake, whatever, all the, I don't know why I've heard this, but that they would do a birthday for all the children that had their birthdays in that month. So then they can, so that the cakes, so they're not bringing cakes three or four or five or six times a month, they're just bringing cakes or cupcakes one day, and they're gonna celebrate all the children that had their birthdays on that one month. So Karen, on item F on the second paragraph, it does mention that, it says classroom celebrations for holidays, birthdays, or other events will be limited to one to two times a month, so they can do that. Oh, so they're not saying that. Yeah, because maybe I read it here too, but I've heard people talk about that, but I probably read it, I read this too, I'm sure I probably read it. Right. Yeah, so that's a, what I think is a very good idea to me, I think that's a really good idea. And then you're just limiting the amount of sweets to one day a month, as opposed to a lot of days. Yes, for someone that celebrates their staff birthdays in the office when we have a week after week and that I'm doing birthday events, I also think about that, is there a way I can just combine them into once a month? But yeah, I think that's, I can only imagine for a teacher with that many students and you have almost every day covered for the month. So celebrating by month is a great way to kind of model that sweets are treats and there's definitely a place for them in your life, but it does not mean that it's a daily thing that you eat and I think that's part of the modeling that can happen. Or once a week even, yeah. Right. But I'm trying to think, did all of you, I can't remember exactly what the letter said, but I got a letter from a teacher about food and it was about something about, something about some kind of plastics to you that we use to cover them that are very toxic. Do you know what I'm talking about? Do you remember, did you get that letter? Not that I recall. Letter from a teacher that talked about food and talked about, yes. You got that letter? Yeah. And he was talking about something about plastic. We got a complaint from a student that all, that the food that is heated up in the cafeteria ovens is all completely ensconced in plastic and he didn't feel like that was very healthy. Yeah, something like that was really super unhealthy. He thought the plastic that used was pretty even toxic and that we need to figure out something to do so that the food's not. Is that something you can speak to? Are there any regulations around the kind of plastics that are used? Anything that we use is definitely food grade level. So there should not be any issue with that. We do hear issues with packaging. I actually hear more about issues with packaging and waste in general and not about any toxic level of wrap that we use. And I kind of thought, whoa, I kind of agree. Well, maybe I agree with what he's saying. That doesn't sound good. I just remember reading his letter and thinking. Right, anything that we're procuring is all through mostly the typical child nutrition companies that all the local districts use and so they're all definitely safe in food grade. But part of the reason that anything is covered is just for protecting health actually. So any kind of food cannot be served open unless there's, if you have a kind of covering over the area, there's kind of a safe way to serve it open but when you're portioning out food and then if there's any kind of transport like our elementary schools are satellite it out from the district office to 15 elementary schools. So in that case, food cannot remain exposed. And so that's the reason for that. Sorry to interrupt you, but I think we get the idea on that. I'm just concerned was this students concerns addressed. I mean, that student should probably get information about what are our practices and what are the products that we are using. So if the trustees that received the letter or if Dr. Rodriguez did, they could forward that maybe to Dr. Rodriguez and follow up with that student. So for me, I have a question on page nine under school health and safety regarding counseling, psychological and social services. Is that new ed code or is that local policy? I mean, I like it, let me just preface that with that we shall partner with programs to provide counseling services to students. All schools shall provide professionals qualified to provide student counseling services to promote social and emotional wellbeing. So I wouldn't be able to speak to anything really far outside of what's required in the wellness policy and what regulations pertain to nutrition services and nutrition meals and the required language. However, part of the reason that the section was added is that in the rules for creating a local school wellness policy, we are required to incorporate what they say is eight components of school health. So what goes into the wellness policy is more than cupcakes at a party. It's eight different components thinking about student wellness overall, right? So how they, safe streets, clean classrooms and it's all of that and not just kind of food related. And so there was discussion from the coordinated school health group about somehow making sure that all eight components had language within the document to kind of highlight this overall well-rounded wellness idea. So that's where the language was added from. And we did have directors from each of those topics add that language. Okay. Is that the counseling, who can answer that more clearly? Because I understand that's not your area. It's one of the eight components. So we added the counseling because we provide those services in the district. So when working with the consolidated health committee, we have representatives from PVPSA. We brought in social emotional counselors so they asked us to add that as part of our policy so that it's an overall encompassing plan that's more as she spoke to than just changing the food rules every three years, but incorporating the physical aspects, the physical activity and PE, nutrition in general, and part of it was counseling services. So we added that language to our policy, which we're allowed to do. Right. So I understand that, but my question was, and I guess you're answering it in a roundabout way, that's a local policy. It's not ed code that says we have to put that in there. I would love it to be our policy because I support that. There's no specific thing around health and safety that it dictates you need to have that section, but what we construct in that section would be up to local control. Thank you. Oh, I was just gonna ask. Okay. Yes. So if we have one more question, then we're gonna have to move forward because we do have much more to do. And our, I'm sorry, Billy's one something, Maria and me. But you can go after, yes, our final comments. I'm not saying you can't speak. Oh, okay. First. So I just want, yeah, and where it says here that health education curriculum should include a variety of topics such as personal health, family health, community health, consumer health, environmental health, sexuality, education, mental and emotional health, and so that has all that kind of stuff in there as far as it says that the curriculum should have that kind of stuff in it. Yes, and that comes out of ed code. That's what we have to cover in health classes. And that is specified in ed code. So we put that into the plan. So students get that during their health education. Okay. So all that's in there. Okay. Thank you. A lot of my questions were already answered. And there's a lot of questions. A lot of my questions were already answered. And there's this whole thing on professional development and PE which I was curious about. I was hoping that our elementary teachers were getting professional development and PE so they know how to keep their kids active. One of the things I've long been concerned about both as a board member but mostly as a parent is that we had cut back recess hours. I mean hours. I wish it were hours, minutes at one time. It was probably before Michelle came and I don't think we've ever put them back in. It says that our kids should have at least 20 minutes of recess per day which seems like such a small amount really for children. So I'm wondering if somebody could speak to that. So I'll say that we actually have more than most. So most school districts only either have a morning or afternoon recess. We have both. The reason why generally it's challenging to put additional minutes in there is because that extends the day for everyone. We're required to have so many instructional minutes. I will say that our students definitely receive their PE minutes because we've had litigation on that and then they do receive both morning and afternoon recess which in most school districts that is not the norm. So by EDCO they have to have at least 20. Our students in general have 30 if not more. Most of them have 40. Does every elementary school have the exact same amount of minutes for their lunch and their recesses? They do not. There's a lot of variability in this school district. Most school districts have all your elementary schools have the exact same instructional minutes and pretty close to have the exact same sequence. Now we do because of transportation almost all school districts have some you know some shuffling of start times but we have a lot more variability than most school districts. Okay and I like that. I like the language in here about not denying children recess based on punishment because kids really need to exercise and there's so much research recently. I just got something yesterday, this big study about when you let kids actually have more recess they can attend and focus better. I think we all know that. The one thing I didn't see in here is just some kind of acknowledgement that we should have later start times for our teenage students even though we don't. But it would be nice to acknowledge that that is a known fact that kids need more sleep and they do better if they have later start times. Their mental health and their depression actually is alleviated if they can sleep in later. So anyway. No you don't have to respond. Thank you. Thank you. I think it was two years ago that two of our schools decided to pull out of the lunch program and went to a private vendor. Do you know if that is still working? As far as I know we feed every school at PVUSD with the exception of SEBA and SEBA has never been part of our school program as long as I've been here which is five years. We actually gained Watsonville Charter and Lynn Scott in the past few years. So we actually added schools that we're feeding. So they're not under contract with the private anymore? Anymore. Thank you. A comment is that if our health insurance premiums are going up six percent whatever. That's like 2.4 million dollars. One of the things that we're trying now in the private industry is for a corporation to put in a very strict wellness plan for everyone. Not only the students but the faculty and so forth to keep everybody healthy, to prevent illnesses. You know we're always going to have some of course that keeping everybody as healthy as we can to reduce claims which reduces premiums. I don't know if that makes sense. The other item is that back in 1990 when we first had the financial crisis the one cut that we made was the laundry service. What does that have to do with fitness? Laundry service provided the kids with a dry towel every day which meant that they showered which meant they exercised. When you take one of those things out it was amazing how taking out the laundry service which saved us a hundred thousand dollars a year reduced the number of kids that actually showered which meant that this whole P.E. program went to a non-sweat program. And so the very thing that we wanted we cut because of that one service. So maybe we should look at again bringing back that type of laundry service which would help us all. This is the institutional knowledge so great because I wouldn't have thought about that. I'm not prepared to talk about the laundry service at this moment but what I am and you'll be proud of us what I am prepared to talk about is we are going to be a recipient or we are a recipient of a grant through Kaiser Permanente through the Alliances of Healthier Tomorrow in which it is all about not only professional development for our teachers around fitness and healthier eating but it also is a staff component for staff health and really promoting events such as jogathons, 5Ks, those type of things and other challenges so health challenges which is paid out of this grant so it's not paid out of the district but really in an effort to promote health and wellness of everyone. And so Kaiser Permanente tapped us as one of the people and so we just had a meeting with them on Tuesday, Monday or Tuesday I think it was, I can't remember I think it was, it was Monday. So anyway, so we're working on that the towels I'll give back to you. Okay, Maria. Well before I go I do want to make a motion to extend the meeting to 1045. Okay, we just have after this item the consent agenda and then read out unclosed so I think that sounds good so is there a second? Second. Okay, all those in favor? Aye. Aye. Any opposed? Okay, motion passes 6-0-1. Okay, so for me I just want to highlight a change that I really like under the parental support it's nice to see that we included collaboration with community partners and connecting parents to community food gardening resources, local nonprofits and providing nutrition education and my question to that is what are we doing to do that? Well could you state the page? Yeah, it's page 8 and it's B is the first paragraph under parental support. I don't have all the resources prepared today but there's definitely working with Second Harvest Food Bank working with Life Lab there's a lot of nutrition education that's available and provided for parents there's food that is food programs that are provided for parents as well we partner with Second Harvest and the CalFresh which is food stamps outreach program when we do meal applications to start the year we link to them on our website so if you want to have something listed I can definitely do some more research to see what's happening outside of my department but I definitely have already attended and been able to see a lot of great things by other departments as well. Yeah, so I think there's just a lot of information to cover tonight to go into much detail so I would recommend to bring back and actually it's Kim's recommendation to come back and give a full presentation on food service and just a lot a little bit more time for us to be able to ask questions. What are health? Another point too to speak to that we also partner with local community non-profit we have community gardens right behind Diamond Tech Bowling Hills Middle and Pajaro Middle School all have pretty thriving community garden programs that support people growing their own vegetables and one at Diamond Tech has grown tremendously so we'll continue those partnerships as well. Yeah, well we have a lot of community garden we have Brad Cliff, Loni has a big community garden a lot of community garden. Can I just ask the point of clarification would you like just the presentation to focus solely on food services or would you like to bring back this entire topic and be able to focus additional attention to the entire document? Not just food but health. We can't hear you Karen can you speak into the microphone? I'm just saying no I hope we do it on health too just not food but just health. Food and health, food and health, yeah. I want the towels. And the towels. Okay, okay. Alright thank you so much for coming and staying late to help us understand this and we look forward to hearing more. So we'll move on to item 11 the consent agenda. Is there a motion to approve? Move approval. I'm calling for a motion. Are you making a motion? I made the motion. Okay wait, wait hold on. I make a motion to approve the consent agenda. Okay is there a second? Then discussion. Okay so. Do I wait now to do the full items? You can say that you want to pull an item tell me the item number and then I'll ask for a revised motion and second. Well I was going to just just a couple questions about migrant programs. Which number is that? The 11.4. That's speech and debate. Yeah and then the leadership institute. That's mostly. What numbers? Okay so 11.4 and 5. Yeah. Okay so we will defer those items and then before we take a vote I just wanted to make mention of the purchase orders. I think that's really important for the public to see a lot of times when we give our budget reports it's a really high level and this is a really good way to see, I mean looking at this one for this month it's a really good way to see where our money is going. It's going into the classrooms. It's going into facilities. It's going into furniture replacement and it's going where our student staff and families have asked us to put our money and that's proof of it. So if anyone's interested in reading a very long what seems like a dry but really exciting to see the good things that's happening. So with that I will ask for a vote on the consent agenda with items 11.4 and 5 deferred. All those in favor? Aye. Any opposed? Motion passes unanimously. Item 11.4. Go ahead Karen. Come up here. What was your question on 11.4 Karen? Okay well I'll ask him. But I'm just so impressed that you have about 5 things for migrant students during the summer. 5 different programs that are migrant students are all going to. I was going to ask you a couple of things. How do you figure out which migrant students go to which incredible programs? The my national one I know that they do that one. Yes specific one about you were talking about the speech and debate. So I did first to... We're on speech and debate right now. We're on speech and debate right now. So the students are selected to go and we won sometimes? This is a state level competition. It's set up by the California Department of Education Migrant Program. It happens every year during the week of May 5th. It actually already happened last week. We took 28 students. They were selected throughout the year. There were some coaches that trained some of the students. 34 students were selected. Only 28 ended up going. It's during the weekend. There's the four teams for the debate. Two middle school and two high school. The middle school is composed of one Spanish debate team and one English debate team. High school is the same thing. One Spanish and one English. High schools and middle schools were represented. In the speech format there's two pro grade level two and six all the way through 12th grade. One in Spanish and one in English. In terms of the competition the the in terms of speech senior from PD High placed second place in the debate Spanish competition in terms of the debate. The middle school level they were from four kids from Rowling Hills and one from Pajaro middle school. They took first place in the debate competition at the state level. So they were amazing. I thought the high school level should replace but it will brand new in this competition. We're lacking coaches who went to their presentations in their debate and I feel like they won their competitions but there were some issues in regards to the protocols that were not followed in regards to our team. They lost points and that's the reason why they didn't play. So we're trying to kind of get a coach that actually not a coach but a coordinator that actually can coordinate all the middle schools in high school and be able to get all the unnecessary coaches to get our team better prepared. Our kids were amazing. They did really well. They behaved really well. They represented PVUSD in Netslyn matter. So we're happy to let you know that we did bring a first place in the middle school debate competition in the Spanish session of it. Will they have a comment too? Yes. The migrant banquet is Saturday. So I just wanted to help and make that announcement and the banquet of course will serve non-fat foods. They'll go be salad. That's too funny. Okay. And it's all at Lakeview this year, not at Mellow in the Lakeview. So 11.5 I just want to ask you so they go to the LA area for the Leadership Institute, right? Because they're going to the Channel Islands. They go to the Channel Islands. Are they connected to UCLA? No. It used to be many, many years ago. It used to be a UCLA program. They moved there many years back. They took it over to Channel Islands. They took it over. What's the university that they work with? The Channel Islands. There's a university there? CSU Channel Islands? I'll be darned. Didn't even know there's a CSU Channel Island. Okay. So that's the university that they work with when they do the Leadership Institute. That's where they are. All right. Thanks. And they're selected. All three comprehensive high schools get to pick. We get 10 slots. They're in application. So who selects the students? The counselors, the migrant counselors at the high school level. So there's a process. They're in application. The old three high schools are represented. And it's based on numbers. What's the largest number of migrant students? So they get five slots. PEI has three. And Aptosize has two. Okay. Five, three and two. Okay. Are there any other questions that you can also notice? We didn't pull it, but the summer program at Cabrillo is on there. And as you know, I got... If you notice that we're adding one more and that's working with Gene in regards to the middle school growth and engineering and voice and engineering. And it's all provided by Cabrillo. They provide the professors, the curriculum, the whatever they ask for. Transportation or something. Transportation is in the Watsonville Center so we don't pay for that. It's working. It's great that all this great work is being done every year. And thank you so much for your leadership on that. Okay, so I'm looking for a motion for item 11.4. I'll make a motion. I'll second. She pulled them so I'll give her the motion and you the second. All those in favor? Any opposed? Motion passes unanimously and I'm looking for a motion on item 11.5. Make a motion on 11.5. Second that motion. All those in favor? Any opposed? Motion passes unanimously. Thank you very much. Okay, so we are going to move to close session items. Item 13 or actually 14. Close session reconvene if necessary and we do not have to. So item 14 is action on closed session. And we're going to start with expulsions and trustee Asmanson. I move to approve the recommendation of the district administration for a suspended expulsion for the remainder of 2017-2018 school year and the fall semester of 2018-2019 school year for 17-18-036. Is there a second? All those in favor? Any opposed? Motion passes unanimously. I move to approve the recommendation of the district administration for a suspended expulsion for the remainder of 2017-2018 school year and the fall semester of 2018-2019 school year with the placement of the same school for 17-18-037. Second. Thank you. All those in favor? Any opposed? Motion passes unanimously. I move to approve the recommendation of the district administration for a suspended expulsion for the remainder of 2017-2018 year and the fall semester of 2018-2019 school year with the placement of another school for 17-18-038. Second. All those in favor? Any opposed? Motion passes unanimously. Thank you, Karen. And now for the rest of our closed items, Trustee Orozco. Under item 2.2, I move to approve the district's administration I'm sorry, I move to approve the certificated personnel report as presented by the district administration with the addition of what administrative appointment, three new hires and three separations? Is there a second? Trustee Orozco, no second. All those in favor? Any opposed? Motion passes unanimously. I move to approve the classified personnel report as presented by the district's administration with the addition of one new provisionary hire, one separation and one separation from service. Is there a second? Second by Kim. All those in favor? Any opposed? Motion passes unanimously. And under item 2.6, the board approved a workers compensation claim settlement for authority claim number 806-01-0037. And under item 2.7, the board approved an MOU with LIBO school district with PVUS day for one special education student. Thank you very much. And upcoming board meetings May 23rd is the next meeting and it will be held here and just as a reminder the employee celebration will be held at the district office board room at 3.30 and then closed session will be here at 6 and open session at 7. Thank you everybody for coming. Have a good evening.


I'm going to give you a talk today about how you can learn from my terrible mistakes over the last two years of using Elixir in production, a little bit about what I've done in myself. For the last nine years, I've built online games exclusively, right out of college. I worked on games like Guild Wars 2, Loaded of the Rings Online, Dungeons & Dragons Online, thank you, League of Legends, and most recently, Moonrise and the State of Decay franchise. Undead Labs is the last game company that I was working with where a lot of these stories take place. It was founded in 2009 by one of the founders of ArenaNet, also the creators of Guild Wars. I joined in 2013. I was a round employee, 20 or so, now the company is up to 50 or 60, and they're a great group of people. One of the games that we were working on is Moonrise. This was the very first online game that Undead Labs was creating. I say was because unfortunately it was canceled, but it was not canceled due to any technical limitations. Just really hard to make a game in the mobile space and also execute in the PC and Mac space at the same time. We learned a lot when we were developing this, and we get to keep the technology and use it on our future projects. Other projects that we have are State of Decay, which is a zombie survival game for Xbox 360, Xbox One, and the PC. Original launch date was June 5th, 2013, then April 28th, 2015 was the re-release. Nope, nothing's wrong, okay. The game was actually created with just about 20 people. It's a single-player game, and the future of the franchise continues. I started using Elixir at 090 about two and a half years ago or so. I was actually really happy with Erlang, and when I first heard about Elixir was actually like 2011, my coworker pointed me out to something on Reddit or Hacker News, and he's like, hey, it's Ruby on the Erlang VM, and I'm like, I don't want that. I like Erlang on the Erlang VM. I don't want any of the garbage collector problems or the threading problems or lack of threading or any of the things that really came with Ruby. But the things that are nice, of course, are syntax. When I read the web page originally, it didn't resonate with me. I didn't quite understand what Elixir was, so I didn't think about it for another two years or so. And when I did start thinking about it again was when my friend Pat again said, hey, it's not actually Ruby on the VM, I was wrong. It's actually a nicer syntax, and that's what he pitched it to me as. And that was good enough for me to look at it, because while I was happy with Erlang, getting others on board was very hard. The Standard Library is fragmented in all these different ways. You don't know what's new, what's old. You don't know the order priority, the parameters, which totally depends on which module you're using on the priority where you pass in for like a collection, what the collection or what the enumerator's gonna be. And it was a little bit of a pain to use and a lot of boilerplate too. And not awesome tooling. So getting people on board was a little rough. The thing about Elixir is I came to it, I would have been happy with just like Erlang Standard Library too. I think Jose, this is really all you ever really need to create for me to be happy and to get a bunch of people on board would have been something with standardized and run by the community and not by a corporation. Which is actually a thing I really like about Elixir is that the community, if some of you are new to it, I'm telling you that it is literally one of the best programming communities I've ever worked with. And that's something that we also were able to cultivate in Elixir. But like I said, Elixir team delivered so much more. A great standard library, a build tool and mix which is extremely powerful and extremely easy to use. One of the, a lot of people in here might have done rails in the past and is really easy to get somebody started with rails new and mixed new is just as easy. I know rebar and concrete have come a long way as well. But two years ago, they were not at this stage. Amazing docs and they are first hand, you write docs that test the code that they're documenting and a package manager which I always really, really wanted in Erlang and that's in Hex. Thank you Eric, that was very much needed. Polymorphism through types with protocols. So for instance, if I created a JSON library and gave it to you and you had a custom type, you don't have to fork my JSON library anymore to determine how to serialize your type. You can now just implement the protocol and hygienic macros which I'm a huge fan of, started using them in closure and was addicted to them. And then I learned not to do that. So I use Elixir every single day for the last two years and I can tell you that I am extremely happy. I wouldn't be here today if I wasn't really pleased with the language, the ecosystem and the community. And it allowed me to be very successful in a very short amount of time. Elixir is, in my words, humane Erlang. It's Erlang made for people in this century. And I know that, again, I like Erlang a lot. I'm just being empathetic with all the other people that don't enjoy it. So Elixir, it's easy to get started, get it on your development machine and creating your first project. And if you're using Phoenix, the web is super quick to get anything going. But it's not so easy at first to get it into production. And once it is in production, it really is brilliant. So the purpose of this talk is to share my learnings over the last two years of what it's like not to develop Elixir, but how to take the baby that you made in your computer and force it into AWS or somewhere else, right? Last year I was here and I promised myself that I'd chat about my experiences. And I'd hope the room would have more people than last year. And it's something like double or more now. And that was amazing to walk into today. So first I'll tell you a little bit. I assume everybody understands video games and is into games and everybody programs video games, but that's not true. I just have never done anything else. So I'll give you a little bit of an example here. Basically, I build server hosted online games. They are a persistent connection. So if you've done web stuff, it's like HTTP too. You hold a persistent connection. They're stateful. So unlike HTTP, we run and simulate the game. And it's not easy to migrate or destroy an application and bring up a new one. So Erlang does something really well and Elixir inherits that, which is code hot-swapping, no downtime.  And they almost always have persistent storage. So if you've played an online game before, like if you ever networked Doom, that was not a persistent online game. There was no storage there and it was just transient you played together. But persistent games, or World of Warcraft, or League of Legends, or Guild Wars, EverQuest, Ultima Online, and almost most modern games today have at least some sort of back end storage. Even shooters like Call of Duty. And my specific role as a network programmer, I build game back end platforms. Game back end platforms are a thing that provides authentication, chat, presence, they're all the microservices. Purchasing, player storage, matchmaking determine if you should play this player or not, who's better, who's less skilled or more skilled.  So if you've ever played a Blizzard game, it's like Battle.net. And we just didn't brand them all the time. But almost every online game has one. And this is a high level landscape of what that looks like. This is not drawn to scale. I only show like four chat servers, but there could be upwards of like 128 of them running. And I only show the chat service. But basically the client is the game client. That's your phone or your computer. And that connects down to something called a lobby server. The blue lines are reliable UDP connections. That's the client speaking to the lobby world in combat. The lobby server, the world server in combat are colored brown. Those are the C sharp on mono servers. And then the purple servers are the elixir servers. And any red lines that you see are fully meshed nodes. And any black lines you see is just TCP. That's important later on to understand if it's fully meshed and not fully meshed, we'll get to that. The client talks down to the lobby, which holds a persistent connection. It then finds the player, a world server to log into, which is like what you'd be walking around in. And then once you do combat, you transition to another server that simulates the combat. And all three of the C sharp servers are clients of the back end platform. So anything in purple is that back end. And the routers are what they talk through. And they talk a binary protocol to the routers. And the routers is job, which we'll get to in a little bit. It's just to send the messages down to the appropriate service and then get them back to the person who asked for them. And then the nodes below it, say chat. Those are called service nodes. They provide a microservice. So in this case, chat and presence. And those are database backed. And they connect to database nodes. So the white nodes at the bottom, in this case, are Postgres nodes in our infrastructure. And they're sharded. Meaning that there's many buckets that contain your data. And then you go to one. It's a way to load balance. So how that breaks down into what it would look like in an Elixir project is pretty simple. And again, this isn't the whole application. This is just the small piece to make a point. Is a protocol library, a common library, a route app, and the chat app. The protocol contains the serializer and the DC serializer for our binary messages. And also the common module contains any active recordy stuff. Something that's useful for every single one of the applications. And then we have the route server. And the route server is a gateway into that backend. Like I said, it speaks TCP with undid server binary protocol. At one point, I had tried to go over what that binary protocol looked like to an audience and they were sleeping. So you can go look at that Erlang factory talk and take a nap. But I talked about it, if you'd like to see it. There's like two people who probably are interested, and one of them's me. And it also disconnects misbehaving clients. While we're not connecting players directly to the back end, somebody as a company grows could create a misbehaving server. And it also does a job of dropping hanging in partial TCP connections and rate limits different types of requests. And what it looks like when you're routing a message is basically you read data from a socket into a buffer. You stop when you've reached the amount of data that the message said it was going to contain. And then we tag that message, which is just in Erlang, sorry, in Elixir struct, with the PID of the route acceptor. So like where it came from. And then from there, we use a routing hash, which is a piece of data that the client sets that's like, hey, it's my account ID or it's my name to perform deterministic routing. Or randomly deterministic routing, because you don't really know where it's gonna go. And it sends it to a service protocol. So the client will say, or the lobby will say like, hey, I'm talking to the chat server. And it's like, all right, well, we're gonna send to this chat server, chat 34, right? And then on the other end, the service nodes receive a routed message. So the service receives that message in, it internally dispatches it to itself and figures out what code path it's supposed to go through. And then at the end, it checks to see if the message was transactional or not, and replies with a message as well. And this is just all in Erlang term, elixir terms. And it bounces the message back to the router that sent it, because we tagged it with the pit of the route acceptor. And that route acceptor serializes a binary version and sends it back to the client. And through this, we are also using a database backed by Postgres with Ecto.  It is so, so good. It's okay, you can clap for him, yeah. And Jose also, we'll get into it a little bit, helped a hell of a lot. Fixed some issues that we had going into production with Ecto. And it really helped us scale to a huge degree. So it's sharded by a schema. Postgres schemas are little buckets. They're inside a database node or an instance you have Postgres running. And you can have different versions of that exact same schema prefix with a name. So in this case, we pre-shared data into X buckets. In this case, it's 128 on X amount of nodes. And in this case, there's two nodes. The buckets are Postgres schemas and the nodes are Postgres instances. And each service node, or say that chat node in the left, services users that belong to shard 1 through 32 and only 1 through 32. And if we wanted to continue to spanned out, we'd just add another database server and move the shards over. Or the scheme is over to another database server and then have another chat server connect to that. So it's very easy to actually scale out. You don't take a full downtime when you use this model. Instead, you just say, we're going to take 64% of the population offer, 50% of the population offer something like that. So that's the high level overview. You don't need to know anything about games or the back end to learn from the rest of this talk. But now, once you've developed this, you're like, hey, how do I get this thing that works on my machine into production? So this is really how that adventure begins. Yeah, so some new people in the room probably that would have come from a scripting language or a non-compiled language, stop and do not deploy your source code to your node. That's like really if anyone has a security guy in your team and you said that he'd be at your front door waiting for you to come out to go to work and bug you the whole way, tell you to please don't do this. It's easy to do it and unfortunately a lot of the tutorials online are telling you to do this, but you don't want to. In the Erlang ecosystem, we do something instead called creating a release. This is still my least favorite part of Elixir and Erlang. It's just not really getting better. I know that Paul might be in the audience here who's Bitwalker on GitHub. I've never met him and he's creating EXRM and unfortunately that didn't exist when I started either, but that's making great strides as well. But creating a release was described to me as dark magic one time because there's like one guy, Sylvester Stallone as Harry Potter on your team. Yeah, I found that image. And when I learned how to first do an Erlang release, I think Relix existed. I can't remember because I was really stumbling through it. It was so painful.  And then there's Relix, which is a hell of a lot better. And then EXRM now, and like I said, EXRM is on GitHub. And it might be the way that a lot of people are using it right now. But it does a couple of things that I'm not a big fan of. So, and I had already created my own release process, but I don't actually believe that operations people should be managing my servers or my applications. I think that they should auto discover and you should use tools for that. So I don't really care that there's a human readable format. I didn't mind that there was an Erlang term. But overall, this is a really great way to get started. I'm gonna talk a little bit about how to do a release without it, just so people know how it works. Because it would be great if people understood and then helped with EXRM more, because I think that that is the right approach for the long term. EXRM basically just calls into Relix, which is a release tool. It made hell of a lot easier to create releases than RelTool ever did. And you can get it on GitHub there. But what's important to know about what's in a release is you can think of it like a manifest and all the files to be given to your VM on startup. It has all your compiled applications, all their compiled dependencies, some boot scripts, which is needed by the VM to know how to start the application. Control scripts to start, stop, restart, assist config, which is optional. VM args, which are optional. And then ERTS, which is actually the early runtime system, which is optional as well, you don't need to include that in the release, but you can. And the config file and the args are just the way you configure the VM or your applications that are contained inside the VM in the case that this is config. You can just really think of this as Mr. Fusion, the DeLorean. You can put anything in it, like my code, trash, same thing, doesn't matter, anything you want. And a release can contain many OTP applications. And this was the thing that really helped me understand. If you created one application, you can just create a release with that, plus the rest of your umbrella, and they all live in one single VM. So when we talk about applications, we don't really mean like a binary process that's running on your machine or Nginx or something like that. Instead, it's an application inside of the VM. And an OTP application is just a bunch of related code and processes. They're wrapped in an OTP behavior. You might have seen it if you ever typed use application. And it informs the VM on how to start and stop your application. It might be helpful to illustrate for anyone what that looks like in the mix file. Because the releases are all Erlang, and then you have the mix files, and you have the config file as well, like config.exs. And this was really hard to get for me to wrap my head around when I first started because, well, first it wasn't a config.exs when I started. But knowing what this mapped to when I was building the release really helped me. So here, this is just a library. It's not starting any applications. All it's saying is I depend on these things. I want logger, and I want crypto. This is the most basic. So if I was to create a release, this would do nothing basically. I'm not starting an application. I'd basically include this in something else. And this is what an OTP application would like. So very similar to the last, but we have this key value that says mod, and then the atom or the module name of the app we're gonna start. The thing that implements use application, and then the arguments we're gonna pass to it. Any registered processes that that thing's gonna start. If you do have any named registered processes, that's useful to fill out. Please do it if you're distributing this application to anybody or if you have a large organization, because if you name your registered process game, and I name my registered process game, they're gonna fight and not be able to be in the same VM. And then the end bit down there actually maps to sysconfig or default configuration for your application. And it's very, very important that you include, I'll go back one side. In the applications like logger, crypto, any of the things that you're depending on, you're expecting that library to be there. Just don't put it as a dependency. You also have to put it inside of your applications that to be started. Or when you create the release, you'll get this module not found. And you're like, what are you talking about? It worked on my machine. And then you go in the VM in production and you find out that the module or the application completely was not included. And that bit me and every single one of my coworkers multiple times. So this is what the mix.exe file looks like. When you wanna include a dependency from a project, I'm sorry, from a package manager. So in this case, discovery, which is an application that I created. And then I make sure it gets started and included by the release by putting it inside of that list of applications. So configuring the release, which should I use? They all do configuration. Well, the mix.exe, like I said, you can put environment configuration there and it's like default config stuff.  All right, well, forget what I said. The config.exe file. This is the file you would configure if you were not building releases. And instead, you're pushing your code out under your nodes. You just can't use it if you're building actual releases. It's not respected by the early VM in any way, unless somebody's created a way to feed it yet that I'm not aware of.  It's elixir syntax, and then there's the mix.exe file, which is another pathway that elixir exposes. Again, uses the elixir syntax. This is what I was talking about, about setting default configuration. And it's respected by release tooling to a degree, but it's only useful configuring your application. You can't configure dependencies with this thing. And then there's the sysconfig. And this is Erlang, like it just straight Erlang code. If you leave the period off at the end of your configuration file, it won't work. And that gets a lot of operations folk. And it gets a lot of people that are just familiar with elixir, and they don't know Erlang. And I have four guys in my team that know elixir very, very well. And they leave the period off of this every time. And I'm like, come on, we've had this chat four times. So it's just a little bit of the wart that I don't like still. So once we've built the release, we'll deploy it out. So let's imagine that you had a build server. I used Jenkins to build it.  It's like a tar file with your application. And well, I used GitHub for my artifact store. And then you have a configuration management tool, which is going to run on the nodes to deploy out the actual code and configure the sysconfig file and a couple of the small things. I used Chef and the hosting solution. You can use anything you like, but I used AWS. So once we get to this spot, we're going to realize that we extracted the code. It's on the node and we go to start the application. But remember, I had a database backed application and I'm connected to the database, but I didn't run any migrations yet. And if, okay, so let's go out to the machine and we could do this in the configuration management. But just to illustrate it easily, we'll run the Ecto migration. So we'll go to where I extracted the code and I'll try to run mix Ecto migrate. And then I'll realize that I can't because my application is in a release now. It doesn't look like it was on my desktop anymore. Mixed tasks just aren't available to me. And so unfortunately, mixed tasks aren't a valid path to running migrations once you're in production. Migrations, I believe, should be packaged separately and put on the database node. So that way, your database nodes don't have your application code or the compiled database application code. And a couple of small things about the mixed tasks are, Ecto migrate actually requires your application code and it requires to start the application. And that's rough if you're using automatic service discovery. Because all of a sudden my database node pops up and it's like, hey, I'm a chat server. What's up? And I'm like, you are not a chat server. And some bad happens, right? So unfortunately, I created, I mean, not unfortunately. I wish I didn't have to make this. But I made something called migrator. And basically it's just a small command line tool. Super easy to use, very small, very easy. All it does is includes Ecto and a couple of the small things. And it allows you to run database migrations without actually having to have your application on the node. And you can just run it with migrate up, path your migrations, and then the connection string to the database. And there's one neat little thing that's in here as well is Jose and I worked really closely. He did most of the work where I was like, hey, I need to have this multi-schema thing where I have 120 account databases and Ecto has no way right now to run those migrations. So this thing also includes something called multi-schema migration, which will allow you to migrate 128 or whatever scheme is at the same time for the same migrations. And if you are using Chef, I created a cookbook that does this for you. If you use Chef, include it and use that LWRP. So you might ask then, well, you get the scheme as migrated, how are you going to get the data out or in of the schemas? Because by default, Ecto thinks that you're just using one database. Well, Jose, again, helped out a lot, put this into Ecto for me, where you can place the prefix of the schema or of your query, which is the schema, into a model and then put it into the database. And if you get something out of the database, it tags it with that schema as well. So as you pass this struct around, it knows what schema it came out of. And that's really nice. And this is a helper function that I created. I think that this might appear sometime instead of you needing to make it. The API is a little buried, right? But those are the helper functions to put the prefix. And that's what it looks like when one of the other developers uses that at our company and just put the prefix and then insert it. OK, so I talked really quick about fully mesh clusters. And the red lines on the graph, that's actually the background of this image, were between the two routers. And in our infrastructure, those are only things that are fully meshed. What I mean by fully meshed is when one connects to the other and then the third connects to the first one, the second one auto connects to the third one as well. And the problem with this is that you can't get really any more than 100 to 150 Erlang nodes all connected together. And when I'm talking about supporting millions and millions of players and needing to scale out, and I have 128 possible chat servers, never alone the account servers, which is another 128, and then the routing servers. I just saw this as a potential bottleneck. So we wanted to avoid fully mesh clusters. And this is what it looks like. Basically, it's a dumpster on fire with wheels at scale. It's just, I mean, that's the line going everywhere. And I couldn't fit what the actual application would look like on this slide. But imagine, again, like 128 chat servers and then 100 routers or whatever. I didn't think I'd ever need more than 100 routers. It turned out I did not need more than two. So then we have this partially meshed idea, which is the routers are connected, red lines again. And then the routers just know about each chat server and catalog and account. And this is a subset of the service nodes. And it doesn't matter what they do. You can kind of figure it out from looking at it. The way you can achieve that is by using something called hidden nodes. Hidden nodes can be started up with the dash hidden flag. It can be passed on the command line, or it can be put into the VMARKS file. And what this does is when you connect the node, it doesn't try to auto discover its friends or create a connection to anybody else. So the routers know all about each other. But the chat servers have no idea where the other chat server is. So that means that if I'm on one chat server and I want to message somebody else, I have to bounce off of a router to the other chat server. So there's a little bit extra latency there. But it still allows us to scale much bigger than a fully mesh cluster would have allowed us to. And we, again, applied this to all the service nodes. And there's some downsides, like you can't use global processes if you do this. And by proxy, you can't use any libraries that leverage global processes. So if you're trying to use the distributed lock stuff inside of Erlang, you can't. But you could use something else for it like console or ETCD. So there's other ways to do it. And you have to manually connect the nodes together. So a nice thing would be if I had 80 nodes running and I connected one, it would just all find each other. And unfortunately now, when you go to connect them, you'd have to do it manually. Or you can use this thing that I built called Discovery. And Discovery's job is to automatically discover and connect Erlang nodes together. And there's a little bit of proprietary stuff that I'd really like to pull into this and make it more generalized and open source. But this is actually how we figure out where shards are. So when a chat server comes online, it tells the routers, by the way I support shards 1 through 10. And the routers will say, oh, absolutely. And then when messages come in, it figures it out. And when that node goes offline, the routers say, oh, shard 1 through 10 is temporarily off unavailable, so we can't service your request. I could give a pretty big talk about how to configure this and how to use it. But instead, I'll link you to the readme where it's pretty extensive, and we use this in production. And it's very useful. It sits on top of console, which is a Hashicorp tool. So once we deployed all of our stuff out, and I ran the database migrations, I started up my applications. And then I realized that nothing is connecting. I have a node here with a router, and I have a chat server here, and I ask them to talk, and they can't. It's because we haven't opened up the ports yet. And you could do a deep dive through the mailing list of 2002, like I did, or read the Erlang docs. But instead, I'll just tell you the couple ports you need. First is the Erlang portmapper daemon. By default, it runs on 4,369. You could configure it with an environment variable when you start the VM as well. And basically, the point of the EPMD is to keep track of what ports, what Erlang VMs are taking up on a local node. So if I'm connecting to somebody else's machine, and I say, I'm looking for the node named Jamie, it'll say, oh, it's at this port, and give it to me. So first, it makes a connection over this thing, figures out where it needs to go. And then it connects to the node on the port it gave me back. And that's a big range. It's actually a huge range if you don't configure it. And you can configure it with this kernel config. This is actually a sysconfig. That's why it's an Erlang syntax. But it's inet dist listen min and max, and you can set this range. And then all you'd want to do, if you're in a cloud, is go on and make sure that you've opened up this port range between any of the Erlang nodes that are going to need to talk to each other, and also with the EPMD port. And that's all you need to do. So once we had the application running, now's the time for the fun part, which is finding all the bottlenecks that I wrote. And there was plenty of them because if you're learning a new language, or you're using libraries that people provided, and it's got a great read me, but you haven't decided to look through the source code yet, you really don't know what's in there. And this is when the real fun part starts. So some general tips before we get into how you can find those are avoid registered processes.  You almost never need a registered process. The only time you'd need one is if you were storing some state that you want to get later on. They typically, in my infrastructure, are very minimal. And if you do use a registered process, it cannot be like a central path. At one point, I had a registered process that every single message that came into the system had to go through. And obviously, it's dumb. It was temporary. And I found it very first. But that's a time when you do not use registered processes. And then another one would be one for one supervision supervisor with a transient restart strategy. And what that means is when you start up a child, if it dies, it's OK as long as it ends with a normal. And you really need to think, do I need this? So for instance, the chat server, if the player is logged in and somehow a crash happens in his user process, yeah, I'd really like that to be restarted. But a time when you don't need this is the transaction manager, where the transaction, if it fails, I never want it to be restarted. So don't do it for that. That was actually a mistake that I made. And then the final tip is to use the observer. It is absolutely amazing, especially when you know what all the numbers mean. So you can start it. You open up an IEX console, and you just type this. If you've never done it, and you have your laptop opened, this is the only time that I'd be like, hey, please, open it and ignore me. But then quickly come back to me. This is what it'll look like. And this is just the landing screen. This is the basic system information. You can see the different tabs. And you can connect to remote nodes with this as well. This is the part about what I said Erlang or Elixir in production is brilliant. Connecting to C++ game servers with Visual Studio was pretty cool, too. But this gave me a ton of power about what was actually going on. I can actually see what messages are coming in and out of processes, and what their whole message queue looks like, and I'll just show you some of it. So one of the tabs is application. So if we look back, it's like the fourth tab. And applications will show you the entire application and its process tree. So this is what you would see on your laptop. I could have started up our entire internal application, but it's really, really big, and this is just simple enough. So this is what the logger looks like when it started. Registered processes are named with the Atom syntax, and then non-registered processes are just a PID. And you can also look at the process list if you press a tab on the right of it that says processes. And that gives you information like the PID, reductions, memory, and the message queue. I'll tell you about all those in a second. And this is what that looks like. The important ones are, again, applications, processes, reductions, memory, and message queue. And then if you clicked on any one of the processes in the process tree or that process list, you can get process information about it, which is what messages are in its queue, what it's waiting on. You can get information about the garbage collector. You can get information about how much memory it's using. You can look at the entire stack trace of the thing, how it was started. You can look at the state of the application. So this is actually neat to debug your application as well if you're building it and you're not understanding why your state's getting into a bad spot. This will help you out with that. And this is what that would look like. So the thing that you're aiming for when you're looking for bottlenecks is to have low reductions or reds and an empty message queue or MSG queue. The memory is important, but memory is so cheap at this point. Those two, though, are how I was finding every bottleneck in my application. And then the question everybody in the room is asking, I hope, is what are reductions? Because I had no idea. Not even a little bit of an idea what they were. And basically, reductions are a way that we figure out how busy a process is. Elixir processes are scheduled on a reduction count basis. And one reduction's roughly equivalent to a function call. A process is allowed to run until it pauses or until it has executed about 1,000 reductions. And this is in the normal priority queue, and we'll get to priority queues. You can pause either with receive or you can actually use a function called yield, which is part of the early standard library. And what yield does is it voluntarily allows you to stop and let another process have a chance. So you're probably not going to use this much. But if you ever get into a spot where you're thinking, I'm a hog, this process is a hog, I need to, and it can't afford to pause, yield is a much more efficient way than saying receive after one inside of Elixir. And the early scheduler uses preemptive scheduling, which basically ensures that at any given time the processor executes the highest priority task of all the tasks that are currently ready to execute. There's four queues in the early scheduler. And this is where the talk will get a little boring. I'm so sorry. There's beer right after it. This is the vegetables. I swear that it's going to be boring right now, especially if anyone's here is just like, I can't wait to make web applications. You'll thank me at some point for knowing how this works. So there's four priorities. There's max, which is reserved for internal use. Don't ever use it. There's high, normal, and low. And there's two priority profiles for these queues. There's strict and fair. Strict is what max and high are set to. The scheduler processes all the messages in the max queue first. All of them. Doesn't matter how many. And then the scheduler goes ahead and messages processes in the high queue. Then it'll go on and move on to the fair queues. There's a little bit of bluffing here, by the way, because if your runtime has SNP enabled, it does things in parallel. But this is a good general rule of thumb. And then there's fair, which is the normal and low priority. By default, any process you start is of normal priority. And the scheduler will process the normal queue until it's empty, or it's done about 8,000 reductions. And then it'll go ahead and move on and process one low if available. You should be careful with this, because you can get priority inversion if you're using the low queue, where low will actually somehow take priority over your normal messages if you have a lot of normal messages. Thanks to Ulf Weiger for explaining to me. That's probably not how to say his name, either. How to use how the scheduler works. And you can set priority yourself. So you can set a process flag that actually tells it. So at the top, I'm setting the priority to high. And then it returns back what the old priority was. And then I'm setting it back to normal. And just like anything in C++, you have to be very careful with it. You probably don't want to set the priority yourself. You might come into a point, and it's just really good to know that this is a tool available. But I can tell you that I have never had to set the priority. I've always thought I had to, and then said, no, I'm not going to do that. One of the reasons, here's an example of a bad time, when you're going to have a bad time if you set priority. Say you set high priority in a process, and it uses dynamic code loading. So it's going to talk to the code server to pull, we'll say protocols, because you didn't consolidate them. And what then happens is the code server is running at normal priority. So you won't cause a deadlock, but you'll cause a slowdown because you have this high priority message that needs to be done, but then it's depending on this normal priority process. So just be careful. And this is a great time then to talk about protocol consolidation. And this was one of the first major bottlenecks that we ran into, because if you recall earlier, I'm very excited about polymorphism in Elixir. And the protocols, if you don't consolidate them, so protocol and implementation, those are a construct in Elixir, if you don't consolidate them, every time you go to use one, it'll hit the code server and be like, hey, I'm using the enum protocol. Can I have that? Yeah, sure. And the code server is a registered process. And if you remember, I told you that registered processes are way shittier than you think they are. And if every single thing is touching the code server, then you have a massive bottleneck. And that's the code server running inside of the observer. Thank you. So protocol consolidation continued to figure out, if you think you've consolidated your protocols and you're in production, you can run protocol.consolidation, pass at the module, see if you actually did. If you didn't, you can look at the path and see, oh, maybe I didn't put it in the path when I started the Erlang VM, which is also a thing that I had done. Luckily, EXRM will handle this for you. Again, because I wasn't using it and I didn't have it available, I created this problem in a surprising number of ways where my protocols weren't available. So then I also ran into, why isn't Ecto starting? My workers at application start. And that was a pretty easy one. There's a flag in Ecto, lazy. You can set it false. And that'll make sure that the whole pool gets warmed up, give you a little bit of extra process boost. I also ran into, why is Ecto not committing data to the database? This was a really fun one. I had no idea how this happened. And I was able to suss this one out, again, by using the observer. I looked at the pool and then saw that it was some reason, a transaction pool, and the test pool, instead of post graphs. And then I realized it's because I turned on a brand new flag that was undocumented because I loved doing things like that, which was to disable build for environment because I don't believe in build for environment. I think that if you build an artifact, you should test it and then put that artifact in production. I don't want to build a test artifact and then put my test it and then build a production one, then put that one in production. So I turned that off. And then I immediately turned it right back on because some libraries like Ecto expect that feature to be on. Some application configurations are only available at build time, like the adapter in Ecto. So don't turn that off. And this was a big one. You really want one Erlang VM per machine, especially if you're on like AWS. The OS scheduler will get real mad at you. So I had one point, like eight Erlang VMs running. And it was a nightmare. That's top running. And basically, I don't know how many people in here have ever done sysab and stuff. But what you really want is for the amount of processes you have, we'll say we have four, my load average, I want to be at least maybe 3.75 or under. The first load average is for one minute. The next is for five minutes. And the one after it is for 15 minutes. And that's a number that you really should be paying attention to, along with your disk usage, if you are writing out to disk much. And then this is the last little bit that I'll leave you with that was something that I wish I learned more about. And it's the VM args. We got quite a significant performance boost by enabling kernel pull, which is just setting plus k to true, and setting up an async thread pool that was larger than the default. And you can just set that again in the VM args. A lot of this information I got directly from Jose, who I owe a hell of a lot to, and Eric. And then the rest of it was either reading Erlang mailing lists or a book that was recommended to me called Erlang in Anger. And I would very much recommend that once you guys are going into production. Read it. It's free. It's a PDF. And it is super, super helpful. So that's it. You can find me on GitHub or on Twitter. Thank you very much. Thank you very much to the AV guys for running the play.


Here's how to practice. Listen. Apartment. Record yourself. Apartment. Then listen. Apartment. Cook. Cook. Cook means to prepare food for eating, especially by using heat. Dinner. Dinner. Dinner is the main meal of the day. Excuse. Excuse. Excuse means to forgive someone for making a mistake or doing something wrong. Find. Find. Find means to get or discover something or someone that you are looking for. Here. Here. Here means in this place or at this location. Live. Live. Live means to have a home in a specified place. Near. Near. Near means close to something or someone. Sorry. Sorry. Sorry means feeling sorrow or regret. Supermarket. Supermarket. A supermarket is a store where customers can buy a variety of foods and household items. There. There. There means in that place or at that location. Want. Want. Want means to desire or wish for something. Wrong. Wrong. Wrong means not suitable or appropriate for a particular purpose, situation, or person. At work we answer with our name. Hello, this is John. Hello, is Anna there? Now you try it with your name. Ask for a person like this. If it is a wrong number, we say... I'm sorry, you have the wrong number. Now you try it.


I'm getting on my knees for this. I'm ready. I'm ready.  I saw this episode already when it released in theaters. It's been like a month or so since I've seen it I'm gonna do a rewatch right now. I have a lot of questions I'm not gonna fake my reaction like I usually do. This is gonna be real There's only one reason to watch this episode. The episode is good. Don't get me wrong You know seeing the demons and the demon with the 55 eyes That's all cool and the 55 tangerose that we saw on this episode. That was all cool But there's two reasons to watch this episode actually two is two This there's two reasons if you didn't see it in theaters. I feel so bad for you. Let's get to the titties I mean, let's get to the video. Oh My first time ever watching this. I don't know anything that's about to happen. I've never seen it I've seen no hush or a titties when I said I saw them. I met another hush who was titties Tengens a youth foldable production. Oh, this is a production.   I know what it will feel like to do a fake reaction I love how they tied this to the ending of the last episode of season two I love when they just continued from there.  Oh, I've never seen that in a theater. I almost came Do you see because I can't bro, I better see this guy fight I need to know more about him the soul blemish on my family. I can't believe what he said that oh I have so many questions this episode man.   I hate gotcha machines the dramatic cuts in this It's just next level bro.   Those are you foldable eyes. I Feel like I'm on the shinkansen. Why is this castle so big? There's like 11 of you Are they real estate agents? I write you foldable. We get it. You're involved That's the 10 time. I see your name on it It's rising and sneak it in on there demon slay the you foldable arc. It's a vase man.   Why does he look better? He animated then you have this guy looks like the Titan that's afraid of people This guy LGBTQ flag man, it's pride flag man didn't they show this guy for Gitaro and his sister Umi, I think her name is in a flashback.  I Just realized is that a hat or is that hair dye bro? You're not Jojo This hat like coming on him. Is that what it is? Well, I saw it as dude's face man I never saw it before I saw it in the theater. This is the first time I saw it I thought it was time to it was dad But now I think it's Tajro's grandpa.    What should I do? Look at this.  I feel like I'm in a real lab. Oh I love that quote. Can't you see you're falling in order of who has the most of their humanity remaining?  It's scripture. I Expect nothing expect nothing. You won't be disappointed. I Can't wait to get dislikes for a plotting CGI. I can't wait I can't wait for all those motherfuckers that dislike any demons layer video I upload because they don't like demons layer Oh, he's applauding CGI. He likes it because it's carried by animation.    I love that All right, we get it you photo what we get it.  He has such human qualities about him like look at that shit, man It looks like ten ten years plus tanger oh unless it's part of his family lineage the Sunbreathers Akaza, I love how he says that it's like the only way ask a lot cuz a dolphin.  That's next season.    This is awesome Look at this look at this these angles I'm curious what this guy's gonna be doing bro if he's this high in the ranks Does he cry?    Oh That's Tantra.  He didn't say it's his father. He said our guest But you could also call it like your family a guest like why are we making our guest? The babysit but he's very cryptic right there because they're leaving a questionnaire for the anime only saying Who is this guy? This is interesting man. I don't know They're being cryptic It's like that's him. That's the demon with the six eyes, right? But forgive me if I don't remember names, man. There's a lot of Bashi's Washi's There's a lot of names and I can't remember everything. Okay, so he's been there for a while You saved our lives and the sword he has That's important and he has the earrings too. I think that's not his parent.   Is that what he means or he's gonna pass down his name You have no hairs He was a charcoal like Tanjiro.   Hold on. Hold on Kokushibo. So his name's Yoshi He changed his name when he became a demon. My name is as worthless as I am That's why he changes it to um Kokushibo.  I didn't put that together when I saw this I think I just put it together. This guy is not related to Tanjiro I he's not the father of Tanjiro's dad He's just like an ancestor probably that didn't save his family So he's living his life and nothing but then he saves son Tanjiro and then he's saying his name means nothing So that's why it changes it becomes a demon. It makes a deal with the devil. Maybe he makes a deal with Muzan Oh my god, did I just put it together?  I think I just did bro.  I'm never gonna remember this guy's name cook Kokushibo He broke a vase it's a symbol for the Vasey's about the fight the vase Hashura the Vase demon Tanjiro is like Aaron bro. They just see visions from their past lives or their family. All he had to do was eat his dad Back to happiness no more infinity castles and 12 eyes and And a vase laughing with three mouths.  I'm becoming homicidal I'm going to Japan. I'm telling you bro. I bet it's he chanted up this season.  I could draw him. I'll say that my submission. I'm right here. I'm by demon sledge studios Yes, we need more characters like you that specify the rage. Good job, and I forgot your name already though Doesn't he look like a cat?  They got him in for one performance and left Oh, it's the three little girls at Tengen assaulted.  There's not a little racist.  I don't know I'm quoting Rengoku. You can't speak ill of the dead I'm glad they showed Where they're going and we're not gonna see them until the end of the season. I'm glad they did this give me some context He's better at the craziest for some known to man. This is great. This actually got a laugh out of me I Love it.  It's hard to make me laugh. Tom put your row I Love it All right, time to drop this hole.  It's time to fight that the demon Come on. I'm gonna come on soon man. Once I see the hot It's fine. It's done, bro. I Love these characters demon Slayer man.    I'm glad I thought he's going to die.  Goodbye.  Where is Chantaro?   Is that what it's supposed to be? That's what I'm gonna believe it. Chantaro has a little cameo Look at those CG towels.  Seriously, it's been so long since I've reacted to Demon Slayer man. It just feels it feels fun Why don't you go to the swordsmith village the next arc?    That's why it's 11 episodes long.   Come on. Come on. Yeah, patreon exclusive here, bro I ain't showing us on YouTube.  I'd pay it double triple quadruple cuz look at that. Oh my god. No, oh Break that bottle.  I love it, bro I feel so cool like I feel like a party.    They think you're part of the mafia. I Don't want to join this village.   I'm getting my knees for this. I'm ready. I'm ready Listen, you had to be at the movies to see this like to see those on a full-ass like 200 inch screen in like ultra for DHK DHD That's how it was meant to be seen.  I don't care It was too much money sell something sell your liver. It's worth it. You don't need your liver.  He could have killed Nesco just now That would have not been worse than what he just did I don't even know her name. I forgot her name.  It's getting dark in here.  Nicholas dark TV Why couldn't you take that off I wanted to see you from the angle. I saw the manga cover. I saw that but dunk-a-dunk This guy Bro, I forgot this guy existed. He showed up once in season two and he bumped the Tadro in a hole I didn't know that was foreshadowing him being in the next season. I Forgot all about him. Are you in the winch way? Whoa? I didn't remember seeing that Is he related to the wind hush? We could bond in our nakedness. I just dressed like a woman in the last arc I'm starting to see different sides of Don Jiro and I'm fine with that. I'm just saying bro He covers up the titties. He wants to bathe with another man of bonds He dresses like a woman in the last season. I don't know man. It's time. It's time to row Her voice is so soothing man. Soothe me some more keep talking whisper in my ear while you're at it I love Nesco's bond with her.   This wasn't fair Yeah, I'm gonna need oh my lord. I need him. I need to find a local temple after this and clean cleanse myself I'm going to the shrine after this man. I've sinned There's a secret weapon in his vision that will make you stronger. I would have turned into a Come drop in this moment.  Oh He grew a new sword right there. We're on a mission to get this secret Sauce, I don't know. It's not a sword, bro. It's gonna be like barbecue sauce. I'm telling you here. We go Tondro seen the Newton did the blue asra.   It is him he didn't age so it's two different people. I don't know man. See y'all next week