What's good everybody Chris from jassescasebreak.com doing our 22-23 pre-meet origins basketball six box half case break random team number two Had a couple fillers for this one.  We have two combos. They have the Brooklyn Nets and the Phoenix Suns and the Los Angeles Crippers and the New York Knicks so what I have here are fresh dice our list here from Jim Way Down to Nancy our last spot mojo and then two fillers mosaic 368 and mosaic 369 I took the liberty of combining the Nets and Suns into one spot right here Alphabetically and then the Clippers and Knicks into one spot alphabetically so it's 28 spots for 28 people Let's roll both dice here. Thanks again y'all good. Look Two and one three times for each. Let's do it one two three times 28 names from Brian down to Mark and Three times for our teams. Good luck one two and three third and final time Quoven Cavaliers down to the T-Wolves. We're playing right now this very moment Let's get this format it up and we'll we'll discuss some trades for the random team so Brian Leads off our first half here with the Quoven Cavaliers Eric with the Utah Jazz Brian with the Raptors Chad with the Bulls Brian with the Spurs Justin with the Hornets Jason K grabs the next N Suns combo spot calm with the magic nice team there Ryan with the Lakers Jim Way with the Grizzlies Jeremy with the Pelicans Ryan with the Bucks Brian F With the Sixers Nancy the last one mojo gets the pistons There's two rookies there Matt J with the Pacers our second half year Dan gets the Dallas Mavericks Brian gets the Quippers and Knicks combo spots Brian W Chris T with the Celtics Chad with the Hawks Carlos with the Kings Josh Smith with the Blazers Eric with the Rockets Nancy with the Nuggets Colin with the Warriors Chad with the Miami Heat Tyler with the Wizards Nico with the Thunder and Mark with the T-Wolves All right We're gonna alphabetize this and we'll keep the trade window open for a little bit when we come back after this pause We'll see if there's anything going on, but if not, we'll rip this six box half case break of the Oranges grab back all right, we're back one offer, but Nothing too crazy. So we're gonna just rock and roll with our six box six box half case break of Oranges Pascal the twos on the box indicate that this is you know random team to Thanks again everyone for any straight-up spots and you filler spots We appreciate you. It's a fun one to do. Let's see last six boxes of this case 19 to our combo spot right there Second comma spot in the middle. All right Let's do it. Thanks again. Y'all good luck redemption here at the bottom Originals Jane Hardy. Okay, Coney hand Benedict Mather and rookie and he had a paces and Matt Johnson First number car gonna be Dyson Daniels 62 out of 75 Rookie for the New Orleans Pelicans going to Jeremy first hit Nicola Jovic Rookie for the heat. That's going to Chad Cromwell Cole Jovic non-numbered and the redemption Jalen Brown had a 41 point game and it will be our hider here rookie autographs It's card number 20. Good luck everyone B Benedict Mather and nice rookie auto for the Indiana Pacers one more going to Matt Johnson Having a nice rookie season. Nice start to the break But it had math or an Indiana Pacers box one down Jake Loviam originals Tatum Steph Curry Killed in Johnson 3 out of 49 San Antonio Spurs. That's going to Brian W first hit Christian Coloco 34 out of 49 rookie for the Toronto Raptors and Ryan next all are going to be Kennedy Chandler on card red 66 out of 99 rookie auto for the Memphis Grizzlies and Jim Way All right box three What a nice RPA in here. Yeah, I think they they save the The nicer patches for the numbers though, but yeah We love like an actual nice patch team origins Donovan Mitchell Sorry, Eastern rookie for the Rockets folks by Agbaji rookie. Here's a Paul Bank arrow None of a little foil for the Orlando magic and calling 32 out of 99 blue Nicola Yovitch One more for the heat and Chad Another one Benedict Matherin patch non-numbered for the Pacers and One more Nicola Yovitch rookie auto this time non-numbered for Chad Chad Cromwell and the heat pick up the Nicola Yovitch second-rounder Just like the Joker see if he can develop and I'm not gonna say two time back-to-back MVP maybe a third but never know Never know Box for Let's see up here for the second half of this Six box half case break another redemption here Had a dawn Paul Bank arrow insert one more for calling in the magic Branham Kyrie Irving and Atari Eastern 87 out of 99 Houston Rockets rookie going to Eric Houston that redemption for last Next up nice Shane Sharp 12 out of 99 on the rookie auto Portland Trailblazers that's going to Josh man. He's the top 10 pick Shane Sharp out of 99 on the rookie auto on card Kyrie We'll hide our redemption. Good luck y'all Elevation signatures should be number to 99 card 30 E. J. Liddell E. J. Liddell elevation E. J. Liddell. Yeah, New Orleans pelicans For the rookie Elevation signatures again should be number to 99 New Orleans pelicans and Jeremy New Orleans pelicans box five Tari eastern originals to our hero can a Chandler Christian would All George 46 out of 99 for the Clippers and Nick's combo spot. It's going to go to Brian W first hit Christian brawn on card auto rookie for the Denver Nuggets. That's going to go to Nancy My number and the next one is a Musa Diabate on card rookie auto for the Clippers and Nick's combo spot one more time One more for Brian. All right, and the last box. Thanks again. Y'all see if we need a nice RPA out of this one Last box from this case. Good luck. I know it took a second, but thanks again everyone for getting this one full Shadow spots and fillers two originals Yannis June Suggs Zion Andrew Wiggins June Suggs out of 75 34 out of 75 second year One more for the Magic and Colin Universal the Hall of Famer John Stockton your all-time assistant leader Utah Jazz and Eric Got that one from a filler to non numbered on the John Stockton on card auto And the last one is a Trevor Keele's five out of 35 on the on card lucky auto What's up? I'm okay. Thank you Trevor Keele's five out of 35 rookie auto for the Clippers and Nick's combo spot. All right, and that's it. Thanks again, y'all That was origin's basketball six box half case for a grand team number two Thanks again friggin at all filled out We'll be back with more breaks soon more basketball football baseball F1 UFC All on the site right now Jaspies case breaks calm. Thanks again. Bye. Bye


Now in regards to your talents in this game it isn't a matter of how it's more of a when, when do you unlock your talents so you can really and truly customise your witch or wizard, well today I have you covered and I will explain exactly when you get these. Okay so talents to the player are very very important, but well they ain't something you get real early on it's actually quite a grind you have to do to actually get these. Now I ain't going to make this video 15 minutes long I'm going to keep it short and sweet but I will explain to you how and when you get these talents. So there is a quest around 17 quests into the game apart that main quest line and it's called the Jackdaw's Rest. Now this is a quest which like I said is quite far into the game but if you're wondering where this is and how far you've got to actually get to here you'll see the quest on the screen now from the main quest from the prologue all the way to this quest here in Jackdaw's Rest. So when you get this quest done guys it's Jackdaw's Rest this is when you unlock those talents. I had myself so confused I was wondering what have I missed is there a side quest somewhere is there somebody I've missed is there an NPC I've missed how do I get my talents well this is how you do it. So yes I sure but sweet one people but this is how and when you unlock those talents so you can truly start customizing your wizard but yeah description of what these talents are and what they do you can see on the screen now I read it for you you've earned talent points plus one starting at level five you may spend these points these talent points as you wish to enhance your spells to increase your combat powers style for more. Choose wisely talent points may only be spent once so the number of talents you acquire will be limited so yes whatever level you are when you get to this point in the game when you complete this quest line you'll get one talent point per level after a level five to spend within any of these five categories and it really is as simple as that people so yes this is when and how you unlock those talent points to truly start customizing your player and there we have it people guys if you enjoyed the video leave a like really helps out feel like what you're seeing once you more hug what's on a daily basis close daily guides be sure to subscribe and hopefully guys I'll see you on that next one


bly important and just knowing that there's a continual feedback mechanism appearing from students about what they like what they don't like and what can be improved same thing from staff but also looking at the diversity of menu items and so the richness in terms of vegetarian local but also just different cultures being touched on in terms of their food it may sound dumb but when I saw tofu stir fry on there I was like personally kind of excited to see tofu on and it's more cultural than it is vegetarian for me but it was neat and it's like see look at me you're like chicken biryani and you know teriyaki stir fry with tofu and you know tacos al pastor I mean Jamaican jerk coming up Jamaican jerk coming up I mean but I think that's actually the kind of stuff that's just it's incredibly exciting I'm hoping that as students are I hope students like that a lot of stuff and we are working with Chef Sam just to also do training for the cooks to make sure that these menus are these items are these recipes are included in the menu and it's done the right way because it doesn't make sense we have this fancy recipe that's getting the simple methods across and have it done properly I mean this is something I already know we've heard tonight but I think I came on the committee when the working group that Mr. Mangana was working on was was coming into a landing on some of its work and starting to present its initial results this has to be one of the topics of greatest enthusiasm greatest care and greatest concern that I could even think of and it's one that has also the capability of unlike most of the other things we talk about we care deeply about of instead of being like joyless and sort of semi-negative to actually being utterly joyful and what we're really doing and what the end result is for kids and for the staff and for our community and so it's just you know we want you I mean we wanted this presentation in like March and we wanted excitedly we wanted it in July and we waited until September and we were excited I mean I just I'm saying this personally because I don't mean it's as affirmative and as affirming a way as I can like we're all like every day we're right behind you and we know so many members of the community are too just bring it in I want to make clear my original comment is only to support you you're doing a remarkable within the existing constraints but why does it have to be a self-sustaining program why do we want to eliminate the deficit we don't take low enrollment classes and say they don't pay their way and don't offer low enrollment classes why does lunch have to support itself you can keep $3 but increase the general funding just a suggestion you know it's over I just want to thank Mr. Mengana Mr. Hart and I'm sorry I don't remember your last name Palmer for the hard work you've done especially on the the summer programs where you've increased it to 10 sites and I just want to say I think we missed an opportunity this summer to get the word out because when I saw the article about the baby burp truck serving teenagers which was excellent and it was moving around I was yelling at the article you know I haven't been part of the visioning group I was yelling I was like what about the schools I know it's like we were serving 10 10 different locations and I think we missed that but we did work somewhat in partnership with the Mass there was communication between the Mass and the schools but we ran an article an article with them we should have next year so how do you thank you how do you like the food oh it's super and I believe that as the program can use to roll out that as more middle schoolers are introduced to that food and finally do get to the high school they'll continue to eat the school food instead of bringing their lunches that's fascinating over the summer when I was talking to Mr. Mangana and Mr. Harb they were hypothesizing around these trends and they said by the time you get to high school I remember you guys said this by the time you get you're already in your junior senior year you're sort of locked into a pattern of how you're eating and I remember you both said you were hoping what you said so hope you're all three right is there anything further I mean so we want to let you go and then move on with our agenda we will do that thank you so much for coming here and as we said on to more cheerful topics the FY18 fourth quarter budget update that's I'm teasing you slowly Sean but misleadingly in every category I said oh my goodness so under budget and projection isn't this amazing until you get to the end and it's like oh yeah that's right and then there's this $148,000 overrun in healthcare and benefits yeah yeah that's right everything adds up to zero I gave away the head I'm sorry Sean so the budget actually ended pretty well considering what we dealt with last year despite the large increase in health insurance really saved us as the out of issue tuition and the students fewer students going out and some students coming back really saved the region last year given the hour I probably won't spend I won't read it my study wants me to I'm going to focus mainly on the pictures from the charts and the graphs because I think they're informative that's what my mind can handle it yeah so I will read the headlines they came to Amherst Congress just published a budget under 2.16% under budget due to savings and payroll tuition so that's sort of the main takeaway we're on the second page so one area that we just we need to keep some closer tabs I think moving forward is our substitute costs I see those have generally been growing over the past few years it's also one of those things that has sort of impacted by when the wage increases going on because we try to keep our self-perience at least so during the budget season it's something we may see an adjustment being made to reflect our actual costs but it's not really discretionary and it's as needed for the subs on the next page you'll also see negative trends are contracted special transportation costs so those have grown from about 50,000 in 2012 to about 270,000 in 2018 you may remember signing a voting a three-year contract with VanPool which is designed to help kids those costs down a little bit and they will but we are having on-kits by the district that require transportation and some of them are going far away so the transportation costs are pretty significant so that's another trend that we're going to keep track of as we go forward one of the unfortunate things is we don't get circuit breaker reimbursement for special transportation which is a huge cost for it regional transportation reimbursement circuit breaker reimbursement for those costs the good news is the chart like below it so it's sort of broken into three phases here so the first one is vocational tuition on the district so the first bar is what we have for at West 17 actual students going out of the district and the next is what we budgeted for at West 18 based on that actual and then the third is what we actually had in that fight so you can see our from what we have at West 17 that we budgeted for at West 18 same thing for Charter you can see our actual Charter that went down from at West 17 actual and it reversed a trend of many years going out like 10 to 12 kids at a time and our at West 17 choice at the district went down as well so across the board all sort of the non-special on the tuition programs we saw students up on the lunch participation presentation on lunch participation and then the last thing so our revenues our revenues came in a little bit under budget meaning we thought we would get more revenue the main reason that they came under that was due to the Charter tuition coming in under so our Charter tuition revenue based on our out of the out of the out of the out of the out of the out of the out of the out of our где out of the out of the out of the kind of kind of out of the out of the student and out of the and we have So that's sort of the highlights of the FY18 Q4 budget update. Happy to answer any questions or revisit any topics that you skipped over. Ms. Creml? First of all, excellent job of setting all your y-axis to zero. Thank you. I still remember that from Tommy. Thank you. Right. So one short question, and then we'll make one more general one. Maybe I'll just start reading it right, and I probably should give the number of presentations you've done on this. So the 569th left over at the end. How does that relate to the first number of 940, 950, essentially? I'm basically trying to understand how they all connect. Well, big picture like, oh, what did we save? And then of what we saved, where is it going? So if you go to the first page of 949, we don't really, so that's after you take up 280. If you look right above that number, you see the 280. So that 280 is a contingency, basically the school committee votes every year. We never touch it, so it comes out of BND. We have it in the budget just in case there's an emergency, and then we don't use it, and it goes back into BND. So ignore that number and look at the one right above it, which is the 669. That's really what we came in under budget from operations for the year. And so we came in under budget by 669, but our revenues were short by 100,000. So the sum of those two numbers is the 569, and what goes back into BND. Do we ever use that emergency? How often does that occur? We haven't used this since I've been here, and the main reason we don't use this is because if you use it, then it's basically taken directly from BND because we're not putting it back in at the end of the year. But we have it used, so we almost used it when we had the gym issue, the flood, but we were able to figure it out during the year without tapping into that. But basically it's just 280 coming straight out of BND that if we didn't have it there and there was an emergency, we'd have to revisit the whole budget, essentially, and have it revoked. Can you follow something on that? Since you said we voted to take 500,000 out of BND with this fiscal year to use, and we ended up putting in 569 in BND, which means there's a net plus of 69. What that essentially means is the savings that were sort of enforced last year because we knew we were in bad times and we have managed to catch us up to the point that some of the more challenging things we had to do for this year, like taking half a million out of BND, we're essentially able to net out and make ourselves whole for that decision. At least for FY18. Well, I mean forever. Yeah, I mean, the main reason we were able to come out to do it for FY18 is because we did restrict the budget about halfway through the year because we didn't know where the health insurance premiums were coming from. So the tuition, so the restricted budget, we have some staff turnover savings, I don't know if part of this is in a good place. We wanted to come in, but at least what we were using for FY19. So do you think the answer to what you're saying is yes? Well, the reason I'm saying that just is because a reasonable person could sit back and say, last year we ended up having to make a lot of emergency decisions to cut back spending that was unpalatable and undesirable. We also had to make cuts for this coming fiscal year. You send then see a half million dollars sitting around and a reasonable person could say, wouldn't it be great to have spent it? And you could make that case perfectly fine to make that case. But an alternative case to be made would be that since we don't know what could possibly pop up this year that would put our budget at risk, we would at least put ourselves on a better or more even keel. Yeah, absolutely. And again, for immediate future challenges. Yeah, absolutely. And if we did spend it and we still took the $500,000 off the EPLINICAN budget then our E&D fund just goes down about $500,000 and that's essentially half of our E&D fund. And so, what we pull out of E&D every year is not necessarily recurring revenue, right? It's basically, if we had my left over there before, it's what we're applying to the next year's budget. So that's why we're always careful about whatever we vote for EPLINICAN the following year if we want to put back at the end of this year because if not, we're basically just spending our reserves. If you're able to. Yeah. If we had been there, then we wouldn't have been able to. I mean, we knew when we were taking the vote this past spring that if there was a lousy enough budget to have to vote that if we were having to cut the margins of E&D a little bit, we were just doing that. I mean, I'm not saying it wouldn't have been anything back, but instead of being 569, it could have been 360. Yeah, to some extent. But also, when we present the budget, we have to make an assumption over what E&D number we use to meet the budget and essentially that could lead to more cuts or less cuts. So when we set that number, we look at what we think we're going to end up with at the end of the year. So the 500,000 sort of the ballpark that we thought we might end up with that. Although there was a lot of uncertainty last year with the health insurance. We talked about it. I mean, I started to believe in this. I apologize. There's no microphone. I'm grabbing the microphone. But I mean, we literally talked about this at the end of the budget type. Like, well, what's the level of uncertainty and the numbers coming in worse than they were? And what you're reporting is that they came in fine relative to those expectations. Yeah, because of the tuition. Yeah, started the labor, for instance.  Basically, last year with the budget, it's not an important topic. Anyone else? So any early signs that health insurance is going to be lower than ever again? So we're in the Maya trust now. So the good news is that we know our rates for this year. They're not going to change. Unless there's something that happens that's never happened in Maya's history, which is the rates changing during their year. So for this year, we're set. For the next year, we know our increase is going to be no worse than the average increase in Maya. So when they do the increases, they get a low, a medium, and a high, basically, and they band communities together based on their experience. And so we're going to be no worse than the average. It's going to be the promises they made to the group when we sign up with Maya. So past history, that's like around 4% or 5%, potentially, which is much better than what we had prior year. So we're really going to know about health insurance this year's enrollment.  So I'll have them update for the first quarter budget. But we have seen, which is sort of good and bad, we've seen a pretty large increase in opt-outs, which, you know, it's unfortunate that people are choosing to go to other insurance because either better or the cost was too high here. But financially, for the district, every time there's an opt-out, we save about, depending upon our single plan, somewhere between 3,000 and 4,000 to as much as 13,000 per opt-out. So we've seen the opt-out numbers go up. Thank you. Anything else? Yes, so because we put a freeze on expenses last year to try to mitigate some of our issues, what was the impact of that and our budget this year is even tighter? So if we put off buying things that we really needed, it's not really in the budget this year. How is that kind of lining up for things that our school is needing? There's a lot of early spending, I think, as a result of that. I think as soon as the budget turned over, they were like, let's get our businesses in and pull up something happen. So I mean, the major impact is that we're going to see a lot more earlier spending this year as a result to make up the last six months where things were restricted. There wasn't no spending. Critical things were purchased, but a lot of the nice-to-haves are things that can be put off or put off. I just had a follow-up question on this. You mentioned the subsidy cost for one area concerned. So I'm sure part of it is the total wages going up and other pieces, improved leaves of absences. So I guess I'm just curious, is that anything that we have control over or the district has control over, or is that something that... I guess just if we were to address this, what would be your recommendations or what should we know about the people and factors behind these? Yeah, and most of these costs... We sort of have two sub-buckets. This sub-bucket is sick, personal, time, primarily. So in terms of approved, it's really things that are not up for approval. Essentially, there's a lot of spikes here now because there's a lot of long-term leaves that year due to medical reasons, and the sub-costs are higher that year. We have a separate bucket, which is a much smaller bucket for professional development leaves, subs for those professional development leaves. That we have more control over with the budget piece of how much we want to set that out. This bucket, which would be seen as a trend, is more sickness in general or more long-term leaves due to medical things during the year. Except budget, right? More wellness programs than some of the medical ones. Eat the healthy school lunch. Eat the healthiest school lunch. I was just on that comment because I think it is kind of interesting. Wellness programs, and I think there was definitely research coming back about the investment that is made by employers around the country in all sorts of different sectors to help employees feel better and just get over illness more quickly, take better care of themselves, all those kinds of things, which of course benefits their morale and benefits their health, which is a plus for everyone, including students, and then also has the added benefit of our bottom line. So maybe something to think about and just explore in the year coming up because I do remember during the negotiations, I was talking a little bit about some of the substitutes cost-related to this. And to me this doesn't feel long. I think it's not like you're seeing this gigantic spike, but we're definitely seeing a trend that seems to be moving up. So maybe there's something there that we can address to help our employees feel better and cut some of these costs. In my presentations I've been at, mine has a pretty good wellness program. I'm getting a survey right now that I fill out twice a month. I can talk about a gift card, a healthy survey, like you know, like you would miss out on your health or something. So they have a lot of little things like that and discounts for the healthy buying. So they have a pretty good wellness program. So I think they know that for them, it saves them money too. I remember when I was 19 seconds and I was in health insurance, it saves them a lot of money. So I think that's something we'll look into more and how to communicate the wellness program for employees. Just to follow up on that, I think in California, where I lived for a while, Kaiser Permanente has this enormous program with all of their staff, but also their members. The focus really is on reducing costs and improving people's overall health. So there's a lot of really great models, whether it's Maya or others, that we could potentially be using to help us do that. Thank you. And I think we've lost our computer AV capabilities. I don't know if you don't need it, right? I don't need it for this one. Okay, that's all right. All right. Thank you very much. Charter Ex-School Expansion on Quest Discussion. It seems perennial, like whack-a-mole, but we're at it again, so it means we've got to get active again. It's relevant. Yes, so the broken record item that never ends. So as you all know, once again, the Charter School in Hadley is proposing to expand and the same process as last year, if you weren't here. Last year for it, basically the commissioner accepts input up to a certain date. The published date is going to be the deadline and note earlier than November 1st. The superintendent mentioned a couple of days ago that he expects that date to hold. So it should be November 1st, where the committee can send letters, the public can send emails to CharterSchools at doe.mass.edu, which is available on places. Commissioner makes a decision. If it's approved, it goes to the board for a final vote. If it's approved, then we would want to ask the board to overturn that ruling, which has happened in the past. And if the commissioner does not approve, the school is very likely to appeal, which it has done the last couple of years. And then we go to the board and make the case again. So I think our standard path is pretty straightforward in terms of we should write the same letter last year, which I think should be fairly similar to last year. It's essentially updating the data, because the arguments haven't really changed, because the proposal hasn't really changed. Except to say that, and we all got this very long document of the proposal to say that the way that it's characterized this year is particularly, I don't know what the word is for us. I'll just read a couple of sentences off the first page here. It says, the PVCICS Board of Trustees wishes to make this change, to satisfy unmet demand for kindergarten seats, expand opportunity and choice for urban, suburban, and rural students to attend desegregated, integrated public schools, and increase the integration of staffing and public education. PVCICS has proven to be an engine of integration in Hampshire County. So, you know, when I read that, I was like, oh, well, I should go and look at the updated data, because they're doing this constant accusation against our board and other boards that we are producing an opposition mythology that we are somehow misconstruing what the data is. So, the Department of Ed has their own enrollment numbers for all sorts of different groups, and it looks at all the high-need, achievement gap subgroups, low-income, Hispanic, et cetera. And so, when you look at all of these groups and you compare the school's enrollment to the region of towns that it serves, they are woefully under-enrolled in nearly every one of these categories. You know, just one example, their surface region is 35% Hispanic, they're 5.5%. The surface region is 20% special ed. They're at 5.9%. In fact, if you look at all 197 individual schools in the 31 towns, they're 196th in percent of special ed students. It just, it's, you can't state the point for the same thing with low-income, high-needs, ELL. So, why do I bring that up? I bring that up because to sort of relate it to one of our earlier agenda items that the MCA president was bringing up, you know, this is an attack on public schools. It's completely erroneous, juxtaposed with the data that the Department of Ed provides, and it's something that's worth, I think, pushing strongly against back. And in this year in which a lot of different towns are going to be coming together to advocate for state funding of education, certainly the charter drain is a big part of that. And so, I've already heard from other towns and we've seen some things in the media of people pretty angry at not just the repeated request, but the nature of the accusation. And so, yes, we should do our letter, but I think we should sort of keep our eyes out this month because it's going to be a fairly contracted time period and it's 11-1 is the deadline for opportunities to stand together with other schools and the MTA and other groups that are advocating on this issue. So, I think our next step is probably to draft a letter and our October meeting review it. Vote on it. Yeah, vote on it. That'll be our only opportunity. So we have to review and vote, so we'll have to get the draft in advance of it. I think the other thing is, because this is a regional committee meeting, the representatives here of all the individual elementary committees or town committees, and I think what we'd love to see is for you to go back to your regular town committee. And if you weren't, or maybe you were to be doing this and if you are, then this meeting with people will be saying, that's awesome, they're so happy to know that. And if you haven't gotten around to it yet, knowing this November 1st deadline and it's thinking of your calendar to try to take a look at providing a letter of your own on the subject would be really, really critical, because the classic mistake here could be, it turns out your committee meeting just happens to meet at a time that makes it inconvenient to draft and approve a letter. And so now's the time to look. And then I think beyond that, one of the things that we'd love to see is if we can reach out to our local school committees and finance committees as well to possibly also provide some formal support from each of our towns. I think beyond that, by the way, I apologize, we talked about this on Tuesday at an Amherst School Committee meeting. So if it sounds like what I'm saying sounds even more declarative, it's because we were part of a longer other discussion and I forgive the regional members for the fact that I'm doing that. But the other thing too is the extent that I think that we take accountability for our own local committees, like boards and finance committees, that's extremely helpful. And then I assume Mr. Demling and those Adonis are going to keep at outreach to other committees in the area. And if you need any help... I'm seeing four thumbs up. No, no, no. If you need any help, well, I can help, we can help. But I'm assuming you were doing that before, so I'm continuing to do that. Anyway, so that's where we're at on that. It's... And actually it's funny because we were talking about this on Tuesday and this approval will come after the gubernatorial re-election, which we're to be blunt. I think the point of the board turning over is a good one that was made most of Demling in a previous meeting. But I think the fact that this is coming after the gubernatorial election is even more pointed because I could imagine putting a lens on decision-making around public and community opinion that could be potentially different after November 6th than it is before November 6th. I don't know if that's true or not, but I don't want to take that chance at all. So, cool. Anything else on this? Yes. Just one note. I don't know if it's valuable to add to your letter, but I thought that the other noted difference when I read their application was that this time not only are they maybe asking for approval to expand, but that, well, if not, you need to give us a number of students for which you are willing to let us expand to, which I don't think is the job of that board. That would just be another... The other thing that makes crafting the tone somewhat challenging, and I remember Ms. Nakajima bringing up this point, is that part of it is kind of reminding the board of Ed what they've said in the past. Yeah. And so it's... And in one sense, it seems a little bit brow-beating to do that. On the other sense, you know, they have... Commissioner of Ed has a billion things he's paying attention to. He's like, what is it? Charter school, and so he's relying on the Charter School Office that relies on there. And so we sort of have to say, well, we've been paying attention to the history. Copy, paste, copy, paste. Here's what we've told them to do the last two years. You told them to go away for two or three years and they haven't. I mean, the funny part is toneally is just, you know, sort of just being like matter of fact. Like we've been reviewing your responses and we noted... Because we did the same thing. Anyways, whatever. Good. Okay. We have our to-do list. Accept gifts. I think we have gifts. We have gifts. We have many gifts. This is going to be a chore. Volunteers? It's a happy chore. Of course it's a happy chore. So then this... Thank you. So I move to accept the gifts to the Interstellar Regional School Committee as follows. Valley Gift Community Foundation of Western Massachusetts. Check number 227-8524 to support the athletic program in the total of $2,456.31. Check from Margaret Lefter and Margo Mace. Check number 11664 in memory of Philip Mitchell-Ted Madden Scholarship in the amount of $20. Donor of Pamela Moser. Check number 2193 in memory of Philip Mitchell-Ted Madden Scholarship, $25. Donor Terry Ray of Margo Mace. Number 13557 in memory of Philip Mitchell-Ted Madden Scholarship in the amount of $100. Donor Thomas Lafler Jr. Check number 1800 in memory of Philip Mitchell-Ted Madden Scholarship in the amount of $50. Donor class of 2018. Check number 6409 to support the class of 2018 for class of 2019 scholarship in the amount of $500. Donor James Marjorie Michael Christopher Abel to support LaCrosse fees for in need students in memory of Judy Brooks in the amount of $250. Donor Drawer Shirley Schmerling and the check number is 3285 to support the Ames program in the amount of $500. Donor Patricia Bodie and Mark Moriarty in the check is 77929 in memory of Ryan at Amherst Regional High School Art & Library Split $3,000 each in a total amount of $6,000. Donor Amherst Education Foundation check number 1529 to support the Amherst Regional High School Light Board performing arts in the amount of $3,347.50 Donor Amherst Education Foundation check number 1529 to support the Amherst Regional High School Restorative Justice Team to improve relationships in the amount of $5,000. Donor Amherst Education Foundation check number 1529 to support coding robotics course for all eighth grade students in the amount of $4,000. Donor Society for Science and the Public check number 110559 to support 2018 Regeneron Science Award OMACS microscope purchase in the amount of $2,000. And Donor the William Penn Foundation check number 45141 to support girls ultimate team donation May 30th, 2018 $250 two-to-one matching gift in the total amount of $500 for a total of $24,748.81 Is there a second? Oh, second. We've moved in a second. Well, she did all the work. You know, we're speaking to a second. They were odd. Is there any discussion? I mean, this is obviously amazing. Well spoken, Mr. Donor. I meant the gifts, but she was great too. Sorry. Anything else? Seeing none. All those in favor of the motion is read. Carries unanimously. Thank you very much. That really is remarkable. Very remarkable. I don't remember a list this long last year. So this is really something else. The last item we have is school committee planning upcoming topics. And unfortunately, I don't think we have the list of upcoming topics that we just take an action to email out the list to the regional group and then we can... Yeah, why don't we do that? And then why don't you... I was kind of... I was getting... I was reading the room that people would like to get that email to them and then they would like to... If they wanted to add something. And then they would like to go home. I mean, I'm not sure I'd stop anyone if anyone... Yes, Mr. Dunlop. Just to add the charter letter and if the committee would like a recreational pot for next... Yeah, okay. Okay, great. Is there a motion to adjourn? Go ahead. Let's adjourn. Is there a second? I'll second. All those in favor? Carries unanimously. We are adjourned. Thank you. Good you.


Hey everyone, welcome to SuperCloud 5, the battle for AI supremacy. I'm your host, Lisa Martin. Thrilled to welcome back one of our many time alumni to the program, Clint Sharpe is here, the CEO and co-founder of Cribble. Clint, great to see you. Thanks for coming on the program today. Yeah, thank you so much for having me, Lisa. So since we last spoke, Cribble hit a pretty big milestone. 100 million in ARR, so passing that actually. Talk about that as a milestone for the organization and some of the key lessons that you've learned along the way. Yeah, first of all, I mean, revenue is a reflection of the value that we provide to customers. And so, first and foremost, it's due to our customers and the great value that we're providing to them. We're one of the fastest growing companies in infrastructure software for that reason. It's faster than just about everybody else in the business, fourth fastest in infrastructure software behind Wiz, Hashie and Snowflake and just slightly ahead of Datadog. So it's great company to be in. Part of that reason is also our net dollar retention is 145%. That means that our customers are not only buying but they're growing and they're growing really quickly. And one third of the Fortune 100 are Cribble customers. And so we're very, very proud of what value we've been able to provide to them. I think part of the reason for that is we're also expanding. And so we're expanding from a single product company to a multi-product company. We have a great new analytics offering in our search product. We have a great new data collector product with Edge. We're expanding our cloud into Europe as well as well as expanding the company into Europe. And that's a huge milestone for us. And I think, as far as learnings go, I think the key thing is just keeping customers first and always that's one of our core values. We innovate from first principles. We do things differently here. We try to look at problems that customers have unique and different for our customers and go provide value to make it easy for them to manage the growing tide of data. Their data is growing in a 25% compound growth rate. Their budget is not. And we're there to help them with that mission in solving their data problems. I love that. That's a great alignment in terms of customer need, customer challenge. Double click a little bit. You talk about Edge. We're going to be more than double clicking into Cribble Edge in a minute. But I'd love to get your thoughts on Cribble's global expansion and why that is also an important milestone that company's achieving. Well, first, in order to expand globally, you have to be at a reasonable scale, right? I mean, expanding globally as a company, if people ever end up running a software company like I've learned to, there's a ton of work that goes into just establishing yourself in a country. Like, how do we figure out how to employ people over there? How do we figure out how to comply with the local laws? How do we find out, is the problem that we see in Germany the same as the problem that we see in the US? And so there's a lot of learnings in there. And then also expanding, since we are a SaaS company and expanding the product into Europe requires a lot of technical work behind the scenes to expand to multiple regions and expand into those territories. And I think we're providing, I think the problems that we see in EMEA are very, very similar to the problems we see in US. They have data growth problems, just like everybody else, but they also have a lot of data residency challenges and data sovereignty challenges that we're also helping them to address. And I think that's a unique set of problems that we're helping them address over there. Well, as you mentioned, the expansion is a multifaceted task and really initiative for any organization to take on its complex. Congratulations on the work that you guys have done so far. I'm sure the customers are very much appreciative of the efforts, the ongoing efforts to do that. I'd love for the audience to understand, speaking of expansion and transitions, cripples a transition from an observability pipeline company to a data engine for IT and security. That's a big transition. It's been in the cards since the very beginning. And when we started building the engine, which is the engine that powers all of our products, streams processing, analytics of data at rest, collecting data, these are all kind of multiple facets of the same problem. And so cripple software was built from the very beginning with the aim that we knew we wanted to help customers analyze data as well as move data, as well as collect data. And so the positioning update is really recognizing our transition into a multi-product company. We started with Cripple Stream. That's where we make most of our money today. It is a pipeline, it's a limited router. It allows you decouple sources and destinations and get data into wherever you want. We expanded with Edge to be able to move the ability to route data to multiple places right at the collector and manage thousands and thousands of collectors centrally. So that's a core value proposition. And we're super excited about our search product. We got more to talk about that on this interview today. The search product is the first search-in-place analytics solution that gives you the ability to analyze machine data, telemetry data, all types of classes of data that IT and security have no matter where that is. It could be in your existing solutions. It could be in your existing indexing solutions. It could be in a data lake. And that's where we're seeing the biggest opportunity is customers want to store petabytes up to exabytes of data cheaply in the cloud, in object storage. And so we're giving them the ability to analyze massive amounts of data at rest very cost-effectively. Because again, their data is growing so rapidly and their budget is not growing so rapidly that they're having to come up with a portfolio of ways of managing this data at rest. And search is really aimed at helping them solve that problem. Can you talk a little bit more, Clint, about how customers are influential in the direction that Kribble is going and some of the decisions that it's making from a search perspective, for example? I imagine you guys are a lockstep with customers helping you really kind of define the direction. Absolutely. And so at our conference over the summer, my co-founder and CTO asked the audience, how many of our features did we develop? Do you think we're done in collaboration with customers? And we confidently said 100%. There's not a single thing that we're building here that is not directly in alignment. And we're also our own internal customers as well. So Kribble Cloud is one of the largest customers of Kribble Edge and Kribble Search. And so our SREs operate a service just like everybody else operates a service. And so we get learnings from there. We get learnings from the unique requirements of selling into financial services, into government, into telecommunications, into online services. And all these people have unique requirements. So it's really, really important that we listen first and then act second so that we can build them the products that are gonna provide them the most value. I love that, listen first, act second. All right, let's talk about some of the news here. Kribble just announced support for Amazon EKS. Talk about what that means for customers. You obviously talked about the huge focus on customers and their needs that you guys have. But what does it mean for them? And why is this important? So obviously Kubernetes is a maturing engine for running compute. All the major cloud providers have a version of hosted Kubernetes. But we find that the problem is just not yet solved for our customers. Getting data out of their Kubernetes clusters remains a challenge. And there's a lot of competition in data collection for getting data out of Kubernetes. There's a lot of open source agents. There's a lot of proprietary agents. But I think what makes Kribble Edge different than everything else that we've seen in the market is the ability to centrally manage, the ability to teleport into an in node to introspect and understand what the running workloads are happening there in real time. The ability to be paired with Kribble Search so that we can search data at the edge directly without having to egress it. In a lot of cases, a lot of this data that they're using especially for troubleshooting purposes has a pretty rapidly decaying value over time. I need to search it right now, but 24 hours from now it's just not really all that interest anymore. I don't tend to go back like long periods of time. So the ability to have a rotating buffer at the edge that you can search and never have to go store actually provides them a lot higher value. They don't have to centralize that data. They can decentralize that data. And it's also working hand in hand with our partnership in Amazon Security Lake. So we can easily grab all the data out of the Kubernetes clusters, get that data in Amazon Security Lake, and Search can also work well with that because we can search the data in place in Amazon Security Lake. Unlike anything other solution in the market we don't have to move the data out of Security Lake. And so our customers are seeing a ton of value with pairing up data collection with their security efforts. Open up that hood if you will, Clint, in terms of some of the key business benefits that customers are going to be able to get from Cribble Edge and search and their unique combination. Yeah, so it's all about, I've got way more data than I have money. And so how can I reuse the resources that I'm already paying for? So if I have a Kubernetes cluster, that is a sunk cost. I'm running my compute workloads there. I'm running my applications on top of those compute workloads. And if I have to move log data, metric data, tracing data to another solution, I'm now paying for that data again. I'm paying for that data to be stored somewhere else. I'm paying for that data to be processed somewhere else. So if I can utilize, and by the way, the average utilization in most of these clusters is not 100%. It's usually more like 30, 50, 70%. So I have compute capacity and I have storage capacity that's sitting there that is a sunk cost, that I can now move my observability workloads, my security workloads onto, and be able to get a significantly more cost effective way of managing this massive amount of data. So the cost benefit savings seem pretty significant in terms of what you're talking about. What are some of the other key features that are delivering big business impact for your customers? Yeah, so data routing is at the core of what Edge and Stream do. And so if you ask me what Cribel sells, I usually say choice. It's the ability for customers to live in a complex ecosystem full of many, many, many tools. My background was as a practitioner prior to becoming a software vendor and it used to annoy me to no end that vendors come in and say, hey look, just replace everything you have with my thing. And that's just not the reality of most of our customers are enterprises, mature enterprises that have been making decisions for decades. And all those decisions were the right decision at the time that they made that decision. And so Cribel I think is a little unique as a vendor in that we really are Switzerland. We believe in working with everybody in the space. And so the routing capabilities of Stream and Edge really allowed them to say, hey look, I need this data to go to multiple places. My security team has one set of tools. My observability team has another set of tools. Why do I need two data collectors? Why do I need two data processors? Let's be able to take one feed, put it in the right place and manage the data in the right place for the right data. So you're Switzerland, you're living in reality as Cribel. What's been some of the feedback from the developers so far and the security folks? Yeah, I think it's really, what I hear from our customers as they're using our products is like, you guys really get me. Like this product feels like it was built by somebody who actually knows what problems I'm solving. And it's intuitive and it works. And that delights me to know in because I use the products, everybody at the company who's technical in some way uses the products, the majority of our company internally has taken our own education. They've been certified in how to use these products. And so they're really just crafted with love for the particular set of people that we sell to. They're under sold to. They're under marketed to. They're just not, there's not a lot of vendors who are focused on kind of SRE, DEVA. I mean, certainly there's a lot of observability vendors et cetera, but from a data company perspective, most people think of data as tabular. And we see all the different types of data that our customers have. And we give them functions and operators and ways to work with that data that go after the very unique different types of data that they have that, that is not the same type of data that the business has. I feel like this is a, Jerry McGuire, you had me at hello moments planning with your customers there. I couldn't pass that up. But I'd love to understand the AWS partnerships ongoing, talk about that and really kind of the impact that it delivers to Chicoville as a business and your customers. For sure.  They're the leading cloud provider for a reason. But they do not universally solve all challenges for their customers. And so they're a very partner friendly organization. And I've always liked that about working with AWS. So one of the key partnerships that I mentioned earlier is on Amazon Security Lake. And I think that that's just a really great new product offering from them. And I'm hearing from my customers out in the field of all the announcements that have happened over the last couple of years. That's repeated back to me. The customers are very interested in the value proposition of Amazon Security Lake. And we help them in two ways. One, we help them get data to Amazon Security Lake so we can get data to them in OCSF format so that all of the tools that are designed to read data out of Security Lake can have the data properly formatted as you're getting in there. And we have integrations for a dozen or more different vendors that are not natively supported by Security Lake to get data into Security Lake. And then as I mentioned earlier, search is the first of its kind to do search in place where we give people the ability to analyze the data in Amazon Security Lake without having to move it. Most of the solutions in the market today take the data that's at rest in Security Lake, they lift it up and then they send it to another solution for processing. So you're really storing the data twice. And so with Gribble Search, you can performantly analyze terabytes to petabytes of data at rest in Security Lake without having to move it. And you pay only for the compute that you're consuming. So you're paying us, it's like a light switch. You're paying us literally for the usage. So if you're not getting value out of it, you stop paying. That sounds like a massive differentiator plan. Double click on that. I'd love to learn for the audience to learn a little bit more on that. Yeah, so it gets a little technical, but in Security Lake, the data is stored in what's called parquet format. And parquet is an optimized columnar format for doing analysis of very, very large data sets. AWS, things like Athena can analyze this data natively. That's like a SQL-centric interface. And so usually that's what you find with more like business analysts. But our customers are security customers, our operational customers are used to a more search-centric experience for getting that data. So we offer you an experience that's a pipe-delimited query language. It's very familiar with products that you've used in the past. And we can performantly analyze that data because AWS is already making the data available in a format that very easy to analyze, massive data sets very rapidly, but you have to support being able to analyze that data in S3 at rest. And we built search very specifically to help people get value out of data and data lakes. And so, you know, these are new, like the, you know, Security Lake is a pretty new product. Data lakes in general are pretty new in observability. And so we are early to market, but I think that this is where the market is going. And it's definitely a future of how we're going to analyze this type of data. I love that early to market, really being led by the customers, but also the ecosystem. We talked about AWS, but I know that Kerbal's partner ecosystem has had a lot of expansion this year, CrowdStrike, Exibeme, Elastic. Talk about why these partnerships and the ecosystem is so important to Kerbal and its direction. For sure, back to those choices. Customers made a choice. Maybe they're, you know, let's take EDR, for example. There's a lot of competition in that marketplace. I think CrowdStrike, we've got a great partnership with them. We've done an OEM deal to help on their log scale offering to help get data into log scale. But maybe they're a Microsoft customer. Maybe they're Sentinel one customer. Maybe they're, maybe you're an AWS shop. Maybe you're an Azure shop. Maybe your, you know, customers make the choices that are well outside the control of what we can tell them to do. And so we really believe in meeting the customers where they're at. If you're an Exibeme customer, great. Like Exibeme has a fantastic solution. We just announced a brand new partnership with Elastic that I'm really, really excited about. Elastic is a phenomenal vendor. They're providing, you know, excellent security and observability solutions across the market. And so we really just want to meet our customers where at it's all about being customers first and we're going to work with anybody in the space if we can help our customers get more value out of their solutions. I love that. It's all about meeting customers where they are interoperability and choice best for customers. I know that's critical to the DNA at Cribble. Talk about some of the, as we were kind of wrapping up here, what are some of the key trends that you guys are paying attention to as we round out 2023 and head into 2024? One that I like to talk about that I think is interesting is that the cloud migration is far from over. And I think, you know, the press especially, you know, tends to grab things in the first part of the hype cycle. So we're all talking about generative AI and we're definitely making investments here and you'll see product announcements from us in the coming months around generative AI. But, you know, when I was a customer, you know, in the practitioner side in the late 2000s and early 2010s, we had to have a cloud strategy. Those companies that were starting to talk about cloud strategy in the early 2010s are just now starting to actually move to cloud it. So we're seeing, you know, very large institutions, which were staunchly on-premises institution, saying like, okay, nope, now's the time. Like we're actually going to go to cloud. And one of the interesting trends that I've seen is that security departments are actually leading here. And so historically they've been somewhat, you know, kind of reserved about going to cloud, but now they've seen, hey, the business is making a decision that we're going to cloud. So I'm going to put my workloads in there first. And one of the first workloads that they're moving are logs. Logs are big, they're expensive. It's most of the data that we process. And so the allure of being able to store data cheaply in object storage or in Amazon security lake, et cetera, is so important to them that can save so much money by going to this approach that security is actually leading in the transition to cloud, which is not something I'm hearing a whole bunch of people talk about elsewhere, but it's something that I'm certainly observing amongst our customers. Do you see some like significant title shifts? We're going to have to have you come back on and talk about that. But I'd love to know in terms of title shifts or even kind of what's next, your crystal ball, your magic eight ball for Cribble over the next six to 12 months. What do you see that you can share with us? Yeah, so I mean, as I mentioned, search what is early to market and it's in our cloud product. It's available in a consumption model. Working to get as many people, we have a free tier by the way. So anybody who thinks this is interesting and what I'm talking about, you can go to Cribble.cloud, you can sign up and driving adoption of our cloud products of our search products, our international expansion. We've kind of laid all this groundwork. Now it's time for us to really go out and take it to market, get customers on that product, get them talking about it, getting them talking to other customers and talking about all the value that they're getting out of there. And any developer, any end user who thinks that analyzing data at rest in S3, analyzing data at the edge sounds interesting. I encourage them to go, sign up for our free tier and get some value. I love this, Clint. Thank you so much for joining us on the program talking about Cribble, its evolution, Cribble Edge, Cribble Search, what you're doing to really meet customers where they are and help them solve problems. Thank you so much for joining us. Once again, on theCUBE, we appreciate your insights and your time. Thanks, Lisa.  Absolute pleasure. We want to thank you for watching SuperCloud 5, the battle for AI supremacy. We'll see you next.


Welcome, everyone. Thanks for attending the Open Source Summit. This is my first time speaking at this conference. I have spoke at other conferences previously. I work at Indeed.com. For those of you that don't know, Indeed is the number one job search engine around the world. We aggregate jobs from across the web and enable job seekers to search and apply to jobs across the world. We're available in 60 different countries and offer a wide variety of languages. My name is Maya. I'm a senior engineer working on service infrastructure at Indeed. This includes everything from the mechanisms that drive how our systems communicate with one another to the actual runtime platform that backs our ecosystem. One of my teams, the service architecture team, is responsible for the delivery of tools, systems, and services that help drive our service-oriented architecture. This talk is largely focused on how we went around vetting this kind of migration and why we would even go about making such a migration. I'll provide a little bit of background about how Indeed has historically written services, as well as the desire and end goal of this kind of migration, provide an overview of a test harness that we built to help address some of the concerns that our product teams and engineering managers had across the company when it came to this adoption. I'll show you some examples of how we built baselines from our existing production ecosystem to ensure that the migration would be safe and successful, and then possibly simulate a few workloads. It's a much smaller audience, so I might go through things a little bit quicker to enable more of an open form discussion at the end. We will look at some things that we learned at the end as well. Historically, Indeed has built its systems using Boxcar, which is an internal distributed services framework. It's largely based on protocol buffers and offers a cogeneration layer. It works by load balancing a set of operations across fixed-size pools of TCP connections, and we only really have great support offered in Java. Two years ago, we first started considering migrating off of Boxcar. We have historically only had a handful of people working on the framework internally. Being able to leverage something like GRPC, which is an open source solution, gave us a lot of value. But in the use of Boxcar, we've also found ourselves re-implementing the wheel when it came to things like circuit braking and client-side load balancing. And so we started to consider the use of a service mesh as well. During the summer of 2017, we evaluated a bunch of different service mesh solutions out there, and the one we ended up coming out with was LinkerD. And this was LinkerD v1, so the Java-based version, not the rewrite that they had recently done. Later that year, I spoke at KubeCon, describing how we were able to leverage some of these pieces of technology to enable communication with our Java-based Boxcar services. In 2018, something interesting happened. I actually left, indeed, for about seven months. Being one of the main technical leads pursuing this initiative, we saw a little bit of a halt in the project. And so the team had to work really hard to back view a few roles while I was out. Upon arrival, I had discovered that we actually switched technologies. We went from using LinkerD 1.1 to using Envoy because of its lower resource footprint. We did evaluate Envoy in that summer of 2017, but there was some performance issues there when it came to having the large number of services that we have running in our ecosystem. Since then, the team has been hard at work driving that solution across our ecosystem. Now, some people might ask why we might do something and do such a thing, where we've taken our existing infrastructure, almost thrown it out the window, and started over. For one, indeed, started to leverage Python in more key business areas. Because the way Boxcar was written, we couldn't really port it to Python all that well. And so we wanted to consider a solution like gRPC, which would give us more native support for Python. Envoy would kind of fill in some of the other gaps that we had, where we had thick client-side load balancing, as well as common circuit breaking patterns. Together, these two pieces of technology also let us transparently handle encrypted HTTP2 connections. Applications would be able to continue to speak the protocols that they were very familiar with, whether it be HTTP or even the plaintext HTTP2 protocol. And then Envoy would encapsulate a lot of the encryption and long-lived TLS connections. In our Apache Mesos ecosystem, both the application and Envoy's sidecar ran within the same container. And as we started to shift more and more components to Kubernetes, each of these processes would run as their own container within the same pod. Once we built our initial integration, my team had gone out and started working with different products and technical leads at the company, trying to get them to adopt or consider the solution. After presenting to a few of our larger products, they were comfortable with the idea of adopting a service mesh if it meant that their application logic was easier to reason about, but they were concerned about the underlying performance of the two different protocols. At the time, we had some anecdotal evidence that proved out our integration and showed that it was relatively comparable, but it wasn't really the traffic or the numbers that these business critical systems were looking for. And so we built a system called Hyperloop internally. It was originally designed to exhaustively test new versions of Boxcar before releasing them to the organization at large. But with a few modifications, we were able to leverage this system to directly compare Boxcar and GRPC. We used Prometheus to monitor the underlying infrastructure. And with the way that our current applications work today, we emit all of our performance metrics via stats D. And we were able to leverage the stats D exporter from the Prometheus community to translate those metrics into something that Prometheus could scrape. Hyperloop itself has two main components, service and a consumer. Service is pretty simple. It's just a ping pong style service. Receives a request, sleeps for some configured duration, and then returns a response. The client can be configured to induce different loads on the service. And there's a few different knobs that we have that can adjust that. With a few more modifications, the same ecosystem was able to test out the actual underlying service mesh integration as well. While we didn't necessarily perform this head-to-head analysis in the case of this test, we do run the triad of systems in our QA and production environments. So we have one setup for Boxcar, one setup for GRPC, and then one setup for GRPC over the service mesh. The note here is that Boxcar would not be able to run over the service mesh as it requires direct management of the underlying TCP connections to ensure load balancing is properly happening across our ecosystem. When it came to tuning and adjusting these underlying systems, Boxcar was really easy. Clients would just set whatever number of connections they needed to maintain, whereas servers would set the number of connections that they were willing to accept. GRPC, for those who are unfamiliar with some of the underlying mechanics, specifically within the GRPC Java implementation, has two key components. An executor, which currently isn't pictured in this diagram. At first, it wasn't really clear what it was responsible for, but after working with the GRPC team at Google, we learned that it was primarily responsible for communication with out-of-band load balancers. So this is more or less the GRPC load balancer, which eventually got replaced with XDS today. The second component is the event loop group. And this is more netty specific. It handles reading all of the responses that are coming back from the server and writing all requests to the server and then reading the responses coming back. If you're using the Google APIs extensions library, you can choose to leverage a channel pool. This is a way to scale out the number of connections that you open to a back end within GRPC. By default, it will only open a single connection per back end, and this gives you a way to kind of improve throughput in the case of having a fixed number of back ends. On the server side, we have three more components. The first is a boss event loop group. It is responsible for accepting all new connections coming into the server, handshaking it if you're using TLS, and then inserting it into a pool. Because of the light demand on this component, we didn't really have to worry about tuning it as we moved through our testing. But it's just something that you might be aware of if you have a system that needs to manage high frequency of connections. The second component is a worker event loop group. It's kind of the server side equivalent to the client side event loop group where it handles reading the requests coming into the service, handing it off to the executor, and then writing the responses back. Finally, the executor is one of the other core components. It's actually responsible for directly invoking the business logic on your server side. And one thing that's kind of important to note here is that it's actually server implementation agnostic. Once we knew the components that we wanted to tune and adjust within the underlying ecosystem, we started to build a baseline off of our production workloads. Some of the top bits of information that we wanted to know about were the scale of requests and the distribution across our ecosystem. Using some metrics that we were already able, or that we were already emitting to Datadog, we could graph the request rates of servers against the total of number of connections that were established to them. On the x-axis, we have the number of connections that are currently established, and then on the y, we have the corresponding request rates. We can see some of our higher-end services performing about 2,400 operations per second. But one of the nice things about this graph is it shows where the majority of our services actually do work, which is less than 500 operations per second. Similarly, we were able to construct this visualization for our client side because number of instances matter. Clients might issue request rates at different... Might issue requests at different rates than servers are able to receive them. And so we wanted to be able to answer this side of the problem as well. Similarly, the number of connections a client had open is on the x, and then the corresponding request rates are on the y. Caviar, or I guess, one call out here is h.in both these graphs represents a single application in our ecosystem. So you can see up in the far upper right, we have a single client that issues about 4,500 operations per second. What this effectively did was it gave us a great way to prepare a mental model that well-described our ecosystem. We knew a few of the extremes being server being... Server being 2,400 operations, client being 4,500 operations, but then we knew that the majority of our systems fell below 500. And so it gave us an easy target to more or less shoot for in our evaluation. The last thing that we tended to wanna ask about was latency. So under these request loads, what latency did we see when it came to just raw buyer time? So kind of excluding what the performance of the server was doing. And when we started to compute that, what we found was the latency was highly dependent around the ecosystem that we were running in. Cloud providers make different guarantees around what kind of latency you'd see at different points in time. And so we couldn't really use that as a way to establish a baseline. Instead, we just went with more of a direct comparative analysis and then kind of sanity checked ourselves when we got into our QA and production ecosystems. From there, we were able to start simulating workloads. And one of the big things that we needed to do was figure out what the number of event loop groups to set things to. After digging through code, we learned that the client side event loop group and the server side event loop group were dynamically sized off the number of cores allocated to the process. Except that in certain versions of Java, the allocation wasn't properly respecting C groups. And so you would say, what number of cores am I allocated? And you'd actually get the total number of cores allocated to the cluster. To kind of mitigate this, we just kind of assumed a single core. So we programmatically set a single core for the event loop group on the client and two cores for the, or two event loop groups on the server. With some minimal configuration, we were able to replicate most of our workloads at Indeed. We had 32 worker threads that were each issuing two calls in a batch. And then repeating that process for until the client side process had terminated. Both GRPC and Boxcar were able to match their underlying request rates. And the 90th percentile client side response time were both identical. It's important to note that the server in this case had been configured to do an 11 millisecond sleep. And so in this case, the client actually saw zero milliseconds of wire time. Zero milliseconds. There may have been some lower thresholds that we could have gone to, but the unfortunate part about our ecosystem was we only met this information at millisecond granularity. We continued on and doubled the number of calls we were performing in each batch. Again, being able to match the request rates between Boxcar and GRPC. And again, being able to see the same exact response time. We increased our batch again from four to six. And this time we were able to again match response time, or match request rates. But we finally saw the first point where the response times were different. GRPC had seen a wire time latency of about one millisecond, whereas Boxcar still had not seen that. We increased again from six to eight. And now we finally saw a difference in the actual request rates that both GRPC and Boxcar were issuing. Boxcar was able to still handle 2,555 operations per second, whereas GRPC lagged behind by four ops. And in this configuration, we actually saw a little bit more latency. Boxcar started to see about one millisecond of latency and then eventually got a little bit better, whereas GRPC saw more of one to two milliseconds of the difference. We increased again to 10, and Boxcar kind of continued to pull ahead. So 2,909 operations per second, whereas GRPC fell a little bit further behind at 2,671. Hanging a little bit more to that two millisecond mark. But what was interesting about these trials was in that last few steps, we saw smaller and smaller deltas between each iteration. In the first three, we saw differences of about 600 operations per second. But in the last three, we're only seeing one to 200 operations per second differences. And this called out to a bottleneck that we had in our underlying implementation. If you think about it, there's 1,000 milliseconds in a second, with each call lasting about 11 milliseconds and 32 workers each issuing a call, the maximum number of calls that Boxcar could have produced was 2,909 operations per second. With GRPC at 12 milliseconds a call, we can compute that to be about 2,666. And so what we wound up doing was going back to that point just before we put that system under contention. And started to work on sussing out some of the differences that we saw, both in terms of request rates as well as in latency. The first thing that we kind of noticed was that the server's request rate was about 100 less than what the client was reporting. And so we were able to go and double the number of event loop groups on the server. After doubling the number of event loop groups, we saw that the request rate for both client and server actually matched again. But we were still seeing that one millisecond of latency on the client. And so we decided to try doubling that not only once, but twice. And after not being able to suss out that one millisecond of latency, we decided to accept that maybe there was a networking issue within our test harness. Maybe there was some underlying, just some underlying reporting fuzziness that we couldn't really deal with. And maybe just one millisecond was something that we would have to accept as the cost of adopting this framework. With that state of configuration, we were able to replicate most of our workloads. But we were still missing that 4,500 operations per second mark. By quickly doubling the number of workers on the client that were issuing requests, we were able to get up to that peak number, or at least close to it, with 4,200 operations per second. In preparation for this, we did double the number of threads on the server side executor pool, as well as the number of event loop groups. And at this configuration, what we had noticed was that GRPC was actually outperforming Boxcar, both at the 90th and 99th percentiles. While we did incur some slight increases in latency, we were able to kinda look at it and say, well, knowing that we run into problems when Boxcar is under contention, this may be a better solution for our ecosystem. We took the opportunity to then also test how the number of channels in a channel pool impacted performance in relation to the number of event loop groups. And what we had learned was that doubling the number of event loop groups was about the same performance benefit that we saw from doubling the number of channels that were established to the backend. There's some key kinda differences in when you wanna do one or the other, and we'll get into those in a minute. But generally speaking, if you can scale your backend, you don't necessarily have to worry about setting up a channel pool. So in this process, we learned quite a few things. When it comes to tuning GRPC clients, you wanna set that event loop groups configuration based on your expected IO, so the number of requests that you're sending to the backend service and the number of responses that you're getting from it. This can be influenced both by application load as well as the use of some stream-based API calls. So if you're doing client-side streaming or you're doing server-side streaming or even by-dye streaming, you wanna take that into consideration when setting this value. And a channel pool, you wanna use seldomly, but the places to use it is when you can't tune or configure the number of backends that you're speaking to. So in the case of a service mesh, you only have one target that you're typically dialing, that single proxy that you're talking to. And so being able to leverage a channel pool can help increase throughput to that underlying proxy. On the server-side, you wanna tune that worker event loop group similarly to your client. So expected based on the number of requests that you're reading in and the number of responses that you're writing back. And the executor can be used as a way to apply back pressure and only prefer a certain number of operations concurrently. The other thing to note about the executor is that the type of executor can influence your performance as well. So by default, we use the fork join work pool, which will allow us to reclaim threads in the event that a worker sleeps or goes off and does some other IO out of bounds. The last question that we had asked or the product teams had been asking us, how did Boxcar and GRPC compare to one another? We were able to tell them that they performed comparably. We couldn't tell them one was better over the other. There was some situations where one was better than the other, but generally speaking, we tended to prefer the kind of newer solution. Boxcar was much easier to tune for our application developers. They would only have to worry about setting one value, but then they would run into problems where when their system was under load or under contention, we had to go in and manually intervene to alleviate problems. And finally, that GRPC was a lot more flexible to the types of workloads that we tend to see where bursts in traffic occur and occur often. One of the final lessons that we kind of took away was from the GRPC conference earlier this year. Anna Berenberg, a distinguished engineer at Google, had presented this notion where the actual client application and server application can be actors in the data plane of a service mesh. So instead of just having to go through a proxy, systems that were really latency sensitive were able to actually partake in it as well. And if you follow the source code for GRPC closely like I do, you can actually start to see where they've started to integrate directly with systems like XDS as a means to act as a control plane for the actual systems applications themselves. Thank you. Comments. I have a mic. I think I have a mic. I don't have a mic. I don't know how this thing turns on. Yeah, I'll reiterate the questions back. Did we mix GRPC and Envoy in our architecture? Yeah, so the actual service architecture team started to require both GRPC and service mesh migration at the same time, instead of doing them as kind of a two-phased upgrade, where one, you might go and upgrade to using GRPC first and then cut over to using a service mesh. The XDS integration has only recently been put in and I'm not sure of its maturity yet. We never went through and actually did the full performance evaluation, but I'd be really curious to go and take some of the same harness metrics that we were able to put together and do that same test. So when we were using the service mesh, in our test harness, we saw some latency, but when we got into our production ecosystem, we actually saw comparable performance. It was rather interesting and one of the things that we were looking at was its possible relation to an MTU issue that we were having inside of our cluster. We had some, the ecosystem where we had run this performance test was an internal open-stack cluster and when the systems administrators had set up that cluster, they set the MTU to be much lower than kind of your default packet size and so we ran into all sorts of problems when it came to handling SSL connections or downloading larger files. It was a really, really frustrating thing from our perspective and it bit us quite a bit during our Kubernetes migration as well. So the question was what's our control plane? We wrote our own internal XDS implementation given that it's just a spec. It was really easy to sit down and take kind of the existing data store that we had. We used console for service registration and discovery. Console Connect was not available at the version that we were using plus the semantics in terms of how we were registering services didn't quite align with the way that the systems were expecting to resolve it so we wrote a quick little, believe it was a Java layer that kind of did that translation for us. So it would read from console, translate it back and return it to our clients. Transparent TLS was a big one especially with the ability to have HTTP2 connections between point to point. That way languages like Python or PHP where you necessarily didn't have this benefit of having a HTTP2 connection from your actual application logic would benefit from this kind of short on box hop to the proxy and then the proxy kind of handled this more performant long distance call across machines. Some of the other big features that we were looking at trying to get was traffic monitoring and traffic control so being able to do more shifting between different versions of our applications. We didn't really have a great way to do that in boxcar but that's been a big feature ask where people want to say like I want to send 10% traffic, 10% of my traffic to this new instance and so the service mesh would have been able to give us that type of capability. So your hand up next. So the question was did we consider SEO? Yes we've actually, we went back and forth on it so in that first evaluation in 2017 we did the evaluation at the time it was only supporting Kubernetes which didn't match our underlying runtime platform. We were on Apache Mesos at the time and so we kind of ruled it out as an immediate option but we kind of put a flagpole in it and upon my return back to NDE I actually went through and did a whole other evaluation of Istio because we had learned that they had support for console and more nomad based ecosystems. Not that we run nomad but given that it supported console as a back end it was something that we could easily integrate with as well. Again it kind of boiled down to the differences in terms of how we were registering the semantics within the cluster and then when I think we were going through and trying to figure out what that migration path looked like it looked very painful. Istio from what I had seen was really easy to adopt if you didn't have any communication in place like if you were already operating in a clustered ecosystem like Nomad or Docker Swarm or Kubernetes it was really easy to just drop in and gain all the benefits from but in a legacy ecosystem like ours it was really difficult to sit down and actually get integrated. You said you had two questions, right? So a question was how are we handling authentication and authorization? Mutual TLS is primarily how we're doing the authentication mechanism. Authorization is currently done off API tokens but we were talking about leveraging things like RBAC and control policies that Istio was originally providing but then possibly implementing some of that within our XDS layer. So a question was around distributed tracing. We do have distributed tracing in place we actually implemented an internal version off the Google Dapper paper back in 2014 that has since been migrated to using things like open tracing and light step as our primary sources of integration that's been working out pretty well. It's actually reduced a lot of the metrics that we no longer have to admit because that actually gives us a lot of what we were looking for. So the service mesh specifically so the question was are we using service mesh in production? Yes there's several teams that are currently using it and they've actually reported better performance than Boxcar in their cases. These cases were things like a good example is our messaging pipeline so the team that owns kind of all of our email integration they switched their services over to using the service mesh and have seen a huge improvement primarily around the ability to do things like this Mutual TLS and then also handling what was the big one? The HTTP2 connections and being able to interleave kind of requests along a single stream plus the compression both ways. So compression on request plus compression on response. We did have that in Boxcar but we had compression on response in Boxcar we did not have it on request. Request is a very difficult problem to solve if you don't have it in from the beginning. Any other questions? I think that's time. I think it's lunch so I'll be up here at least for the time being feel free to come up and chat. I'll also be out and about at lunch so feel free to reach out. Thanks.