Good afternoon. It is 3 o'clock and at this point in time we are just waiting for our library trustees to be able to join us in the virtual platform. And I see that I have one trustee that is able to join, so we will just wait a bit so that we get a few more people so that we can have quorum. So Nancy, thank you for joining us and we will just wait a few moments to see if we can get a few more people joining us. Thank you. And I'm just going to check my emails and my phone just to see if there's any trustee that is having difficulty trying to log on. So I'm just going to look at that quickly. For those of you who are library staff that are already on our virtual platform of our meeting, thank you all for waiting patiently. We look forward to hearing your reports in the latter part of our meeting. Only Nancy so far? Only Nancy so far? We have Nancy and we have Meg and we have Father William. Okay, so we have four. And if there's any members of the public that are watching right now, we are not starting our meeting until we actually do have a quorum of our trustee which would be a total of six. So sometimes technology is very easy and helpful and sometimes technology is a little bit of a challenge for everyone to try to get on to this virtual platform. So for those of you who are able to join us, thank you. We're just going to wait and see if we can get a few more people. Thank you. That's fine. Thank you, Mary Lynn. And at this time, we are just waiting for one more trustee and we will, at that point, we will have quorum and we will begin our meeting. So thank you all for patiently waiting. It's a great time to reread your minutes. I don't know for the third time. But and Sydney, since you are on the line at this time, is there any trustee that has shared that they are unable to make the meeting? I just want to make sure that six people were planning to attend today. All right, thank you. Terrific. She's on my schedule, too. So at this time, we do have our quorum of six. So we can go ahead and begin. So I will call our meeting to order at this time. We do have a quorum of six. And it's not very often that I get to do the next thing on the agenda, which is to do a very warm welcome to our new trustee, Reverend William Bolson, who is to my right, more than six feet away to my right. But if we can all just extend a warm welcome to him for joining our trustee board of Mead Public Library. Library Director Garrett Erickson and I had a wonderful tour of the library with him yesterday, as well as a meeting in which I officially had him take the oath to be the trustee. He is actually completing the term of John Motiska, who stepped down in the summer, in last summer. So he is very kindly stepping in to complete that term. And we hope that he's able to continue on to a future term. But it was very nice to meet you. And in the near future, when it's a little easier for all of us to get together, we can learn a little bit more about his areas of expertise and his interest. I can safely say that he was absolutely wowed by Mead Public Library and just had a lot of wonderful comments to share that I wish I had recorded so that we could share it with all of you. But it'll be wonderful having yet another member so that we're at a full table of ten for this library board. So at this time, if someone, if everyone would like to rise and join me in the Pledge of Allegiance. United States of America and to the Republic for which it stands, one nation under God, indivisible, with liberty and justice for all. Okay. And next on our agenda is public comments, which we do not have anyone here at this time. And moving on then to 1.5, if someone would like to make a motion to approve our minutes from, I'm just trying to get the date here, from January 28th, 2021. Would someone like to make a motion? Mary Landonahue made a motion. Would someone like to second? Meg Albrink seconded. Any further discussion? All those in favor, please say aye. Aye. Any opposed? Motion carried. Thank you. Next up, I'm just trying to get the buttons here. Under 1.6, I just wanted to take a moment to share with you a future communication. I am in the process of writing in my role of President of Mead Public Library Board of Trustees. Many of you are probably aware that recently the Wisconsin Department of Health Services announced additional members of our community that would be eligible for Phase 1C of vaccinations, starting on March 1st. This is incredible positive news for teachers and child care workers and, you know, people who really do work with so many members of our public and they really provide such a valuable service for the education of members of our community. Interestingly, for me, was reading a little bit more about those who were qualified for 1C in that Phase 1, 1C is the fact that despite the Center for Disease Control recommendation, library workers were not included in that very important group. And I do know that their State Professional Organization, the Wisconsin Library Association submitted, I believe, over 80 pages of testimony from all the libraries across the State of Wisconsin because they feel in their role, they are quite similar to the retail educational support staff and social services that are offered to our citizens in the State of Wisconsin. And in my role as President of the Library Board, I am planning to send a letter to the Wisconsin Department of Health Services as well as our local public health department as well as Governor Evers, just sharing my concern that our library workers are essential workers in our community. And we have been so impressed by their courage and their dedication in continuing to offer vital services to our citizens that have been so essential during this pandemic. And we have managed to keep our library open for many months, even though other libraries in the States were not able to because we knew how important our services were to our citizens. So I just wanted to let all of you be aware that that is something that I am just going to be writing as President of this Board so that they're aware that there are trustees that are quite concerned with the decision to not include library workers in this important classification of people who can now start receiving vaccinations starting March 1st. So I just wanted to make sure all of you were aware. So with that, we can move on to, unless there was anything else, Garrett, that you just wanted to share with me. Next up would be committee reports and it looks like there's, for once, nothing listed. Oh my goodness, I think I should highlight that. So we're going to move right along to items for discussion and possible action. 3.1, the DPI annual report. And I'll take this one, Maeve. This is something that is required for us from the public library standpoint each year that we submit our annual statistics as well as financial report to the Department of Public Instruction which oversees libraries and schools. And so you should have gotten an email with the version that was turned in to our system, the Monarch system, and then they take a look at it and then it does go on to the state. However, I do need to get approval from the trustees on this. So I guess hopefully everybody got a chance to take a look at it. And if they had any questions, please let me know right now. So just looking over, any trustees have any questions or comments for Garrett Erickson? But like the report was pretty standard. It was actually unique this year in the sense there's a whole bunch of questions about how we handled COVID services. So like, for instance, on the first page, it talks about limited service and so on. That was due to COVID. So this was a fairly unique. There were several questions that were unique on here compared to past years. The rest of it is very similar though. Wonderful. So at this time we would need someone to make a motion. Mm-hmm. So, yep, Maryland Donahue has moved. Oh, maybe I have a question. Thank you, Maryland. All right, thank you. Any other questions, comments? All right, would someone like to make a motion? Oh, okay, I'm sorry. So it has been moved. Thank you, Maryland. Would someone like to make a second for this DPI report? Kyle Welton makes a second. Any further discussion? All those in favor, please say aye. Aye. Any opposed? All right, motion carries. Thank you. 8.2, National Library Week, April 4th through 10th. Thanks, Mayf. On this one, we are asking once again that during National Library Week that we can provide amnesty for late fines to patrons that bring their materials in. So this does not cover replacement costs if they were to lose an item. It really just allows them to bring in items that are late and we would not charge them. It's a great opportunity to get a lot of materials back and to build some goodwill in the community. Who would like to make a motion for this idea? Meg Elbrink would like to move. Is there a second? Nancy, second. Any further discussion? All those in favor, please say aye. Aye. Any opposed? Hey, motion carries. Thank you. 3.3, Mead Internal Committee Structure. So this was something Maeve brought up in the fall that she would be very curious to hear about how we make decisions and bring things to the library board. And so I thought about how we wanted to do this and I thought back to a book that I read. I actually have to hold up. It's called Primal Leadership from Daniel Goldman, if you've heard of him. Daniel Goldman was the one who wrote Emotional Intelligence as well. It's a very good book, a classic business book about leadership. In the book, Goldman talks about four positive types of leadership. There's the visionary. There's the coach. Another style is called the Affiliative, which is sort of a person that connects emotionally with coworkers and subordinates. There's the democratic style of that type of manager who goes out and gets a lot of input and is very interested in building consensus on ideas before following through. And so I think a lot of what we do at the library is been put in place by Melissa Prentis, our public services manager. And Melissa, to think about her and her style is very much a democratic or democracy-style leader. She really puts together committees and tries to get input from all of the people that work with her and really build consensus and gets a lot of input. And so I thought that was really positive for her. And so what you're going to see is a document that we attached to board docs. There's quite a few committees that she now has, and it's really a change from when both her and I started in 2013 and 2014 when we had a much bigger management staff, quite honestly, but the staff also didn't have the input. We didn't have the processes in place. So Kudos to Melissa, and I know now Cheryl as well is working with her on even expanding that further. So I'm going to actually let the two of them talk about the document that is attached and shows all the different committees that we have within the library. Kind of manage the decision-making. That really does come from the staff members. So that's sort of the broad overview. Any questions about that? So just looking at everyone's faces, I just personally wanted to share that, you know, I think we've all recognized that there are so many steps that need to be taken to make all of the programming and all of the aspects of the library work so flawlessly, and now we've just been given a glimpse of how much work and the structure that's been put in place. And I have to tell you, leave it to the librarians to not only give us a written document that lays it out perfectly, but then they give us a visual diagram at the end. So, you know, I'm a visual learner, so I in particular just really enjoyed being able to visualize how this all works. And so I really thank you for taking the time for trying to educate us a little bit more about how you really get so many things accomplished every year. Just looking at my trustees to see if anyone else has something that they'd like to ask, or I can just tell you from the head nods, Melissa, and some of the glances, everyone is just really impressed with what you and Cheryl have put together with Garrett to try to, you know, manage our library. No wonder we keep meeting our vision and mission all the time. So with that, I will say thank you, and we can move on then to 3.4, which is a different topic, COVID service responses. Yes, and so what we actually, to step back, I guess, since October, we have closed down the second and third floor, and that was really due to some staff getting COVID and then the folks that were interacting with them, having to quarantine as well, and we really just didn't feel like we could provide the staffing levels that we needed to keep the library open. And so Melissa's compromise, rather than closing the doors and just providing curbside, was to perhaps we can go down to one floor for a while and see how everything pans out. And we've done that and managed to keep the first floor open and still provide access to all the services such as computers and so on, and it's worked out pretty well. Some people, of course, want to peruse the collections themselves, and that's understood. We're at a point now. I guess I've been asking the managers when would be a good time that we can go back to opening up those other floors with still somewhat limited hours, but when can we open up those floors? And Melissa and Cheryl did come to me a month or two ago and said, I think we should set a target date. And so they had set the target date of March 1st to open second and third floor up, and we have been monitoring everything and the staff have been really responsible at home and we've got very few cases to be honest of COVID within the staff. So we're feeling that not only are we doing the right things at home, but obviously our safety protocols at work are doing well as well. And we're ready to open the second and third floors. So I guess with that maybe, I don't know if Melissa has anything else to add to that. We're still concerned about staffing quite honestly. We're worried that if we open up that something could happen and we could have a few people out and then we'd be back to the first floor. But Melissa, do you have anything to add? Not too much just to say that. I agree it would be great if it's all smooth sailing from here on out. And I think that was one of our bigger concerns with reopening those floors too early is would we see that post-holiday surge with that impact our staff? Would we then be having to close down again? And that causes a lot of chaos for the public and for managing a schedule and that sort of thing on the staff end of things. So that March 1st date was really trying to give us enough time to see how that all played out. And then of course, having a couple of floors closed, we were doing projects on those floors like painting and moving things around. So planning to get everything back in order takes a little bit of time as well. But I can say that by and large, the staff are excited to get the rest of the library open again. It's really difficult when we know that there are things that patrons need and want to do on those other floors and to have to say no, we can't do that right now. So I think everyone's really excited about getting back to some level of normalcy and routine. And of course, I think that we've been doing this for almost a year now to some extent. I think folks are more confident too in the safety protocols that we are taking and that they're taking at home and there's a little bit less anxiety around that as well. This does still mean we're on unlimited hours and it also, we're not opening up, for instance, the meeting spaces. This would just be the main areas. The quiet study area on the second floor would remain closed, but we would open up basically a lot more computers on the second floor and people can peruse the collections themselves on second and third. So we think that's a huge benefit. And then we'll continue to watch what the health department recommends and so on to move to the next step down the road. But we think this is a really positive development for the community. Do you have a sense for how you're going to share this information to the public? Are you going to kind of do a soft opening before really sharing on social media? All three floors are now open. What are your thoughts for how that's going to be communicated with our public? I think the last time this governing body met, that was the sort of, the directive was sort of a soft opening just to make sure everything went well. And so we'll open up Monday and hope that everything goes well and kind of play it by ear and then continue to promote as it seems as though everything's going well. Thank you. Any questions or comments? The next steps for March 1st. All right. Thank you for that update. Moving on then to 3.5. DPI inclusive services assessment and guide. Thank you. We have wanted to start using and looking at this tool over a year ago already and then COVID hit and it's sort of been put on the back burner. We knew this was an issue that we needed to deal with and so hopefully everybody got a chance at least to look at the tool itself. So obviously there's like over 10 pages of different items to checklist on what the libraries can be doing and this is really intended for board as well but the staff as well. So I guess first I should open it up for comments and then as far as a process what we're thinking is that the staff could meet and sort of go through where we think that we are and then present this to the board next month to start to prioritize areas that maybe we aren't doing at this point but that's kind of how the process that I saw moving forward is for us to take a first look at it because some of these things you wouldn't as board members know what we're doing internally anyway but I guess I'll open it up just to hear comments. Any thoughts or questions on the report? Nancy mentioned. I just wanted to comment that I went through it and counted out there are 22 areas and over 300 questions so for Melissa and committees who go at answering that's really a large task and an important one too and also that board members are listed there. Number one, to be familiar with the document so we're on our way to assessing. Thank you. Any other questions or comments? My thought is that it was another example of just how well libraries just gather information and present information and I agree with Nancy. I was a little overwhelmed by the number of questions but they really ended up making me think even more so I guess we sort of look to your guidance scared of how you envision us moving forward. So I think Melissa and Cheryl and I will meet and try to figure out how to sort of do our own self-assessment and then present that to the board at the next meeting but I'm glad you guys had a chance to take a look at this document. It's super detailed and there's a lot of work to be done on it. It's a great document that they put together for us to use. I'm so happy we don't have to come up with this sort of a thing from scratch. So then the next thought I guess would be that we present this at the next... We don't have a meeting in March due to spring break but we would present this by April so we'll try to get this to the trustees as soon as we can get that accomplished and then they can start to look at that themselves and I guess I think the next step after that would be that you guys start to give us some priority areas that we need to work on that we're not perhaps doing yet. Great. All right. Thank you. Oh yes, Nancy. Nancy. I just had one more comment and as I was going through it I thought that it's also a useful document for our committees. I had the art committee in mind but this is a document that can help guide and direct conversations within our committees also. Wonderful. Thank you. All right. And then with that we'll go to 3.6 the 2021 holiday calendar and I will turn this over to our library director. Just Sydney, thank you for putting together the list of holidays and so on. So at the end of the year you can see we always struggle with holidays over the weekends. Normally when you have a Saturday holiday we bump that back to the Friday before that is taken off and the same with Sundays we take the Monday following off so you'll see a few of those dates in there. And then I guess I'd open it up to questions at this point but otherwise thank you Sydney for putting that together. Did anyone have any questions about the proposed recognized holidays for the 2021 fiscal year? Would someone like to take a motion to approve our new 2021 holiday schedule? Okay. Just trying to see. Okay. So it sounds like Sherry Speth I believe made a motion and is someone second? We're getting some interesting noises but all right. All those in favor please say aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Aye. Maryland has done a number of cooking classes at the library that have been really successful and it's something that our staff love planning with her and have missed since COVID times. So they had this great idea to basically film it like a cooking show and have the basic ingredients available as a take home kit like we've been doing with all of the crafting kits for kids since the pandemic. And we had slots available for 20 people and they filled up within two days. So I think people are really excited about it and I hope it goes well. And then we are continuing along those lines to do those take and make kits for kids. And we have one right now that we, again, we ran out of all 50 kits within a couple of days for making a snow volcano. So it's a little bit of a science experiment you could do outside with the snow. There's some food coloring and all of the ingredients you need to make a snow volcano. Very exciting stuff. For the entire month of March, we are doing our peeps diorama contest which we have done in the past. This one will be virtual this time. So we've had some really amazing submissions in past years. Usually you use a shoe box and peeps to create fun dioramas. And then we run that through spring break and choose a winner at the end. This year, we're doing a virtual version of wintergreen. So for those of you that are familiar with that program, I believe this is the fourth or fifth year. We've done it and that is really it's a day of programming around the end of winter and the beginning of spring. Even if spring doesn't really come here until May, we do it in March anyways. Just see it beat those winter blues a little bit. So there's going to be a number of virtual programs that day on seed starting. Marilyn will be doing another cooking class for us, urban homesteading, environmental stewardship with the Lakeshore Natural Resource Partnership. And then a few virtual STEM classes for kids. One on measuring sunlight with a photo resistor using an Arduino and another Arduino based program as well. So we're excited about that. There's been some good registration for that already. The Optimus Club is doing a spring break activity make and take kit with us as well. That will be available starting at the end of March. And then I mentioned last time the new racial justice book discussion group that Judge Natasha Tory is facilitating for us. We had our first session a couple weeks ago and it was really successful. Thank you, Meg, for that suggestion of doing breakout rooms in Zoom. That worked really well. We ended up doing two breakout rooms of 12 people each and that was just the right size to get good discussion going. And then we came back in and shared all of our thoughts as a group. The next one we'll be meeting in two weeks, just a little less than two weeks, the second Tuesday of March. The book we'll be discussing is Homegoing by Yad Jassi, a fiction book but really excellent. There is still some space in that one but it's filling fast. And that's all I have unless there's any questions. Any other questions or comments? Melissa, did you want to mention the radio show too with John? Yes, I haven't mentioned that yet because I know there's some debate on the go live date for it. But I think that we've talked about the launch of this Meade radio project with the board in the past, maybe a few months ago. John Tully is a fairly recent hire. He's been with Meade for about a year now. And he has experienced, years of experience working in community radio. So he had this fantastic idea for, given where we're at with the pandemic, to launch a community radio out of Meade. And it's actually surprisingly, especially if you have someone who already knows what they're doing with that kind of thing, the tools out there make it really easy and relatively inexpensive to basically create your own web-based radio station. So right now we're in the process of developing a whole slate of programming for that. And we'll be working with people out in the community, some of our community partners, to be creating radio shows around that. But right now, John and Jeannie Gartman have been working with Lisa Veehose, our poet laureate, to create a poetry on the air show. They've recorded three shows so far. And they're going to start airing those, I believe, next week. So look out for some information from Josh on Facebook about that. And then we also have, for the times that we're not going to have on air shows, we'll be streaming music that's part of the Meade Library music collection. So we're really excited about it. There's a lot, I think, a lot of potential. And I'm really excited to see where we go with the radio project. For those of you who have gone from the main floor down to the staff area on the bottom basement floor, John Tully is also the artist who put the mural, the Elson Wonderland mural on the wall. It's amazing, so very talented guy. Moving on, then 4.2, Cheryl did not have a support services report. They've kind of been working on the same things since last time. And then I'll move into 4.3, which is building projects. That's maintenance, which is under me. Just a few things that Greg wanted to point out. There's a document attached to board docs. But if you go down to numbers five and six, we are hoping that the vendor will be able to install the fencing in early March that we've been talking about for about four or five months now. Maeve, you had requested, especially we get that generator covered up in the front area that's not the most attractive thing to look at as you walk into the library. So we hope that that project will be done within the next couple of weeks. And then the other big thing that we're looking at right now is furniture replacement. We have some pieces of furniture that are worn out. We have some others that we're having cleaning issues with. And we also have further the friends that had purchased a couch about four or five years ago were sort of moving away from furniture where you have a lot of people sitting close together in close proximity. We don't think that that's going to anything with COVID is going to change over the next few years. So we needed an update anyway. So we're going to be moving through some of that furniture and we'll be talking to the board as we get through that as well as the foundation to fund those projects. So those are things that we're sort of inventorying right now and trying to figure out what we're going to do with. So and then last on the director's report would be monthly statistics. I think everything is sort of going in the same trajectory that it has been. We still have another this month and one more month that we're sort of in this COVID state. And then COVID unfortunately becomes the norm where our stats have all been sort of deflated compared to normal times. I guess any questions about our statistics that we've kept? It looks like there are no questions. Okay. So then moving on then to the early eighth on reports just turning this over to Nancy mentioned for any updates on the Monarch Library system. Thank you, Maeve. The Monarch Library trustees met on February 9th. A strategic planning session is going on using a tool. And we have been asked for input into that. So that's forward thinking and assessing. Also the job descriptions are being updated for Monarch. And that's also happening. Monarch Library, the library system I should say published its annual report. And I thought as Meade is one of the 23 libraries and Sheboygan County, one of the four that you'd be interested in seeing in addition to what Meade does, what's happening in that larger group. Circulation last year, 2.4 million items that's very good among the children. We had in the system over 1 million minutes of reading. And that so that's a good statistic to know. There were very close to 500,000 items shared between or should be among says between. But among all the libraries, also the digital collection over 300,000. So applause to Meade and the other 22 libraries for what they've done. A comparison of statistics from 2019 and 2020. In one perspective, statistics of course have dragged because of COVID, but in the other perspective services have continued and done very well in spite of the fact that we had to deal with the COVID. So that's it for the Monarch system. Great, thank you Nancy. Anyone have any questions or comments for Nancy? I remember, you know, when I first heard about the whole plan for how the Monarch system board would meet. And because we're now 34 libraries, there's a lot of travel that used to be part of these meetings. And I think one of the benefits for you Nancy is that you've been able to really be able to do some of these meetings virtually. Because once upon a time you were like on the road for a couple hours before showing up for a meeting. So that's one of the benefits. It'll be interesting to see as time goes on whether or not they will continue the option of virtual. For you, I hope so. So. Well, I have to say that I always enjoyed before we weren't meeting in person going to the different libraries because, you know, they all have their own personalities. There's wonderful artwork. The people who are working there are so hospitable and helpful that I miss that part of it actually. Well, the fingers crossed and toes crossed that you can get to that point again. Maybe by the end of this year. So thank you again for representing Meade Public Library every month on that board. It is so good to get your input and we know that you speak very strongly about the wishes and the needs of Meade Public Library. So thank you, Nancy. Moving on then to the 5.2 Meade Public Library Foundation. It did meet last, actually yesterday.  She was elected president of the foundation in the previous meeting. And then yesterday, because she had been vice president, we elected John Donovan as vice president. Some of you may know him in the city of Sheboygan. So that entity will now have some, a good team to lead it forward in this new year of 2021. The other highlights from the foundation meeting is the fact that there is a lot of interest in trying to have an in-person Wisconsin Academy event perhaps sometime in the fall. And then if, you know, things with public health seem to suggest that that's not a wise step, the foundation will look into an alternative option for that type of programming. And the same for the Yuletide reception in December, there's great interest in having that wonderful tradition come back. But it will kind of depend on where we are with public health issues. I'm just looking over at our library director, Garrett, to see if there's any other key detail that I needed to share from the foundation. But I think those were the main highlights. Next up, 5.3, the friends of Meade Public Library. And I will turn this over to our, to Sydney, Meade. I think she's, there she is. The friends actually did not meet this month. There wasn't a lot of business. And so they decided not to. But I will report, because it just happened today that the friends have agreed to host a furniture sale to kind of help us with offloading some of the excess furniture that we have at Meade right now, as we've had a replacement furniture put in. So that will be a nice way for us to be able to open up all of those meeting spaces. That's all. Sydney, also, wasn't there a book sale scheduled for April? Yeah, so the book sale is scheduled for April. They have dates selected and are moving forward with that. But in the possibility that it's not able to go forward, they have backup dates of May selected as well. Any questions or comments for Sydney? If you could also just share with the wonderful friends that the additional signage in their bookshop, right at the entrance, seems to have garnered a lot more interest in people stopping by and purchasing some of the books. So, or maybe it's just everyone's feeling like spring is really here and they need to stock up on a more personal collection of books. But it looks very active. The last few times I've been going into the library. So with that, our upcoming meetings, we have no meeting in March. So our next meeting will be April 22nd at 3 p.m. And at this time, would someone like to make a motion to adjourn? Don't move. Yeah, Meg, Alvin, has moved. Is there a second? Then I'll second. Mary-Lindon, you second. All those in favor, please say aye. Aye. Aye. Aye. Aye. Aye. I don't think anyone's going to oppose. So thank you all for a very good meeting and maybe have a wonderful rest of the afternoon. Thank you. Thank you, Mae. We'll adjourn. Thank you.


It's Scorpio season and that means it's time for me to give birth to a big baby microcontroller board. It's an RP2040 Scorpio. So this is a 8-channel Neopixel driver. It uses PIO. That's what the Scorpio stands for. It's got the level shifters and this is the output section. It also got like STEMI QT and 8 megabytes of flash and you know we set button and all the feather stuff you know and love. So coming soon to the Adafruit shop we're gonna have this ready for sign-ups but great for anytime you want to drive like 8 IOs with 5 volt logic and if you like even on the back you can change it so it's an input. So folks who want to turn this into maybe like a logic analyzer or something you could have 8 channels in all consecutive GPIOs.


Now recording, welcome to the Open Tracing and Specification Council meeting. Everybody, good to see all your lovely faces. We've got a fun presentation today from Jonathan Caldor and Michael. Boy, can I not pronounce your last name. That's exactly right. You got it. Yeah. Great. From a long line of, can't pronounce your last name, family tree. But it'll be on tracing at Facebook. And rather than ramble on myself, I'd love to let Jonathan and Michael introduce themselves and then kick it off. So welcome. Hi. So, yeah, I'm Jonathan. I used to work on the canopy team on Facebook. I recently moved to another team. And so Michael recently joined the team as well. And so we're using this also as an opportunity to kind of like move me off of the advisory board and move Michael onto the advisory board as well. And so yeah, Michael. Hi, I'm Michael. I think I'm unmuted. Yeah, no. So I'm Michael. I just recently joined Facebook and the canopy team. Before that I was actually at Comcast for a while, but I actually worked on one of the things I worked on there was actually our sort of internal tracing system as well. You know, which was, which was the sort of X trace if can, you know, dapper style one. So it kind of got a, you know, some experiences with both worlds. So, which is kind of interesting. Yeah. Cool. So, yeah, let me attempt to share a screen, which is like the last frontier and giving a presentation over the internet. And so can everybody see this. Yes. Okay. So we're going to be talking about canopy which is Facebook's distributed tracing and analysis system. This is kind of an amalgamation of a couple of talks. We published a paper in SSP 2017 last year, two of our engineers talked to Q con in New York. And we're focusing this mostly on the instrumentation and representation of canopy. We'll talk a little bit about some of our distributed or some of our trace analysis and trace analysis pipelines. And so we'll be happy to sort of like talk more about that but we're kind of focusing on the representation side. Yeah, canopy is an umbrella term for a wide set of things for tracing at Facebook. The encompasses both are single and cross system tracing pipe or single and cross system analysis. We have instrumentation available in a number of languages CC plus plus Python Java and because Facebook PHP. Other languages are sort of supported through C or C plus plus bindings. Our instrumentation is integrated into both our like common RPC stack that's shared across all services. It's integrated deeply into our dub dub dub stack. So the overall page load process both on the client and the server, as well as some other common pieces of infrastructure. And then we're also sort of able to ingest data from other sources. So we have tracing in our mobile applications. The Profilo, I think is the name that we open sourced it under. And so we're able to ingest data from there and incorporate it into other traces through our backend systems. It also combines an extraction and processing framework. And so given a trace that we receive from some source, we're able to run custom user code to extract trace patterns, information from a trace, write them to datasets that we can then do aggregate analysis on. And then there's a separate team that works on performance visualizations. And so they work on both single trace and aggregate visualizations for these traces. Great. So can people see this again? For a second. Yeah, PowerPoint is not happy with screen sharing. Okay, I'm just going to share the whole desktop and hopefully we won't get infinite recursion with the preview windows that also pop up. Okay, so canopy is a little bit different than sort of other like span based tracing systems. We're an event based system that we then in our backend will take those events and parse them into a higher level model. Because we're not span based, we also sort of have explicit edges between points. We still enforce that the overall structure must be a DAG so we don't allow any edges that go backwards in time. But otherwise you can have sort of like edges between arbitrary points within your trace. And then we also have sort of metadata that every other tracing system has. We've sort of layered types on top of them and I'll talk a little bit more about what this means later. So this is what our overall model looks like. We have sort of five basic objects in our trace. We have the overall trace that encompasses everything. The trace is broken up into a number of execution units in a like explicit manner and execution unit represents a sequence of or a sequence of trace data that comes from a single clock. In practice this usually represents either a single host or a single thread within that host. But it can be used for modeling other primitives as well. Within an execution unit and execution unit contains a number of blocks. These are sort of the closest analog to spans. A block represents some duration of time. It contains a start point and an endpoint and then zero or more other points that may occur within the trace. And then a point is just sort of an instant in time that captures some single instant and then edges connect two points together. And so then all these objects can also have sort of arbitrary metadata attached to them. And so I said that we're an event based tracing model that we parse into a higher level model in the back end. And so here's an example of what I mean by this. Suppose we have a simple case with four events. This is sort of the classic RPC call and response events. These sort of get broke. You know, we have a call to receive on some back end service that back end service and some response and then the parent service gets a or records a complete event when it received the response from the RPC call. We're then sort of we take these events and interpret them as okay there's some parent block that the the call and complete are part of. They call to some other service which is generating a block at the start in the end. And then those are all both within some sort of execution unit as well. And so this is the base original instrumentation that we've had. We've since extended these events over time as we've added new model elements. And so as we introduce execution units and explicit edges, we've introduced we've introduced events that allow users to create these within the trace themselves. So one benefit that we get from having this decoupling of events from the actual model is that we're able to update cross system instrumentation without having to. Carefully arrange releasing instrumentation versions to both services at the same time. In practice, given you know service release schedules it's impractical to assume that the instrumentation version on both sides of the boundary are going to be the same. So you need to have some compatibility across the boundary. And so this decoupling allows us to say interpret events on one side of the boundary differently if we update the instrumentation while keeping the instrumentation on the other side the same. And so then this allows us to handle sort of all combinations of cases where service A and service B may either have old or new instrumentation or potentially even different instrumentation entirely. So coming back to the explicit edges, this is probably the like biggest single difference between and a span based model and our model. All of these things can technically be represented within spans. We found the benefit of having explicit edges meaning that we haven't needed to change the structure to add additional features change the structure of the trace to add additional features. And so for this I can walk through like one example that we have in our current system. So we trace through browsers, you know, which includes both the client side JavaScript that's executing, as well as the PHP running on the server. And so you can imagine some sort of JavaScript execution unit, which is recording all the JavaScript that executes on in this particular page load. And so there's actually three separate hierarchies that can occur within or three separate causality hierarchies that can occur within the trace here. So you can have the sort of standard one which is your JavaScript is making some sort of remote call in this case fetching a resource and then at some point that resource will return and JavaScript is able to use it. However, we also have a second causality hierarchy, which is the function hierarchy.  And so we found that, you know, this is useful for sort of representing nested blocks so you can have one block that is entirely contained with inside the execution of parent block. And we're able to use edges to sort of say that this child block is part of this parent block without sort of having to represent it as the parent function is making an RPC call to the child function which is occurring within the same thread. The third causality is actually an interesting one and it occurs in like, I guess more and more languages over time but like JavaScript and other continuation based languages and so here you can imagine our schedule function queues up some future that will then be executed later on. And so in this case, like our causality isn't necessarily between the root functions that we're executing, but in this case we've scheduled some function and then we have some, you know, common infrastructure stack which is executing these or pulling these entries off of the queue and executing them later. And so we need to connect to the actual function that we're executing instead of say the parent process function because that may also be calling other futures that are not actually connected to the one that we scheduled.  But what we found is like they represent different, you know, different parent child relationships, and having edges and specifically types for those edges, allow us to say that the function, the function hierarchy relationship is different than the RPC hierarchy relationship is different from the continuation hierarchy relationship. So one other example where we found like having the ability to create explicit edges has been useful is representing application flow. One of the like common tools for understanding our traces is critical path analysis, particularly for browser traces. And so we ran into a problem where we end up with say our JavaScript requesting a couple of resources and our JavaScript thread tends to be fairly busy. And so we would get traces that look like this and we wouldn't necessarily know which resource fetch is on the critical path. And you could actually argue that, you know, either one of these is potentially the resource fetch that is blocking. However, if we have additional information from our application, say, when we actually end up using these resources, we can actually see in this case that we use resource to immediately but resource one, we actually don't need for some, you know, non trivial amount of time. And so we could have actually delayed resource one significantly without affecting our overall time, but it looks like resource two is actually our blocking resource. And so in this case we want to represent some sort of application flow that says we've received this resource or this result of this RPC call, but we don't actually need it for some period of time. And we should take this into account when we're actually computing the critical path. And so here when we're computing the critical path through resource one we can say that there's actually a slack in the critical path corresponding to the length of this required for edge. This is also allowed us to experiment with different representations of application application based logic. So for instance we've also experimented with saying that, you know, certain events must happen in order for other events to even be considered. And so again in the page load process there are some synchronization points where we know that we won't receive, we won't process a receive event until some other synchronous event has happened. And so we can say that this synchronous event is going to be blocking anything before it. And so it's a prerequisite for anything else that happens afterwards. So coming back to our metadata, we have sort of the standard string string annotation map that's common among a lot of tracing platforms. These can be attached to any object in the trace. So points, edges, blocks, execution units or trace, all of them have a metadata object associated with them. We've made a distinction between what we call core custom and error properties. And so there are separate types and separate maps for each one of these. The distinction is a little bit fuzzy between some of these, but effectively a core property is something which is used by our back end to interpret the trace. And so for instance, the type of an edge is a core property. This also allows us to distinguish between annotations that users add and annotations that we absolutely must have for, you know, loading or displaying the trace. Custom is then sort of a general bucket for any trace data or any annotation data that users add through their own instrumentation. And then error properties are typically used for noting errors in trace construction as opposed to errors in the overall execution of the trace. And so for instance, we might use an error to indicate that the trace instrumentation never closed a particular block versus like an RPC call returned a particular error. The other feature that we have is typed counters and so they're an explicit separate type from the string annotation map. And so these are counters that have a numerical value associated with them, a particular type and then also a precision. And so this allows us to say things that like 1024 bytes is distinct from 1024 milliseconds, which is distinct from 1024 kilobytes. But it does allow us to say that if the user records 1024 bytes in one place and one kilobyte in another place, those two values are actually equivalent. We've also extended it over time to more types as we've needed them. And so we've introduced sets of strings that can sort of be appended to over time. We've used this in particular on say like execution units or traces and then also for stack frame. So capturing either like sampled profiling data or the stack frame at a particular RPC call. So coming back to, I guess, putting this all together in between like the metadata and our events, we've run into, I guess, some fun challenges in modeling. And so going back to our old instrumentation where we just had a call, receive, response and complete events, each of these has some associated metadata with them. And one problem we ran into was, well, when we wanted to extend this to blocks and points and execution units, a call event now does more than just create an edge. A call event actually ends up creating a point and an edge to the RPC service that you're calling to. And so there's an open, there's a question of where does the metadata actually apply to like, does that metadata apply entirely to the point that it creates? Does it imply entirely to the edge that it creates? Is there a mixture between? We sort of made the decision that a call represents the edge and the point is sort of a side effect of that. And so the metadata gets applied there. But this does mean that, you know, when users are using the old instrumentation, they can't actually attach metadata to the original calling point instead. So this is why we sort of extended the instrumentation over time to allow more places for this metadata to apply. And with that, I will hand it over to Michael. I don't, do you want to try sharing your screen instead or do you want me to walk through the slides as you talk? All right, let's see. That's, that's like, I feel like this, this is dangerous either way. I'll give it, I'll give sharing my screen a shot. Yeah. There's a green thing. There's a green thing. Oh, the one, literally on hovering over. All right. All right. Did anything good happen on the other end? Yeah, I can't believe it worked. All right. Okay, let's let's just. All right. Thanks. So, so yeah, they kind of pick up from what's that? Oh yeah, presenter. I know how to use PowerPoint everyone. Oh, don't do that. Man, disaster averted. All right. Great. So, so yeah, to kind of pick up from where, where Jonathan left off. You know, it was kind of interesting before I, before I came here, you know, I worked with. We had a, well, we've open source, Comcast has open sourced it, but a, you know, a very span based sort of a span based tracing system. We called it money as in like follow the money.  Like the money bank was where all the traces lived and stuff. So it was fun. But the, you know, we did run into like some of the modeling issues that Jonathan was talking about and actually two in particular that like, we kind of ran into there and then read the canopy paper and then I quit and came here. But like, you know, that we were like, Hey, this, this could actually be useful were, you know, one is we have these sort of situations where we had a trace on a particular system.  We just wanted to, to sort of attach like a profile of what was happening on that system at, you know, various levels to the trace. And, you know, kind of the best way we could think of to do that in the sort of span based model was you'd have a sort of a top level span that represented like the entire scope of the execution and you sort of start profiling and then end profiling when that thing closes and try and attach that profile to that top level span. But then it was kind of like interesting. You had to know that span was kind of like special, right? Like that was the one that, that like had the profiling information, right? And it wasn't that bad, but it was actually kind of clumsy as far as the tooling we were building around it went. And in sort of the canopy model, it's actually kind of more natural to just annotate the execution unit that represents that, that like request handling, right? Just because we use that sort of naturally represent, you know, here's the entire span of processing an individual request, but not span and tracing. And another interesting one was if you just had to put some work in a queue and you wanted to understand, you know, how long it was in there, right? And when it came out, you know, that's actually pretty naturally modeled by another execution unit with points for NQ, DQ and edges for causality, right? Whereas like if you sort of put it into, you know, you could model that as a separate span and this starts, you know, starting into the span or when it goes into and out of the queue. But it sort of means a very, very different thing than like most of the other spans do where it's like an actual RPC graph.  So those are a couple of things that we actually did struggle with from a modeling perspective that we were actually pretty interested in when we read the canopy paper to sort of help us out with. So, you know, just kind of worth noting it was kind of an interesting thing to sort of see it from one side and now start to see it from the other. So that said, I wanted to sort of move on a little bit to talk a little bit about what we're doing now and sort of where we're focusing.  So one of the things we're focusing in on is getting the back end APIs that, you know, Jonathan alluded to into a point where they're safe and like clear and usable. So that means for us actually sampling isn't enough. We also need rate limiting and I'll kind of go into that a little bit later. We also want these sort of somewhat tailored, you know, high quality APIs and instrumentation layers that are for, you know, back end use cases. Right now we're, you know, what we really want to be is where we have just sort of like most of the complexity in dealing with, you know, the underlying model is handled in sort of instrumentation layer and an end user of the system just sort of has like a small set of functions like log a point, right? And, you know, the instrumentation layer just says, okay, well, here's the active execution unit, here's an active block and we'll put a point on it. Or if you do need to put a new block alongside of, you know, the sort of default one, you can sort of do that in an easy way. You know, so what we really want to do is just make tracing on the back end and just like really, really easy for folks that are building back end services. The PHP instrumentation we have that John had mentioned actually kind of does that to an extent already. But, you know, we sort of expose a lot more of the underlying guts to back end folks right now.  So, you know, one of the things that we did, we're just working with one of our teams that has some like really, really stringent sort of perf requirements as far as memory usage goes and then sort of, you know, they really worry about things like thread contention. The sort of flexibility in the underlying model is actually going to let us fairly easily create like a fairly tailored, you know, hey, if you need to, if you have a really high performance, you know, RPC system and like you don't want things going on behind the scenes that could cause additional thread contention or memory allocation, like use this API. You know, so that's one thing. And then the other thing we're working on is actually a sort of a revamp of, if anybody's read the Canopy paper, there's a, I think they're referred to as, I think, custom extraction functions or something like that. But it's essentially a DSL for working with traces that happen to run in our back end where we're working on sort of revamped, you know, expanded version of that, that'll run, you know, in a separate set of processes elsewhere and use, you know, then be based on sort of Python rather than this completely custom DSL. So it's really the two things we're working on now. That's actually super important for us because we tend to look at traces in aggregate a lot and we just sort of compute like summary, you know, information about traces often, right? And, you know, the stuff that's sort of covered in the paper, but I think the way we're going to be doing it is, you know, going to be different. So on the safety and API clarity side, like, this is kind of a, this is sort of an overview of what the instrumentation stack really looks like for us. At the bottom layer, we've got like a layer of sinks that do nothing, you know, just yield serializing the events that Jonathan mentioned and flushing them somewhere. You know, we have sort of an internal Kafka-esque system that, you know, is used. On top of that, there's a trace model that really just, you know, represents that trace model as an object model, but doesn't let you do things that don't make sense, right? Like you can't create like a block on a point, for instance, right? But it just makes it makes it a little bit easier to work with. But, you know, when you do things that they're, you know, when you do things with that model that will give you pointers so you can sort of keep reference to them in your code and then flush them, you know, it'll flush things to events usually right away, but you can do some things with more of that if you need to. And then on top of that, we've got sort of a set of code that deals with creating instrumentation layers, right? Because we don't want people to have to worry about things like, you know, propagating context, either through thread boundaries in their system or, you know, across system boundaries. We don't want people to have to like understand, really, really understand that trace model deeply and understand which parts are active, right? We just want an underlying instrumentation to take care of that.  So that's not great. But then the whole idea being on top of that, we've got this sort of instrumentation kit to build back in instrumentations. We've got a set of instrumentations built on it that we are that are either built or will be building on top of that. And then what we really want to have most folks leverage is sort of this high level API that really just lets them do like a few simple things, right? And all of the more complex pieces are handled by instrumentation, right? Like at the end of the day, you know, so to lack something more like a logging framework, right? Like you log a point or you create a point and it goes into the right place on the trace, right? It goes on the right block or execution unit and, you know, and so on.  And then if we need to do something more sophisticated, you know, people would have to go sort of layers down in this API stack to do it, right? And then like I mentioned earlier, another thing that we're talking about doing sort of on top of this is creating just a really, a really performant but much more constrained API that sort of just does that RPC trace model for some, you know, particular use cases. And it's kind of nice that like this, not only the underlying, you know, event and object model sort of gives us that flexibility, but, you know, sort of the instrumentation that we've got built up. Let's just do that, too. So this is a quick overview of the other big chunk of stuff we're working on now, which is actually borrowed from the QCon presentation of two of the other gentlemen in the room. Joe, Aaron, Edison, say hi. Yeah, we're good at video conferences over here. So basically, like I was saying, this is really a domain-specific stream processing engine, right? And language for processing streams of traces. And if you look at sort of the way that we tend to use trace data, we do a bunch of different things with it. One of those things is obviously just like looking at an individual trace, which is, you know, kind of useful if you know which trace to look at. But if you don't even know that, you know, or if you want to compare, you know, data and aggregate before and after some event, right? Like, you know, a deployment or something and see what's happened, right? Or if you just want to compute, you know, summary statistics off of something that can only be derived from, you know, a trace, right? You know, that actually ends up being, I think, a more common use case for us than just looking at an individual trace. So really with this, so yeah, so this is a domain-specific stream processing, you know, system for getting at that sort of stuff, right? And we've got a bunch of things that are built on top of it.  And that's run by this whole set of sort of machines that will go and run those on a per sort of use case basis, right? So it's sort of, you know, in the old system that's mentioned in the paper, we kind of ran the moral equivalent of this all in sort of one, you know, set of infra that was also doing a bunch of other things. You know, this one, the use case are actually split out, separate sort of, you know, chunks of machine for different, you know, use cases different, essentially going back to different users of the system, right, that are going to be doing vastly different things. So that way we get some more glad isolation and they're not stomping on one another. And then we sort of export summary statistics to, you know, an internal database called SCUBA and folks build things on top of that. So those are really the two big things we're working on for the time being. And then, oh yeah, the other important thing about this that I didn't mention, which is different from the stuff in the paper is that this actually does allow us to do sort of ad hoc queries, right, which was not possible with the old system. So that's super useful if you, especially when you don't even know what you're looking for yet. Yeah, I got one more image after that. All right, let's see. Yes, all right. This is the slide I stole from Joe. So that's the stuff we're working on now. And this is just some stuff that sort of, you know, kind of keeps coming up again and again that we're thinking about, don't know to what extent we'll actually end up really tackling this. But I think they're interesting things that normally I think we're thinking about but have sort of seen other folks thinking about too. So I thought it was worth mentioning. One of them is how do we sort of safely do more arbitrary context propagation at scale. And the interesting thing is scale here is actually more the diversity of teams and workloads than sort of the scale of, you know, our overall infrastructure. You know, there's sort of a, and I think anybody who's done this at like a big company has almost certainly came across the use case where somebody's like, hey, there's this stuff that will like magically propagate like this idea around and like I want that to propagate my session idea. So, you know, the notion of having a more abstract way of using the underlying, right, wherever you've got the tracing instrumentation of being able to propagate context through which I know is, you know, in open tracing with baggage and the paper that, you know, Jonathan Mace I think wrote like is has some really good stuff in it. But I think one of the interesting things is like once you open up that capability, you know, like how do you keep people from doing really, really bad things with it and such bad things that you end up having to turn it off is kind of an open question. And I think it's kind of worth thinking about like, is there a difference between, you know, that sort of really common use case of like, hey, I just want to propagate an ID sort of within my system boundaries for some sort of session or something versus the broader, you know, I need some context that really might propagate across a very wide set of system boundaries and just in terms of like safety for some sort of context problem. And then another one is, which Jonathan mentioned earlier, like one of the interesting things that I think this does fall directly out of the model, the sort of canopy data model is that it's actually really, really good at doing single node traces as well. And we get like really detailed traces of mobile clients and dub, dub, dub, and folks actually want to get like really detailed traces of their own back end systems, you know. But it's like, well, how do you then take that really detailed view of a little piece of the system and working into an overall, you know, distributed trace that that's probably broader? You know, you sort of create a lot of noise for people that just want that broad view, but then sometimes, you know, the person that's like looking at that really detailed view might want to look at something, you know, a few layers back. So there's actually a couple of different ways that we've talked about modeling that and there's some different things we've talked about doing that, doing with that in terms of like this tooling, you know, and so on. But it's actually kind of just, you know, it's kind of built the question of how we actually get like a good solid end to end trace while still having like these chunks of like really, really detailed trace at various parts of the system in there and, you know, because they really are different things with different audiences, you know. So yeah, so those are two of the things that I think we're thinking about and, you know, may or may not do anything useful with. Cool. So anybody have any questions? That was great. But I don't want to ask on this group because then it's going to take forever. Well, Jonathan is available for all questions at any time. So just, you know. I have a question about metrics and aggregates, actually.  But are you doing kind of all metrics extraction based on the system or do you have a totally separate metric system? And if so, are you kind of sharing, are you using the tracing system for context propagation and like how would those two things relate to each other? So we have, there's an independent metrics-based system for sort of like operational management. The traces will tend to, the traces can share some of the data from those metrics. There's some caveats usually there where like the metrics capture our system level but say we want to capture request level metrics within a trace. But we can sort of pull the same like overall system, CPU utilization, things like that. What was the, oh yeah. And so then that has a separate aggregation piece because they're kind of collecting at some regular interval over hosts whereas we're sort of like very fundamentally kind of required base. And then there was a question about context propagation, I think.  But the degree to which you may want to dimensionalize some metrics is at all happening, you know, that context tends to get propagated in the tracer. I was just wondering how that relates. Yeah, so one of the places where there is actually overlap is if you want to understand, say, the overall global efficiency of a particular system and in particular the resources that it's utilizing as a whole, you do need to kind of look at the resources captured through traces that, you know, the trace starts at some particular point and then you're looking at sort of like what are the resources used by this particular request and then aggregating over all of these requests in some sampled fashion to understand like request-based utilization through the system. We're currently using tracing for that. Like this is fundamentally at Facebook like context propagation and tracing are fundamentally tied together for better or worse. And so that's where, like as Michael said, like we end up with use cases where people are like, man, I really need to propagate a context. Like let me turn on tracing and we're like, you know. Yeah, and to that, like we don't have a decoupled generalized context propagation system. And I think, you know, to that sort of like how do we do that safely with the number of teams that we have is sort of an open sort of operational question that, you know, we'll have to think about and see if we can tackle at some point. But yeah, it's interesting when you have this many different folks sort of that, you know, yeah, it just gets interesting. Yeah, I hear that. Thanks. Anyone have anything else? So on the safety piece on context propagation, like you've kind of given us the problem. And I totally understand the problem of web through this. Do you have any thoughts on how you're actually going to solve it? I have some thoughts. Although the interesting thing I realized I somehow when I redid these slides, I skipped my slide on rate limiting, which is actually the important safety piece we're tackling now. So actually maybe I'll just go over that really quick.  So we're actually adding rate limits on a per trace and trace size, you know, before we can actually sort of start new traces to cover that. So that's actually one important safety piece. As far as generalized context prop goes, you know, I think it's something we've been talking about for a while and starting to think about, I do think that opening up a sort of completely generalized system and like saying, you know, like any engineering team and an organization of our size is free to like go and attach baggage to this thing is like a non-starter.  You know, and then I think if you look at some of the safety that was sort of in the paper that Jonathan wrote, I don't have a note piece around, but, you know, essentially it comes down to like having a principled way of like, you know, capping the amount of size, the amount of data that gets propagated. But then it sort of has this like downside of like, oh, maybe you really, really rely on a particular piece of data, you know, now it's not there. I think to the extent that I've thought it through, I do think it's really, really worth thinking about how do you separate out the use case where somebody wants to propagate an ID within their system bounds and then sort of attach metadata to it after the fact that can be processed by like some other system, right, and have it emit that data, but sort of like then not have that ID cross, you know, be propagated outside of the bounds of their particular, you know, set of systems and then separating that out from like the generalized context prop, which should be very strictly controlled and really only used for like a specific set of blessed things with like a decent amount of process around putting things in them upfront, right, and like some like, some like set of configurations that like can't be changed outside of like, you know, review by some accountable team. So, you know, to the extent that I've thought through it, like that's sort of where I've landed, but, you know, that's just, we haven't really, really worked on it heavily yet. I've tossed around the idea of like prioritized namespaces this realm, but I haven't actually implemented any of that yet.  But then I think you do sort of run into, at least in our world, we kind of run into this thing of like, well, okay, what if, you know, the data in your most prioritized namespace is larger than like the amount of data that like the most, you know, sort of the most conservative team is willing to accept.  Yeah, I mean, I think this like, it connects back to the, you can have a really flexible API, but you also want something higher level and very restrictive that most users interact with.  But each www request should have its own ID and like, making sure users understand like what are the boundaries where things crossover and are able to do it in a safe way. I think like we're it's a very, very open question on our side of like how to make that work. Yeah, the boundary issue is just pernicious for any form of context propagation.  Yeah, yeah, it's, it's, yeah. Michael, can you elaborate on rate limiting? You said you do it after sampling. So what happens to the trace if you start rate limiting? Yeah, so, so the way we're planning on doing rate limiting is we're planning on still doing it at trace start and not killing traces that are in progress. So like, for instance, like, you know, from the point of view, let's say that you're an individual node and you know you have one in a thousand coin flip, right? But when we configured that coin flip, we had like two nodes running and then for some reason now there's a thousand running. What would happen is you do the coin flip, the coin flip would pass, and then there'd be an additional rate limit check and we would have a set of centrally configured rate limits that say, essentially, okay, this, you know, this this policy gets to start five traces per second. It would check the rate limit after that and the rate limit would then fail. And that's also where it would check the trace size rate limit, right? So, so we're only going to do it at start.  Right. Yeah, that all. So does that answer the question? Yeah. Cool. Thanks. I mean, we've done something similar, but not for regular sampling, but for like specific box level sampling that people were abusing. So we rate limit those, but for the reason we didn't do it for that the regular sampling is because we extract some extrapolations from from the statistics we get from overall traces. And, and there the probability of sampling actually very important. So if you start rate limiting that you cannot do extrapolations anymore. So, yeah, that's something we'd have to watch out for because we do it, we do it and we do that sort of thing in some cases as well. You know, the intent of the rate limits is just for sort of our own safety. Like we don't mean we don't intend them to be things that are going to be hit under like normal operation, you know. So and we're going to monitor them, right? Well, monitoring on there, somebody like hits them in that sort of situation will let them know, right? But, but yeah, it's just sort of something that, you know, we sort of can't get away with anymore and continue to onboard people without being kind of skittish. They're kind of good for like new use case where when you have, if we want to have like, we're exposed to like dozens or hundreds of teams, we can't manually go in and understand like what their limits are going to be or maybe volume of data or number of traces or there are like their requests per second or all of this. And they might not even know. So like, there's just so much overhead in this space that if we can, for the start case, have like a same bound or here's the amount of traces you can get or the amount of data you can send us. Like our system doesn't have to fall over and they can iterate on that. And then they can kind of move quickly without us. It's also good for the case of maybe not the, the new, the new whole tracing scenario case, but maybe I have this distributor request and somewhere in that request, we can't continue on because it didn't have instrumentation or something like that. And now maybe it's like a huge, like I said, everyone hits and they want to add a ton of like instrumentation and now their service has 10x the amount of data pumping out that it was before. And this gives us like a good way of understanding. Oh crap, maybe sampling was the same, but the size of all these traces went up significantly and we need to have a quick pushback mechanism. So our service doesn't fall over. So it's usually like those are the scenarios where it's like allowing us to move like quicker and not fall over and then come and react to it and then have someone be like, okay, what do we do? Do we do this properly? Yeah, one of our, one who isn't here right now, he's, he was working on more of a sampling based approach where we dynamically changed the rates based on how much they're outputting. I mean, we have both, as Yuri said, because of the debug override that we allowed in the system. So like, we still need that case. But yeah, I mean, changing the probability is a nice way of doing it. If, if you can, it seems like it's pretty static right now. Yeah, you take into consideration like size, like, does that, like, probability will be, it depends on like what you care about, right? Like if it's not, if all your traces are the same size, like, Yeah, size is an interesting point. We haven't done that, but we've seen it definitely. Yeah, that's a big, like, we started with like number and we're actually transitioning more to size, because our colleagues tend to be on, or sizes is more, is a better indicator. We have such diversity in what traces look like. And especially when you're, when you're not doing the case where it's a new tracing scenario, but updating an existing one that just a ton of like an updating existing source that a ton of people hit, it's really hard to understand, like, is this going to make us fall over immediately? You might have someone who's like 50% of all of our traces, but they're only like 1% of all of our data. But if they were to increase their data significantly, like we might just fall over immediately. And they might not understand that we don't have a good pushback mechanism. Yeah, are you doing the dynamic sampling just by propagating the sampling rates through the baggage? Oh, actually, we have like, we have like a remote sampler. It queries basically on a minute by minute basis, I think. So it's like we, the ingest side is sort of calculating the throughput from multiple areas and figuring out a good strategy of probability there. Yeah, yeah, that's good idea. It definitely adds complexity. I mean, I didn't write it so. Yeah. Facebook has something similar where we'll like get a constant volume of traces and we'll adjust the sampling rate accordingly with some feedback mechanism like running every couple of minutes. And that sort of has the same like, you know, there are, there are challenges with it. Yeah. Well, the main challenge is that there are some workloads which are periodical. And so when they're quiet, their probability kind of climbs up because there's nothing coming and then suddenly boom, they flood you. Yeah, we've gotten hammered with things like that, particularly around like aggressive rollouts to where somebody has like a high rate during testing and then it gets rolled to production. And there's like four to seven minutes that are not fun. And then the ability to adjust. Right, I think we're out of time. Yeah, that was that. Thank you so much, Jonathan and Michael. That was a great presentation. We'll be posting that on the internet and see you all next time. The internet. We'll be propagating it. Yes. See you all later. It's been fun.


All right, a few closing remarks before we depart. I hope you all are as excited as I am for a taste of Detroit tonight. It's our all attendee party sponsored by Cast and by Veeam and Snake, and it's from 6 p.m. to 9 p.m. Come reconnect with new and old friends and enjoy the night's entertainment. And just like how cloud native is cloud agnostic, this evening's event allows you to be party agnostic as you visit any of one, two, or three of our parties. That's right. So the parties will be at the Grand River Ballroom right here at the Huntingham Place, the Detroit Princess River Boats, and the top of the punch at the Windham Hotel. I can't wait. And I hope I run into all of you. Breakout sessions are going to start again at 11 a.m. And we hope you have a great second day here at KubeCon Cloud NativeCon. See you tomorrow for our last set of keynotes. Enjoy your day, everyone. Thank you. Thank you.


Everybody forced here ripping a two-box case of 2021 flawless football hobby for Antoine G Antoine buddy. Good luck, man. Here we go. Let's see a shield Let's see a shield We should send a legal of Christmas card next year though. Yeah, it's too late now Do I need to talk about what? I don't know these questions. I'm gonna focus on this break right now though. It's 2021 flawless football hobby. Here we go Antoine First box first briefcase What the heck? Look how this is packed. What the What are they doing? All right, I can't even get the other box out Oh my there's for sure a book in there. Yeah I've never seen it like that before that's so weird. Usually I have the two and then there's like no one I don't know what you need doing, but uh, this is gonna be a good box. I can tell you that much right now This is already a good box. We're starting off with a nice one Triple relic Roger Craig Frank Gore Garrison Hearst that one's numbered six out of 20 Craig Gore Hearst Player worn Craig and Gore Game worn Hearst All right some Niners legends right there Triple patch card Peyton Manning three of ten the sheriff It's player worn on the Peyton saw him in that dang David Beckham commercial so much So much Peyton Manning right there. All right, Steve Atwater career progressions out on number four 15 Steve Atwater out of four 15 rookie debut signatures Cadarius Tony three out of 20 Signature gloves Ron Dale Moore.  I think 15 of 20 on the Ron Dale Moore signature gloves It's kind of weird six Never seen that. Oh, that's a rookie Clyde Edwards a lair. That's for 2020 rookie signature gloves Clyde Edwards a lair one of 15 so previous year rookie That's fire. That's sick Two of four dual auto of Randy Moss and Justin Jefferson Oh That is gas congrats Antoine man. That is sick.  What a way to end the inner box right there Okay, I haven't seen Antoine in chat Lucas a Sapphire gem five of five tj Hawkinson jumbo rookie patch Justin Fields 12 of 20 Three-color jumbo patch Justin Fields All right. All right Well, we know it's a book.  Remember seeing that celebration a few times in the world cup. Yeah Jalen waddle three of 10 rookie booklet That is freaking awesome Beautiful waddle gigantic four-color patch That's awesome, man Is it from the is it from this number? Oh man, that's awesome though There's the pant leg. No one more One more. I just missed the trash case right next to me I feel reeling from the last four losses Okay Let me mourn a little Because I don't think it's gonna get better What if the jessi players are you guys? Well, I'm just doing I'm saying don't tell me that I need two wins, bro. All right. May 2nd Eiffel tower sleep snoring All right See you LeVar later LeVar rookie jumbo patch dwayne escridge three of 15 the scree the screech Dual patch patrick mohomes the second josh allen 21 of 25 You guys excited for uh Bangles bills this weekend. That's probably the biggest regular season matchup this year huge Player warren dual patch mohomes and allen potential buy on the line, right? Mahomes and allen very nice Follitt's performance is auto flipper anderson 12 out of 20. It's a march second March 6 whatever Sean's busy march 2nd rookie dual patch auto travis etn jr. 5 out of 15 nice etn Much better. Uh, well, I mean he didn't play his rookie season So i'm not gonna count that but awesome season for him this year really for his being his first full year in the nfl Stuff for him, man him and trevor James lofton auto number nine out of 10 penmanship auto james lofton Yes, it's bigger than that A kellen demand rookie dual patch auto one of 10 man.  Congrats ants one Let's get to the inner man.   This is a really good case. This is a really good case like very good Because the last card No, that's right.  Thanks again, buddy. Happy new year. We appreciate you. We'll get these out to you


Rwy'n gweld i chi'n ôl y bwaith i dda i fo. Y ddedig hyn sy'n mynd i ar ddigonion o'r bod yn profi. Felly pan i gred ovni ddau i dowel y Maes cyhoedd, ond i fynd i ddweud o'r hyfer skeptiaidd a gweld i gyd. Dwi'n edrychwch, rwy'n ben i'n ei ffordd, dwi'n fdefnyddio bobl yn maes i dda i gyd, felly wedi dod o'u profi yn ei ddigonion, ond gyddoch chi'n gweithio ffwrdd o ffyrdd o'r cynnyn ffyrdd o'r bobl, ond wedi bod yn ei ddyn nhw'n gweld yn ffwnt i mi nawr, yw'r iron ffymysunol ac yw'r iron ffym caelwyddi, ond yw'r compremau yn ei ffymu yn gwybod, yn ei ddyn nhw, i gael. Ie i fynd y cwrdd, mae'n mynd i ddifennu ar y ddweud yr un. Felly mae'r drwng sy'n gallu'n bwysig iawn. Felly mae'r un o'r 223 yw'r un. Mae'r ffordd beth sydd ar y ddechrau'r iron, ar gyfer y ddechrau'r iron, ac yna'r iron o'r ddweud ar ddweud ar gofynodd. A'n sizes i chi, mae dywedd ddod yn celol â ddwy�ch hwnnw i ddweud i llain yw i'w ddatu yn y llun o dd�lau'r corffwnau, felly nid i ddweud i dydym ni'n cael ei ddweud. ac mae'n gallu ceisio iawn ei gweithio gael synnal i ddarlwydau. A'r ddweud i ddweud i ddweud i ddweud i ddweud i ddweud i ddweud i ddweud i ddweud i ddweud i ddweud. Ond oes y cyfnod, mae'n gweithio'r cyfnod yma'r unrhyw gwaith o'r ffordd yma. A mae'n rhaid i'r gweithio ar gyfer cyfnod. A when you got a situation where the performance is so similar, I think you're then down to the sort of, well, the personal bit I suppose when it's kind of sound and feel and the kind of looks. And that's a key factor for me in any decision anyway. You know I think you're sort of first of all by with your eyes and like what you see. I think what Apex have done, they've stuck with a kind of shape that has been around for a few years now. They've repeated it. It's been a very successful model for them and I suppose if it's that kind of thing, if it ain't broke then don't fix it kind of mentality maybe. But I think it's starting to look maybe a little bit dated now. And maybe it's not even so much dated just in time for a change I think maybe for the Apex is a little bit of a change up. But in the 223, I love this old style and lettering of Mizuno Pro first of all. But forgetting that cavity back iron, lots of chrome looks really well. But it's more about the profile than the shelf appeal looks for me where again the 223 just seems to nick it and what they've done really well with this Mizuno Pro range, whichever in terms of the 221, 223 or 225, they've slimmed down the profile of all of these particularly that 225 and 223 where they've become really neat and tidy but still very playable if you like. And if you have a look at the sort of ad-address position now, you'll notice that there's a very much different top line between these two irons even though I would have definitely put them in the same category, very noticeably different. And the fact that you can see again ad-address even only in the 7 iron just a little bit at the back end of the apex is the kind of thing that may be the kind of the player who's looking, the player who's looking for the player's iron doesn't like to see that. They've managed to hide it away in the two, hide away the kind of the kind of bulk that gives you the confidence is hidden whereas in the apex you can still see it. So for me, if I'm looking at shelf appeal and then I'm looking at address and then I'm looking at kind of top line and even so much as the sole plate then for me the 223 gets it but it gets it on what I like to see in address and that mightn't be what you like to see. So for me a head-to-head is all about finding out the differences between clubs and one of the major differences is the fact that Mizuno split their set in terms of how they're manufactured. So let me explain eight through to pitching wedge is a forged iron very much exactly the same as what they've got in their 2-2-1 range in their forged player's iron. But then from a 7 through to 4 iron they introduced that material Cremoloi into the face. Now what Cremoloi is effectively trying to do is give you a little bit more forgiveness but also a little bit more in terms of ball speed but it's a different material. So therefore are you going to notice a difference in terms of sound and feel between when you go from an eight iron into your seven iron. I'm not really sure I like that idea so I don't know how close they've done in terms of being able to match those two things up. But for me if you say nothing feels like a Mizuno when it's a true forged iron but then your seven iron changes into a different material then surely there's got to be a change a noticeable change. And then with the apex obviously it's very much exactly the same through the set. So it's a forged body with a bit of tungsten waiting in there. So you're expecting a more consistent sound from the two. What I'm going to do first of all I'm just going to play for you some recorded audio. I've dropped a mic down to the floor. I've recorded some seven irons first of all for you to compare. So the first three are going to come from the apex. The second three will come from the two to three and see if you can notice a audible difference. Now it's worth pointing out that audio is recorded using range balls and I'll comment further when we get on the course with these a bit later on. But I'm also going to just play these nine irons as well because that's again where don't forget these now are the forge club within the two to three. And obviously we've still got exactly the same makeup in that of the apex. So let's just see if we can find any audible differences between these two nine irons. Now I don't know what you heard out there but certainly we've got two sets of ears in here which I asked Hannah again could she notice a difference and it was quite a difference. So in the seven irons I've got to be honest not a great deal to split them but when it gets down to the nine irons then yes there is a massive difference. All of a sudden you do get that much softer feel out of the two to three and there's a lot louder almost hollow sound again that comes out of this of the apex. So a big big difference in that end of the bag and I suppose arguably what Mizuno have done they've put that sort of sound and feel that you want in the shorter irons in your scoring irons and perhaps logically then introduced that other material into the irons where we need a bit more help. So in terms of sound and feel I would certainly at this stage now we'll comment a little bit further when we get on the course at this stage very much the two to three that would be the better sounding and feeling of the two irons. So the purpose of this video is to find out the differences between these two clubs and price is one of those things that differs and just scrolling through the internet I've just found out that the two to three when we first did the review a couple of weeks ago of the new Mizuno Pro range we only had RRP prices so never knew what they were going to be when they arrived at retail. So the two to three is now being sold at £152 per club in terms of UK British Pounds and the Callaway Apex is now down to £142 but that's a bit of a change because I think when the originally brought out the Apex they were a lot more expensive than that but there is literally £10 to split them in cost and I think in all honesty if you're buying seven irons at £70 I don't think you'd perhaps be swayed one way or the other when you're spending £1,000 on a set of irons based on just saving £70 so it's still down to other things other than price. That's almost gone in. Well one good thing to report there that the yardage that we've seen in terms of dry ball data is pretty much a resonated iron on a course exactly the same to the pin and I think they both travel very very similar. The one thing I will say yet again is for me I think it's the sound that's probably better from the two to threes. So these are two seven irons that I've just played by the way so don't forget this is not the forged end of the bag in terms of the truly forged end of the bag in terms of the two to threes. So there's a lot more similarities in these two seven irons but like I said it's the sound that's probably better and not the feel and there is a difference between the two but I would say that overall it just feels that little bit softer coming out of the two to three in these two seven irons. I just want to finish off by it in a couple of short irons and seeing how that truly forged club in the two to threes how much sound differs out here on the course. That's where the difference is massive. Is that going to be enough? Oh yeah. It's half decent. Pulled the first one. The first shot I played was with the apex and forgetting where the result was. The difference in these two yet again more so than ever at this end is the short end. That was two nine irons so don't forget the iron I just played that is a grain flow forged proper forged iron if you like. I would say in these videos not a big deal to everybody but for me I really like the feel of a forged iron and you can tell the difference between this nine iron and the seven iron in this set. I'm not sure I'm overly keen on that and why they introduced the Cromolo. I kind of my summary would be I kind of understand that bit so that extra bit of help if you like with ball speeds at that tougher end of the bag certainly four and five iron. I suppose I could get that but I much prefer the feel that comes out that nine iron. The overall summary would be that the two to three kind of ticks more boxes now than the apex did and I thought the apex was a fantastic iron. So just think they've done an incredibly good job with what they've put together in these two to three. It's a real interesting iron. It's a massive challenge that apex and anything that you think that might sort of fit in and around that sort of profile not really played anything better. And like I said the only thing for me I just wish that that seven iron felt as good as that nine iron did other than that the performance out both clubs has been really really good and I suppose like I said a lot of these things where you've seen the differences have been down to personal preferences but I do think putting everything together the two to three from this Mizuno pro range certainly edges it and it's a bit of a decent set of irons you know. Anyway right as ever I'm done thank you for Cardin Parr for allowing us to use this facility again it's been in great condition out here on the course as well surprisingly how dry it is considering how much rain we've had here in the UK. So enjoyed that hope you enjoyed all the testing as ever we've done our very best to get inside out on the course really put it through its paces and give you a fair evaluation of the two clubs. Right comments down below. Let me know your thoughts and I'll see you all very soon.