Starting now. Hi, everyone. Good morning. Happy Sunday morning here. Thanks for joining us at an early time and in this vast space. Hello, everyone. I'm Rupika Sharma. Welcome to the readership panel. That's right. This is the readership space. I'm Zach McKeon. Really happy to be here with Rupika. And today we're going to be speaking about readers, a very important part of our ecosystem of free knowledge. And something that we think many of you probably have some experience with. A question we come to a lot is... What is a reader? But actually it's a weird question, isn't it? Very weird. Who is a reader? Let's hear from the audience. Who's a reader is better? Hi, Anas.  Who else is a reader? Any other readers in here? Yeah? Yeah, cool. Yeah, so we are all readers, but we wanted to ask you just to shout out to us. Could you name another reader? Somebody else you know who's a reader. Your mom. Your husband. Your kid sister. Flossom yourself. Yeah, there are many readers. And we know them very well. And why do readers matter? Let's hear it again. Why do readers matter? Yeah, why does your mom matter? Why does your kid sister matter? Why does your husband matter? Flossom, why do you matter as a reader? If you stop to read, then content creators will suffer. Aha. Very nice. Readers are people, we are people, and we are also knowledge consumers, so readers matter. Readers matter. We like to remember that while this conference focuses so much on the people who make free knowledge happen, readers are a part of that system. They're the people that have motivations like us to learn and to grow and to improve their understanding of the world around them. And in that regard, they depend a lot on the work of Wikimedians like you. And let's see how many readers we have in our Wikimedia. A lot, as you see, and one of the most interesting things I find is how many languages we have in readers, 294. A lot. A lot. And how many readers we have from different projects? Wikipedia. And as we see, there are lots of other projects. Wiktionary, Wikicomments, Wikidata. Who here thought Wiktionary would be number two by Pageviews? Or who is surprised by that? Surprised. Yeah. Wiktionary. Currently number two on Pageviews. So this is just one metric here, like looking at Pageviews, right? Not unique readers. And this analysis actually includes machine visits as well. So bots and crawlers for the web. And so this is from actually last year. But we expect these numbers to change a little bit. As for example, Wikidata becomes read by more and more machines and networks. We're going to see this shift again. And how many readers are still missing? Well, by one measure, looking at data from last year, if we imagine that all of those 1.5 billion unique devices were readers, and then we compare that to the expected amount of internet users, we would see that in 2018, we had a gap between the approximately 3.7 billion internet users and our 1.5. But as we look forward, the gap would grow. So as we meet here in Stockholm to think about readers and to work with you and to work with our panel guests to think about how we reach more readers, we have literally billions of people to think about and to reach. Yep. And what can we do to increase readership? What can we do, Zach? We've got lots of things to show you today on exactly that. Things from around the world. First up, from Mexico, we have a video that was produced last fall in October with the aim of explaining Wikipedia to internet users in Mexico. Let's take a listen. Why don't you hit it here? Okay. Oh, yeah. This was produced in Mexico last year, and Ivan Martinez and Carmen, who are representatives from Wikimedia Mexico here at the conference, can tell you a lot more about this project. It also featured a young, as you saw, a young lady, a superdotada, a know-it-all, a girl who knows everything, which became the title of this film. So let's take a look at some more efforts. Not that. There we go. How do we advance? Next up, we have a project that has come out of the New Readers program at the foundation. And Angie, who is seated in the front row, will tell you a lot more about this project in detail.  It's called Cricut Info, and the URL is CricutInfo.io. Cricut Info was a site that was prepared in time for the Cricut World Cup. If you're not a Cricut fan, let me tell you this is a big deal. It was in England this year, and it wound up with a thrilling, literal overtime showdown between New Zealand and England. But the site was prepared primarily for New Readers in India. And so when you're on the site, you will notice that not only is there a great deal of content about teams and players, there's also the ability to switch between languages. What were the languages again, Angie? Hindi and Tamil. All of the content on Cricut Info was Wikipedia content. But the ways in which you can interact with it, as you will find, are very different than the Wikipedia site design today. There are stories in an Instagram-like format where you can actually advance between rich images. There were polls, which I have been told were very popular, where you could talk about your favorite parts of the tournament, your favorite players. And of course, there were little modules like the ones you see on screen so that you could learn about individual teams through kind of a team hub. We'll talk about that a little bit later. We also are excited to show work by Blossom, who will be joining us on the panel a little bit later. I think we can just let this video play, because Blossom has done amazing things in Nigeria introducing Wikipedia to New Readers. Let's take a look. Hello everyone. This is Blossom from Zerumba. My name is Kaiba. My name is Mara. I'm from Makayabu, Wikipedia. What's your name? I'm from Chuku. Chuku? I'm from Makayabu. What's your name? Wikipedia. I'm from Makayabu, Fembo. I'm not from Fembo. I'm from Makayabu. I'm from Makayabu. I'm from Makayabu. My name is Ije, not IG, not wikipedia, not oldRTG. I know him very well. I know what he's like, he's always open. Thank you so much. Yes, man. Thanks Blossom. There's a lot more of those. Blossom, how many did you make in total? Forty. Wow, alright, well that's one. I think they're at all at that level. And let's see more efforts by other community members. Next we will talk about Instagram campaign done by Sam Oye from Nigeria. Let's hit it, okay, I think we need access. Oh. Okay, I have that in, okay, we should then maybe just skip it. Sam Oye, who is also here at the conference, is a community member from Nigeria and he produced a series of videos where he asked people across Nigeria to explain what Wikipedia is, to basically create a campaign where readers promoted Wikipedia to other readers. So if we're able to load these videos, what you'll see is the wide range of ways in which people responded to that challenge. Some people created songs, some people signed their explanation of what Wikipedia is and others went ahead and actually did it as like a lecture or a sermon. It was very eclectic in the ways that people did this but it enabled us to reach hundreds of thousands of additional readers in Nigeria in a way that was entertaining, authentic and original. Rupika, it's okay if we can't play it. We can leave it for everyone else. Okay, great. Okay, let's take a look at the next campaign which was by Bitplub and Tulsi from Nepal. It's not going down, okay, great. So every billboard tells a story and every time you pass a billboard, there is something that you know. And in Nepal, Bitplub and Tulsi presented billboard and every center spot of the city. So every time people passed by, they knew something about Wikipedia. So amazing efforts to increase readership, yeah. And let's talk about community marketing experiments. Back over to you. Over the past year, the foundation has collaborated with lots of communities that want to do marketing projects and these have been all funded with small what are called rapid grants or inspire grants. If you're interested in that, we're gonna be talking a little bit about grants later today with Woobzena who is the specialist for grants at the foundation. Currently, a inspire grant or rapid grant can be up to $2,000 in value. And here, this was a collaboration between Bengali Wikisource and Tamil Wikipedia that use that $2,000 to make a series of short videos and social media posts. So their target locations were two areas, West Bengal, India and Bangladesh and then Tamil Nadu and Sri Lanka. And they were for two different projects. So this morning we've been speaking a lot about Wikipedia but in this particular campaign, Wikisource was a project that was promoted. For that first group promoting the Wikisource two videos were made and a series of holiday social media posts were made as well. You can see on either side, the moving gifts here are the two videos and in the center that's social media copy. These videos were really fun. You can see the one happening with tea, some great chai being produced. That video featured two people arguing about politicians and finding common understanding by going to Wikisource and finding the original document that they were debating and reading it closely. Studying brings people together. The second video titled Watchful Eyes is one of my favorite plots I've seen recently in a short film. What happened was that a young gentleman was basically trying to leave the home to go be with his friends. But his mother stopped him on the way out and said, where are you going? And he said, oh mom, you know me, I'm a hard working student. I need to go to the library and prepare for my exams to which she replied, no worries my son, you can just get everything you need on Bengali Wikisource. And he didn't get to go see his friends. So that's where Wikisource is positioned right now. And the results were astounding. During the campaign period, we saw that site views really increased. And even after the campaign ended, we were pleased with how the site visits remained rather high. On the Tamil campaign, most of the media files were actually just commons images that were featured on Tamil Wikipedia. The question was asked to readers, do you know these animals? And if not, come on to Tamil Wikipedia to learn more. There we saw similar increases in page views during the campaign period and afterwards. So not only did we see that more people came to the site while this content was going on and being shared in social media, but again, after we saw that the resting level was now higher. So you are in this room with us in the readership space and you may be asking, how can you help increase Wikimedia readership? What can you do? And you can stay with us for the panel and you can also consider hosting events and producing different animation videos, other campaigns that people have been doing in other countries. And you can also apply for a grant and we will have a grant sessions in the next coming few hours. You can stay tuned in the readership space and I would like to invite our panel, Blossom and Angie to on stage.  Let's hear it for our panel. Thank you so much for making the time this morning. Welcome, Blossom. Hello. Blossom, welcome Angie. Thank you. Hi Angie. Hi, how are you? Please tell us a little bit about yourself and your project. Good morning, everyone. My name is Angie Mwegai. I'm a Senior Product Manager at the Wikimedia Foundation and I'm part of a new team that's thinking about how we drive new readers to our projects through product interventions. I am from and live in Nairobi, Kenya. Explain it. Good morning, everyone. Good morning, Rupiker and Zach. My name is Blossom Uzurumba. I live in Abuja, Nigeria. I am a public servant and I volunteer as we all are to various projects of the Wikimedia Foundation. I love television. Yeah, I love television. So it's, you know, the project I did was just only natural for me. I'm light as a compare, so I always would love to hold the microphone. So when I got to know about the Inspire Compare and what really stood out for me was, okay, it's an opportunity to go into the streets and just be the talent in front of the camera. I don't want to be behind the camera. So it was just fun. And then having all these conversations in my local language, it was just this feel good, exciting kind of shot videos and it's something I'm really, really excited about and I hope we can continue probably expand the scope of the audience with time as well. Thank you. Wonderful, thank you so much. And tell us a little bit more about your experiences, how the idea of the entire project came to be and how we developed on the way. Okay, so why is that? You can start and I'll go next. Okay, so the idea for me is, we always have this funny member videos where in my country, we have a lot of people just go into the streets and then just ask random questions. And then the response you would see, sometimes you would wonder, did they actually hear the question that they were asked? Maybe they asked you what is A and then you're answering Z is this. So it's totally divergent. So we felt okay. We can also do that kind of stuff with Wikipedia and then see if people know it. So if you watch some of the videos, I was just holding this pillow that has Wikipedia written boldly on it. And then some people would, I would ask them, can you pronounce this?  And then stuff like, wiki badia, you know what I mean? It's all shades of funny. Not because they don't know. Maybe they just did not take the time to look at that very well. So we decided to just include all that and then just make it a whole three, six thing. So people know about Wikipedia. Some people don't. So for those that know, applause for those that don't, then it's an opportunity to then introduce that to them and then hope that they would use it. And then for those that could not pronounce it, it was also an opportunity to teach them how to pronounce. You have to know how to pronounce this thing before you then embrace it and adapt to using it, you know? For us, we were thinking about people who have not heard of Wikipedia before, do not have any expectations of what Wikipedia looks like because they're starting to come online for the first time. And for the people who are coming online for the first time, the introduction to the internet is products like Facebook and YouTube and Instagram. And so what does Wikipedia look like for people who are introduced to the internet in those modern forms of experiences? And so what we did was we took some content from Wikipedia, we played around with how it looked, made it feel easy to consume, put it into little chunks and pieces because the type of readers that we wanted to reach with the experiment was people who are not quite comfortable with English. And so when articles are only available in English, how do you simplify that for them? And we were starting this experiment in India to begin with and cricket is big in India. Like a lot of people like Zach said, love the sport. And so we chose that as a subset of content to experiment with and we saw results around how people decide when to switch languages, when they prefer to read content in their local native language. We experimented with how when it's available only in English, what does that look like? People have expectations that they're gonna see lots of images. So we put that in as well. And we're also continuing to experiment at this conference. So I'd love to be able to show it to some of you and get you to play with it and give us feedback on how we can start to bring these types of experiences to our project. Fantastic, very inspiring. Thank you. And what are the learnings and best practices? Blossom, you think community members can adapt from your project and maybe even replicate if they want. Initially, the audience actually had a mind at the start where mostly young people. But sadly, I live in the central part of Nigeria where you don't have as much native speakers of Igbo language. So yes, that was the location we chose, but then what that meant was I had to sort of not focus anymore on young people. So I had to go into the marketplaces and where you have a lot of artisans. So you see a lot of conversation with much older people. So that was not the initial plan. But because I could not see the target audience that I really saw in the location that I was. So I had to make do with that. But then the learning for me really is, it's always easier for you to be able to form that synchronize between what you originally want and then probably in location scouting and stuff. So like she said, cricket is big in India. So it was just easy to marry what they would naturally want to consume with the location that it is tailored and fit for. So that was just the first learning for me. So if I am to repeat this process again, I would definitely not want to do it in Abuja, Nigeria. I'll probably want to go down to the south where you have a whole lot of native, evil language speakers. Wow, great. And Angie, what were the best interesting things you found about the project during the development and even now? One of the interesting things is we all know people don't consume content online. The way it was when some of us were first coming online. When, if I look around the room, some of us have been online for a couple of years and it was mostly text heavy and expectations have changed now. So the challenge that we faced was how do we take the content as it exists today on Wikipedia and start to break it down into those forms? And we found that there wasn't really a simple way to do that. So ways that the community can start to get involved is if we want to break Wikipedia content into small, snackable, bite-sized chunks, how could we do that? Either manually or through technical means. How do we make that available in several languages? Yeah, those are some of the challenges we face and I think it's a problem that we need to come together to start to solve. Great. And how can community collaborate with this project to make it more expanded and diverse? Like, what's next? The next step would normally have to be two flanks. So I don't know if it's something that the Foundation would want to continue with, but on a personal basis, it's something I definitely would want to continue with. But I am also looking at, I had to get cameras, make it all high-tech and stuff, but I'm thinking that I probably might just want to step it all the way down and make it something that is just quick to do, quick to post, easy to consume. So probably I would want to use a mobile phone and then let it just be something that you don't need to plan for. You don't need to set aside a specific day to go and shoot and go with the crew and all that. So it's something that I can easily probably just with a mobile phone have a quick conversation with someone and then not too much technical editing and all that and then just upload it and then probably just make it a bit more of a continuous thing that wouldn't even require much funds to do as well. Wow, that's interesting. Thank you. Angie, would you like to share how can we further collaborate and what's next, the cricket info? Sure, as people who are creating content, the types of new readers we want to reach, particularly in emerging markets in Southeast Asia, India, parts of Africa, Latin America, think that most of these people's experiences is on their phone first. So as you write your content, think about what it would look like on a mobile device first. We would love to show you some of the ideas that we're thinking about to get your feedback on them. We would like to get ideas from you as well. So if there are some ideas on how the readership, how the reading experience could be better, please drop that to us where the new readers engineering team on Media Weekie. And once some of these features start to come out, then spread the word, let people know that there's different ways that they can read content and then contribute to it. Cool, thank you. Plus, would you share some of the hardships or challenges you faced during the implementation of the project from designing it to devising it and to creating such amazing video? How did the process go? The only part that was a bit of a challenge for me was like I earlier noted, just having that technical skill. So I was more or less at the mercy of people that could provide that service. But then if it's something that can be quickly done with a mobile phone, then I don't think that would portray a challenge going forward. It's just easy. So people, yeah, people would want to look out for excellence, but in this age of mobile first, a lot of people will just be interested in quick things that can get their attention before they move to the next attention to demanding conversational content. So I would want to probably create a balance between professionalism and just getting your content out there. So if I'm going to look at mobile first, I might not have this challenge again because it's something that I can probably simply do. And I'm also looking at not just being me. We have a lot of community members that I can also probably encourage wherever they find themselves. It could just be a good opportunity to ask if people know about Wikipedia and then see it as an opportunity to let them know if they don't know and then just document that whole conversation or the whole experience with your mobile phone. Wonderful. And how did the collaboration go with the community members and people you outsource for creating the professional video in the end? It was quite easy in the sense that we had to... I think it was just exciting because number one, like I say, that lived in Abuja, Nigeria. So by the time I went into the areas where you have a lot of artisans and native speakers, the field that knew about Wikipedia, it was like, they were just looking at us and see if we came down from space, like, oh my goodness, this thing we only see and hear about, it's come down here. So we're feeling funky like journalists or something. And then we had this branded t-shirt with Wikipedia on it and then boldly written Dewo in Igbo language, Dewo means greetings. So it was just this amazing reception and then we went with some very small gifts.   And then they have to sum that. They can't speak Igbo and then they felt really bad. Oh, why can't you just do this in English or in house or something? But sadly, that was not the scope we were looking for. So but then there were still some that we just had to just interview like that in English language just to make them feel like a part of the whole process and then to also make them go away quickly so we can continue with what we're doing, you know. Overall, it was an enjoyable process and it was also a good opportunity to gauge the feel of people in terms of accepting this brand called Wikipedia and its other sister projects. Thank you. Wow, wow. And Angie, can you tell us more about how the development team worked on this New Readers project and how the collaboration of this project came from start to finish? So what we wanted to do as we were thinking what new ideas we could have to bring New Readers, I see like we're on the other side of the coin. So Blossom is trying to bring new people on and then when they come, what do they find? What do they see? Do they find information easily in their language? And so what we did was based off of knowing what topics of interest there are, we put that together in a way that was easy for people to consume. So what we did was we went where our next set of target users would come from. We decided to do this in India. We traveled to Bangalore and that's where we did ideation and talked to people who matched the kind of readers we would want to bring on and wanted to understand, speaking to them in research sessions, does this work for you? Does this make sense? Would you find frustrating what could be better? And that's how we put together what you see on the Cricket experiment site. Wow, wonderful. Blossom, what advice would you give community members who want to do projects for increasing Wikimedia's awareness? Do you have some advice they can follow? Yeah, the first thing is if you look at the user interface for Wikipedia for instance, it's already too serious. It's already too technical. It's already too, you know, just imprim and proper. So the only advice I would have to give is when you're welcoming people into that space, just go infuse a lot of play, infuse a lot of excitement. And fun. Just make it look so simple and easy so you don't push people away. You know, in Nigeria where I come from, we already have so much problems. So you don't want to add to people's, you don't want to put so much pressure by saying, oh, come. Okay, thank you for content translation for instance. You know, before now, if I'm doing, let's say translation, translated turns, it's a big issue. I'm battling with trying to remember the few technical stuff I know and then transferring that knowledge to someone that has never gone to a back end of anything before. And of course, next time, the person will not show up. So it's very important that we infuse a lot of play in how we communicate and in how we also sell this to people so they can enjoy it so it comes naturally. I think play is something that comes naturally to people, a lot of people, not everyone. So if you go with that approach, I can assure you that you would have a lot of people that would be willing to listen to what you have to say. At the point, you saw me dancing in the streets and all that, it was just part of the plan to just make this thing as simple and easy as foreign as it could get, you know. Well, thank you for this wonderful advice, yay. Angie, what is the message you want to convey to Wikimedia community and our readers out there? I would say for those who are contributing to projects, don't only think about writing. There are several ways that people consume content online now. So create videos, create audio files. Many of the languages in parts of Africa where I come from, I think it's similar in parts of India and Latin America that people want to consume content in videos and audio because that's how they live, that's how they communicate in their language and with each other. So let's continue to add text, but also put that in mind. And let's continue talking. Let's come together on ideas on how we increase awareness and bring people to our projects and get them to continue contributing to the knowledge that we have because that's how we'll start to become more diverse. Thank you so much. And now we will open the panel to audience. Any questions from audience? Questions here, here. Any questions in the room? I can bring this to you. Ah, there's a question, yay. Here we go. Hello. I have a question about the India New Readers program that you did. So Cricket was number one. Did you have any follow-up topics that you thought would be the next areas to focus on? But let's say number two was something else that was popular. I'm just curious as you gathered that information what that list looked like. That's a very good question. So one of the things we thought about is how do we make it current to what's happening there or in other parts of the world? And so one of the things we talked about was the political situation. So there's a big election that just happened in India and how do people find out about who's running, where they come from, and things like that. So that was where we were thinking of next. But we would love to welcome ideas in terms of what topics would drive reading or watching in this small format. Not every topic fits that. So books, for example, could be one of them finding out about an author or movie stars because Bollywood is big in India. But it's not just about India, we're thinking about other parts of emerging markets as well. Thank you. Any other questions? Is that a question up top? I'm sorry. No, I don't want to put anyone on the spot here. We know it's early. Going once, going twice. Well then, perhaps we can just leave it there and if you have any additional thoughts you can find us at the front here or throughout the conference. Thank you again for joining us this morning. Thank you so much. Thank you Blossom. Thank you Angie for being a part of this panel and sharing your experiences and wisdom with us. Thanks everyone. Thank you so much. Great work. We will be on the lightning talks shortly after 15 minutes break and stay tuned to know more about interesting case studies by our community members. Thank you.


I wish everybody a happy new year. Yes, we wish everybody a very happy new year and a healthy new year besides. Challenger? Here. Warren? Here. Carlson? Here. Decker? Here. Donahue? Here. Hammond? Excuse me. Heidemann? Here. Koch? Here. Lassard? Here. Van Akron? Excuse me. Vanderweel? Here. Verse? Excuse me. Wangeman? Here. Thirteen Presence. Is there any legibility? I pledge allegiance to the flag of the United States of America and to the Republic for which it stands, one nation, under God, indivisible, with liberty and justice for all. Move to approve the minutes. Second. Seconded to approve the minutes. Is there any changes or additions? Are you none so colorful? Warren? Hi. Carlson? Hi. Decker? Hi. Donahue? Hi. Heidemann? Here. I'm sorry. Kathy? Hi. Lassard? Hi. Wendusky? Hi. Manichek? Hi. Racler? Hi. Vanderweel? Yes. Did he turn? I think he was deeper. There are two resignations to the Harbor Center Business Improvement District Board. Caitlyn Brotz-Mcnit, who's resigned her position as board member and president effective 12-1-20-12 due to time concerns and increased family and business concerns. And Mike Miller, who resigned effective 12-1-20-12 as his business is no longer located within the Harbor Center Business Improvement District Boundaries. Alderman Carlson. Thank you, Mayor. I move to accept and file the resignations. Second. Been moved and seconded to accept and file the resignations. Is there any discussion? I just call the roll. Carlson? Hi. Decker? Hi. Donahue? Hi. Lussard? Hi. Lewandowski? Hi. Manichek? Hi. Racler? Hi. Vanderweel? Hi. Wonderman? Hi. Bellinger? Hi. And Boren? Hi. That's one, too. 13. Motion carried. Confirmation of Mayor's Appointments. I'm going to members of the council hereby submit the following appointment for your consideration. Third, to be considered for appointment to the Historic Preservation Housing Rehabilitation Loan Commission, to fill the unexpired term of Jonathan Dolson, whose term expires for 22, 2013, signed by the mayor. Alderman Carlson. I move to confirm the appointment. Second. It's been moved and seconded to confirm the appointment. Is there any discussion? I just call the roll. Decker? Hi. Donahue? Hi. Heidemann? Hi. Kass? Hi. Lussard? Hi. Reisler? Hi. Vanderwiel? Hi. Wongerman? Hi. Bellinger? Hi. Boren? Hi. Carlson? Hi. 13. Motion carried. Consent agenda 2-1 through 2.5. Alderman Carlson. I move to accept and file all ROs, accept and adopt all ROCs, and pass all ordinances and substitute ordinances. Second. It's been moved and seconded to accept and file all ROs, accept all ROCs, and pass all resolutions and substitute or pass all ordinances and substitute ordinances. Under discussion. Clerk will call the roll. Donahue? Hi. Heidemann? Hi. Kass? Hi. Lussard? Hi. Rilandowski? Hi. Maddachuk? Hi. Wongerman? Hi. Bellinger? Hi. Boren? Hi. Carlson? Hi. Becker? Hi. 13. Motion carried. 3-1 through 3-9 will be referred. 4-1 will lie over. 4-2 through 4-5 will be referred. Alderman Boren. Thank you, Mayor Ben Akron. I would also, with document number 4.4, resolution by Alderman Hammond, authorizing, executing the combined dispatch intergovernmental cooperative agreement in addition to finance. I would also like to refer that to the Committee of the Whole. I don't think we need a motion, but we can do that. Thank you. Oh, and I just wanted to mention for the Alder persons, tentatively, I'm planning a Committee of the Whole meeting. A week from Thursday, I believe it's a 17th. Time to be determined. I'm working with Alder person Donahue on that. She has a document from November that she wants to bring before the council. So I think I'll include this one at the same time. So tentatively the 17th. Thank you, Alderman Boren. 4-1, we have already did 4-4 through 4-6. Other matters. Alderman Carlson. Second. Seconded to adjourn. All those in favor signify by saying aye. Aye. Opposed? Motion carried.


where we discuss solutions to issues that impact sports. I'm your host, Angela Hazlet. Today's guest is Jason Augi, the Senior Vice President of Sports Tourism for the Tampa Bay Sports Commission. We are here to discuss from Super Bowl to March Madness, hosting major sporting events in Tampa Bay. Welcome, Jason. Hey, Angela, thanks for having me. It's a pleasure to join you today. We're excited to have you as well. And I know that you work for the Tampa Bay Sports Commission. You've been there for quite some time. It's a nonprofit organization that prepares bids to host sports and entertainment events in the Tampa Bay region. So as the Senior Vice President of Sports Tourism for this group, what does your day-to-day job usually look like? Yeah, sure. So I think you hit the nail on the head. I mean, working for this company for almost 18 years now, we've been fortunate to host some of the biggest and best events in the sports and entertainment realm, if you will, from Super Bowls to NCAA Championships to WrestleMania to some of the bigger youth and amateur tournaments in traditional and non-traditional sports. My job is really to kind of guide the company, if you will, in our overall sales and marketing efforts, overseeing our bid process, how our recruiting efforts, and then making sure that we're delivering on the promises that we make in each of the proposals that we submit to these different organizations to try and host their events in Tampa, Florida. Tell me a little bit about the contracts that you have. What kind of requests or requirements are pretty typical for hosting an event? Yeah, so it's a great question. So every event that we host, and when you look at the youth and amateur space, we're traditionally hosting anywhere from 70 to 90 events in a given fiscal year, again, in just about any sport you can name. And then when you get to the bigger ticket major events, it obviously is quite a production on somewhat of a parallel path, I would say. And each entity, when they're putting out their request for a proposal or RFP, they kind of stipulate what's important to them. And that's the thing with our organization that I think we do really well, is we customize each proposal. It's not a cookie cutter approach where we're taking one proposal and then just retrofitting it in a way where it's like, okay, yeah, this is good for everyone. It's more so, how do we customize to show each individual group that we're working with that we value their business and obviously want to cater to their needs? So, whereas you may see a NCAA championship RFP that could be anywhere from 40 to 50 pages, you may see a Super Bowl bid that's 75 to 90 pages. We're creating huge proposals that are the size of phone books, if you remember those from back in the day, with content, and then in a lot of cases too, we're able to streamline on the youth and amateur side where we're creating a lot of memos of understanding and agreements with those event organizers to basically explain where we're coming from and what we can do and showcase our commitment to them as far as what our contributions will be both financially and in kind. So, 75 to 90 pages, that's a lot. I imagine it's difficult. How do you keep things from getting missed or lost or misinterpreted even? Yeah, no, it's a good question too. I think you have to be very judicious about planning out your efforts. And a lot of the time you have different bids and proposals that are running concurrently. So, making sure that you're on top of your planning and coordinating and execution efforts is obviously paramount. So, we'll work internally as a team to kind of set the course of who's going to handle what and then what those timelines are. And as being the intermediary or the main agent and brokering the deals, to help bring the venue component to the table, you obviously have the accommodations piece where we're bringing all the hotels into the fold. And then you have all the ancillary or special events that come along with it. What does the marketing and promotional plan look like? What does your volunteer support system include? I mean, every single proposal has kind of some key elements that are relatively central at each one of the things that we do or the events that we go after and target. But you've got to be really buttoned up to ensure that you're working, very effectively and efficiently so that things don't get down to the wire. And then all of a sudden it's like, oh wait, we don't have an answer from this partner or we don't have an answer from that partner where we're establishing the right timelines to be able to do it at a high level. And have you ever found yourself getting the bid to host an event and deciding in negotiations that it wasn't a good fit and kind of backing away after the fact? Yeah, normally I would say 99% of the time when we go through the RFP process where and the timing on this is fantastic having just returned back from a, one of the lead industry trade shows last week where we had 55 one-on-one meetings like almost in a speed-dating format, pre-qualified appointments with groups that we felt really, could be a good fit in Tampa or we wanted to learn more about what their business model looked like. So many conversations now taking place just a few days later with those prospective organizations as far as identifying opportunities. So what happens is, and looking through the request for proposal we'll look at it with a fine tooth comb to identify, okay, these are the things we feel like we can make happen based on the request. And there may be some things that we denote as an exception things that we wouldn't necessarily be able to agree to. And then from there it becomes a conversation. Are these things that you as the event organizer have flexibility on to where you may be willing to grant an exception from our side. And then maybe there's some things that we can do on our side where we raise our level of support in another area to kind of make up for what may be a shortcoming. So long way of saying, as you go through the process the biggest thing we've found is just being transparent and candid throughout. Like a lot of the relationships we're trying to foster or ones that we hope will be deep rooted long lasting. So there's not a sense from obviously our organization or a lot of the event organizers that we work with that anyone is trying to pull anything on everybody. Everyone's in it for the same reason trying to find mutually beneficial solution. And more often than not, we're able to reach it. And but yes, there are certain groups that we'll engage with where based on the business model or particularly the return on investment if it's not worth it from how we operate then we'll pull away from the deal in a respectful and professional manner. Absolutely, that makes a lot of sense. And I would imagine with your history and success of hosting large events you're probably a trusted partner. So a lot of groups are probably eager to work with you and maybe make those exceptions as you mentioned to develop that relationship. I know in the month of November you're promoting events like skateboarding, soccer, hockey events. So what dictates how involved you are with each of these events? I imagine some of them you're probably more hands-on and others probably other partners can take the lead. So what kind of dictates your involvement with these events? Yeah, I think from the genesis of the conversation and that if we're fortunate enough to be awarded usually those RFPs and the conversations that proceed submitting the proposal and then if you are awarded will kind of dictate what roles and responsibilities will be. So whereas certain groups require a lot more involvement others may be more so, hey, can you help us secure the venue and then give us your contact list for hotels to it for us to go and source and hey, we may need some volunteer assistance too. And if you don't mind helping promote, we'll arm you with the press releases, social announcements, things like that. So we're still involved very actively with those but those are kind of the lesser of the two I would say. And then you have some groups where we're along for the ride, where it's planning meetings or planning calls weekly to try and help build out how the event is activated, encompassing a lot of the points from the first example but then even bigger ticket on a larger scale with all the ins and outs and idiosyncrasies that go with each one of the events that we host. So I think each individual event, there are separate conversations that take place to identify what our role is. But at the end of the day when the ball tips or is kicked off, that's when the event organizer or the league or the entity that's running point really takes the reins on the execution side and then we'll help with some of the ancillary programming. And the events like the Super Bowl, March Madness, are those events where you're more hands on and along for the ride, as you mentioned? Yeah, I would say so. I mean, probably one of the best examples is the college football playoff national championship. We started, we hosted in 2017, we were the first location in the Southeast United States to be awarded that event and that was a five year process. And so from start to finish, we'd lost out on the initial bid to Dallas knowing that they were likely going to be the first host but our goal was to come to the table with a very strong offer. And then when we were eventually awarded as the first location in the Southeast, we wanted to knock it out of the park. To your point earlier about your reputation, we take a lot of pride on, as I mentioned at the top of the show, delivering on the promises that we make. And so for that, I mean, you're in the trenches with groups like the College Football Playoff or the National Football League or the NCAA or the NHL on a lot of those marquee or major events that you host where, I mean, every single aspect of behind the scenes that goes on with hosting a major event, our team is heavily involved to the coordination and execution side. And then obviously complimenting the efforts that are being put forth by those entities. So yeah, it was an interesting dichotomy, if you will trying to balance or blend your responsibilities with your youth and amateur events which are truly our bread and butter and foundation for what we do on an annual basis with the roles and responsibilities of what is expected when you're hosting these marquee major events because in theory, you could be spending every single day on one of those too, but we had to find a way to balance and obviously execute at a high level. Absolutely. So when you are in the trenches, what kind of challenges have you encountered? Well, I mean, first and foremost, the COVID is probably the first one that comes to mind. I mean, we were a week out from hosting the men's basketball first and second rounds and then three weeks out from hosting our first ever WrestleMania. And so the process and work that had been done leading up to those moments and then you have that whole unwind period where you're having to work through all the different logistics that had been coordinated and calling everything off to then it became, okay, how do we tactfully, when the timing is right, engage on ways to get the events back and then we're fortunate with WrestleMania to get it back and then we had Super Bowl, then it becomes how do you operate safely against the backdrop of the pandemic which was no small feat. So when we were going through the planning cycle, if you will, with both WWE and the NFL, it was trying to work through plan A, B, C, D, E, F, if these things were to happen. And so constantly being fluid and thinking on your feet to a degree but also planning for every situation became really challenging. But in the end, it all worked out. I mean, it was icing on the cake in the sense that the Buccaneers became the first team to play in their home stadium for a Super Bowl during the pandemic had never happened before. That brought a whole new slew of challenges because obviously you have all these people in the Tampa Bay area who now wanna be a part of Super Bowl above and beyond the normal interest level because their team is participating and with cap ticket numbers, with special events that had only a finite number of passes available, trying to socially distance and be cognizant of everything that went along with it, presented some challenges. But in the end, a lot of smart people came together and worked to create an environment where we were able to operate it effectively and then obviously safely. And for what it represented in terms of the first major sporting event to be hosted during a pandemic, what it represented for hope, I think for the country. In a lot of ways, it was an incredibly amazing feeling in terms of the intrinsic value that we were able to derive by pulling it off. In Florida had fewer restrictions in some other places, so that probably assisted in your ability to execute the event. Yeah, I mean, yeah, without getting into the politics of it, I mean, the fact is Florida was open for business and that certainly opened a lot of doors and led to a lot of conversations. I can't tell you how many, and then back to your point about challenges, that challenge actually became an opportunity because we were able to host at such a high level, Super Bowl, and then three months later with WrestleMania, which was the first time a city had ever hosted those two events in the same calendar year. It opened the door to where we were receiving calls, emails, nonstop from different groups from different parts of the country saying, hey, we'd like to bring our event to Tampa. And so it's created this huge boon for us in terms of future events that we've been able to host and then book for down the road. How do you think you were able to secure those high profile marquee events that you mentioned? Well, I think that old adage, you only have one chance to make a first impression, really does play true. And we're not a community that rests on its laurels. We talk internally about, you judge an event success by whether or not it returns. And as evidenced by the fact of this April, we'll host the NCAA Frozen IV, the Men's College Hockey National Championship for the third time. In December of 23, we'll have the Women's Volleyball Championship for the second time. In 25, we have our fourth women's Final Four. And then 26, March Madness is coming back for, I believe, the sixth time. So, you look at that, and I think that's a really important and impressive metric for us. Groups that are continuing to come to Tampa are doing so for good reason because it's not just, okay, here's the offer. We're coming to Tampa. Good luck to you. We're there almost every step of the way to ensure that we're delivering either a solid student athlete experience or, and then obviously for any and all the visitors, rolling out the red carpet for them. Every geographic location has its unique weather challenges and Tampa is no exception to that. Recently, Florida was hit with a hurricane, category four hurricane, Hurricane Ian. Luckily, Tampa was spared the brunt of that impact, but I imagine that hurricanes, weather, weather incidents are really a strong piece of your planning process with being located on the coast and the lots of shoreline. So, talk to me a little bit about what kind of plans do you make for weather? Sure, yeah, I mean, well, hurricanes are obviously front and center, right? You know, in terms of that period, where you traditionally see them in that August, September and even into October timeframe. Yes, we were very fortunate being spared with the late shift. We feel terrible for our neighbors to the South and talking to our counterparts here, everyone's doing well, but yes, I mean, the rebuild process as part of that is incredibly challenging. I remember when our convention and visitors bureau, Visit Tampa Bay was hosting the Republican National Convention in 2012. It's a fall event or fall convention and there was a threat of hurricane, but similarly, it shifted a couple of days out, but the planning that goes around that is pretty intense because ultimately when you're hosting an event, safety is one of the most important, if not the most important things. And there's a trickle down effect to that too. Obviously you think about hurricanes, but then also in Tampa Bay, you're in the lightning capital of the U.S. And so when you're hosting outdoor events, you have to be aware and work with event organizers, work with your facilities to say, okay, we have lightning strike detectors that, what is the distance in terms of that distance from that strike where we're going to shut down play and send everyone to their cars or send everyone to safety. And that's something we take very seriously as well. You've had situations from college football games or professional football games at Raymond James Stadium where you see a lightning strike and it's delayed another 30 minutes. And while it's frustrating, you do it for the right reasons because of safety and the same thing holds true at a youth soccer tournament or in a senior softball competition. I mean, it's a huge part of what we all do because last thing we want to have happen is someone who gets hurt or even worse. Yeah, like in 2008 in Atlanta, that the SEC men's basketball tournament, the fact that the game went into overtime likely saved a lot of fans who stayed for the game when a tornado touched down outside of the Georgia Dome. So it's amazing how some events like the game going into overtime can really do a service to protect the spectators. Yeah, and that's a good point because you can plan and plan and plan, but there are some things that are outside of your control. I don't know that anyone, and I was at that game. I was inside the Georgia Dome when that hit and it was unlike anything I've experienced. And you have to think sometimes the luck plays a role as much as you want to plan. Sometimes there are things that are unforeseen or that happen that are outside of everyone's collective control. Absolutely. Did they hold you over in the Georgia Dome then when the tornado hit? They did, yes. So all the games were essentially stopped. Everyone was moved to shelter in the concourses and then once it was deemed safe to leave the area, we all went outside and saw sky bridges torn apart, car windows blown out. I mean, just a ton of damage, flooding. I mean, all types of things that you're just like, it was almost surreal, but that's what happened. And then other incidents happened like at the Miami Dolphins game and tailgating in the parking lot when someone pushed their grill wasn't quite put out. The fire wasn't quite put out and ignited the vehicle and caught other vehicles on fire as well. So how do you kind of manage events where you have spectators that might be tailgating or they're kind of out in ancillary areas next to and adjacent to the facilities that you're operating? Yeah, I think inherently with any event there are different situations or scenarios that can present itself where something could go wrong. I think by mitigating what is allowed and obviously communicating kind of an FAQ if you will or almost now more than ever, you see these know-before-you-go type communications that try and lessen those types of things from happening. Sometimes it can be common sense. Sometimes it's bad luck, but I think with security and patrolling and all these different things that are enacted to try and help not have those situations. I think most locations and venues and events do take that into account, but every now and again, you're going to have stories like that that arise because of either just, it's unfortunate timing with bad luck or just you know, you miss something and that seems what has happened. In regards to transportation, are there still a lot of your events require shuttling or transporting of spectators? And how does the new, how does rideshares like Uber and Lyft change the way you're planning for events when people are trying to get dropped off and picked up? Yeah, I think all of it is encapsulated into the overall planning process. And normally, a transportation committee will be formed or a specific part of the hosting effort will be dedicated to ensuring that transportation is done effectively. I mean, I've been at events where it's run seamlessly and usually you work with traffic groups and special event departments who are in charge of that. And it's something that you don't want to have an event where you can't get to the event and then you don't want to have an event where when you're leaving the event you're sitting for 45 minutes to an hour plus trying to leave. That's something we take very seriously and we're fortunate where everybody plays nicely in the sandbox in the sense that one hand is talking to the other and making sure that all of that is taken into account. Rideshare obviously is a huge part of that. It's open doors where you're lessening the drinking and driving component for those who want to partake. But even so, it's making sure you have the correct amount of inventory on hand. I was at an event a few years ago where it was documented on not only local news but that also had made national news this one organization did not plan accordingly and they had people in the rain with an event after midnight that were waiting an hour and 15 to an hour and 30 to try and get Rideshare to get out of this isolated stadium that was in the middle of nowhere.  You can work for a year, two years, three years to plan everything out and everything goes absolutely according to script perfectly. And then you just have that one thing and that's the last thing image in a lot of people's mind and it's just not a good look. So you have to, I mean, I would put transportation and proper planning right up there with safety. Absolutely, absolutely. And then the communication piece, marketing piece talk to me about how integral that is to promoting and branding a good message for your partners and what kind of considerations do you have to take into place? Yeah, I think the marketing and promotion side is obviously hugely important. Like recent example, we just came off of hosting USA gymnastics, their national championships and national Congress, huge success. And the big reason for that was the collective work that we had done in concert with USA gymnastics to build out this marketing and promotion plan to reach all different folks and organizations throughout the community. And what it translated to was huge attendance numbers where, be fitting of what an event like that would expect. Same thing with the SEC men's basketball tournament when we hosted back in March again. You know, maybe you may not necessarily think of Florida as a huge basketball haven to a degree. But again, when you walked into Emily Arena and felt the energy of all those schools from the SEC playing, I mean, it was quite an environment. So the marketing and promotional efforts that are deployed as part of that are so important. And then using all the different tools in your toolbox if you will and leveraging relationships and is so critical. And that's why I think we've been successful where we've hosted these different youth and amateur events that have a huge economic and social impact for our community at a very high clip. You know, a lot of events each and every year. The major events in theory only come around once every couple of years, but we've been on a roll where we're almost hosting one every year. What that does is it creates buy-in. We talk a lot about this Team Tampa Bay mantra where everyone is in it together. It's a unified front. And what it translates to is everyone buying in and supporting the event. So when the next event comes out, you know, it's more so how do I get involved? How can I help? And I think that is very organic, very genuine and authentic here. Well, it sounds like you're doing a lot of great things, Jason. Are there any final message or final words that you'd like to share with us? Yeah, you know, it's crazy to think. Well, first off, thanks for the opportunity to join today. And it's amazing to think, you know, 18, coming up on 18 years here, still have the same passion for what I do. It's just great to see Tampa growing as exponentially as it is because it makes our jobs that much easier and Tampa is such an incredible sell.  And I feel like we've carved out a pretty good reputation and niche in this industry to be one of the leading sports commissions in the country. Amazing. Thanks for your time and for your insight, Jason, from Super Bowl to March Madness and hosting major sporting events in Tampa Bay. Thanks to our viewers for joining us on the sports playbook. In two weeks, our guest is Mark Sartori from RMC Events, who will discuss how his organization provides guest services and security staff for sporting events. We will see you then. Thank you so much for watching Think Tech Hawaii. If you like what we do, please like us and click the subscribe button on YouTube and the follow button on Vimeo. You can also follow us on Facebook, Instagram, Twitter, and LinkedIn, and donate to us at thinktechhawaii.com. Mahalo.


Hi, I'm Dr. Bill Regine and I'd like to welcome you to the University of Maryland Department of Radiation Oncology. For more than 100 years, we have been providing patient-centered and compassionate cancer care, advancing scientific knowledge, and providing state-of-the-art educational opportunities. Our team treats every type of cancer at six locations throughout Maryland, collaborating on treatment plans with leading-edge treatment modalities and clinical trials. Our depth and diversity of experience make the University of Maryland the right place for patient care and the ideal place for a professional career. Hello, I'm Dr. Sarah McAvoy, the Program Director of the Medical Residency Program. I'd like to give you a brief overview of a select few of the many therapies that we offer. We provide stereotactic radiation therapy using linear accelerator-based systems such as the EDGE, as well as Gamma Knife. The Gamma Pod, invented at the University of Maryland, provides a stereotactic therapy for early-stage breast cancer with minimal side effects. At the Maryland Proton Treatment Center, we offer Pencil Beam Scanning Proton Therapy, the most advanced form of proton therapy in the world. We offer a robust brachytherapy program and are a regional referral center for complex gynecologic brachytherapies such as interstitial treatment. Additionally, we offer brachytherapy for prostate, breast, and head and neck malignancies. We have one of the most experienced selective internal radiation therapy centers in the country to treat liver cancer. We utilize thermal therapy, also known as hyperthermia, which heats the tumor and sensitizes cancer to other treatments like chemotherapy or radiation. As a department, we are committed to continually updating our technology to provide the best possible modalities for our patients. The Department of Radiation Oncology is a cornerstone of the University of Maryland, Marlene and Stuart Greenabomb Comprehensive Cancer Center, which ranks among the top 35 in the country for cancer care, according to the U.S. News and World Report's Best Hospitals list. We work collaboratively with a robust multidisciplinary team and pride ourselves on caring for the whole patient. Patients have access to services such as concierge, social workers, dieticians, and geneticists. At the University of Maryland, our calling is to train the next generation of leaders. We have a number of educational programs led by outstanding faculty, including medical residency, physics residency, a medical physics proton fellowship, summer fellowships, the Dr. Nancy K. Welker Fellowship in proton therapy and advanced radiation modalities, radiation therapy and dosimetry training programs, and medical student rotations. In addition, our department hosts annual educational courses in hyperthermia, radiobiology, medical physics, and oral board exam preparation. We are committed to providing our patients with access to cutting-edge investigator-initiated clinical trials, as well as National Cancer Institute-sponsored cooperative group clinical trials. The department has consistently ranked among the top 10 NRG main-member sites worldwide in terms of patient accrual to energy oncology trials. The Division of Translational Radiation Sciences conducts novel radiobiology science to advance new drugs, medical devices, and therapeutic techniques for clinical trials with additional biodefense applications. The Medical Physics Division conducts medical physics research and collaborates with the clinical care team to drive significant amounts of research funding for the department. We are proud to be the University of Maryland Department of Radiation Oncology. Simply put, we provide hope to patients and families during one of the most stressful times of their lives. We are inspired by our patients and driven by passion. It is who we are and it is what we do. Thank you.


I'm the CTO at 8th Light, which is a software consulting and custom development shop. And one of our core principles as a company is about learning and improving in our craft. So it's great to be here at RubyConf among a bunch of all of you who are clearly by coming to a conference motivated to get better at what you do. So it's great to be here. And we as a company and I as an individual have historically focused sort of more on applications development, so web, mobile, desktop, back-end sort of services sort of stuff. But over the past couple of years, I've gotten more and more interested in the system side of things, the systems level of the actual full stack, and in particular on performance. So like things like how do I make the system go faster? Why are these things so, so horrifically slow? What can we improve? And I've noticed over that time that I've had enough wrong ideas and intuitions about performance that those wrong ideas have sort of added up and made it very hard for me to trust my own intuition when I do run into a performance issue. And so I find it to be pretty unsatisfying when I or somebody like me sees a slow application and instantly jumps to solutions, right? Solutions like rewrite in your favorite language, right? Change from Ruby to go. Use a different database, switch from Postgres to Mongo, whatever.  The list can go on. I'm sure you can imagine your own. I've just seen too many times where it takes a while to write this new solution and hopefully it does the job. Hopefully it solved the problem. But whether it does or not, you've invested that time and it's definitely going to uncover new problems to be solved, right? After that re-writer, that big change, that big solution has been implemented. And so based on deserving these tendencies of myself over time and reflecting on them, I'm investing more time these days on the other side of this sort of cycle in understanding problems more fully. Really grasping all of the pieces that are causing the issues before I start to think about solving them. Or at least that's what I'm attempting to do. And so in this talk, I'm going to share a bit about why, a lot about how. I've been trying to understand performance problems in particular better. So I'm sure it's no surprise, since it's part of the talk title, that we're going to explore how to get insight into problems by using detrace. But first let's talk about what detrace actually is. So detrace is a dynamic tracing and observability system. So that's a fancy way of saying that it lets you see in depth what's going on in your computer at any given time. It's from the Solaris Illumos world. It's also on FreeBSD, Oracle's Linux, and it also does ship and work with OS 10. There's some caveats if you want to, or Mac OS now I guess, there's some caveats as of El Capitan. If you want to start using it, there's going to be some hoops you have to jump through and I'll have a link at the end to show you what those hoops are. But you can't run it on Macs and that's the angle that I've taken with detrace. So all these things you're going to see are in the context of me just running locally on my development machine, who's built for production, but this is the context I'm operating in. Okay, so detrace, if you're familiar with Strace on Linux, it can be convenient to think about detrace as basically a fancier version of Strace. If you're not familiar with Strace, no big deal, Strace basically lets you spy on system calls that are going on in any given process. So at a higher level than that, any time your application code asks the operating systems to do something, whether that's open a file or read or write from the network, Strace knows about that happening and can let you know. Julia Evans incidentally wrote a terrific zine on Strace that I'd recommend everybody check out whether you're familiar with Strace or not, it's great stuff. Okay, so detrace lets you spy on system calls as well. In fact, this is a pretty common workflow, especially as you're first starting to investigate some issue. Okay, but I said fancier, so what is the plus plus part? So detrace lets you write programs to trace events. In this initial case, those events are all system calls, but those programs can let you filter, aggregate, and take other actions based on what events are being traced. The programming language that you use to do this filtering aggregation and other actions is pretty limited. It's a bit of a weird programming language. D is the programming language. It's not the same D that's like the successor to C or C plus plus. It's a whole different thing. It's a very limited language, and that's by design. Your programs that you're writing to trace these events are running in the kernel, so kernel very heavily protected area of the operating system has permission to do essentially anything. But detrace is designed to be safe to run on production systems. So it's built for production, built to be safe for production, and therefore by limiting the programming language, the detrace implementers have the opportunity to prevent you from doing things that might go bad. For instance, there's no loops in this language, which to be like, when I discovered this, I was like, well, this is crazy. How can I get anything done without loops? So there's some convenience that you trade off, but this way detrace has an easy time of ensuring that you don't have an infinite loop, for example. So your program's actually going to halt. But it's safe is the idea. So what does running detrace actually look like in practice? What does it look like? So you can fire up a terminal, run detrace from your terminal, and give it your program as input. This is just me typing a program essentially on a command line. You can put the program in a file, of course, for more complicated programs. You may even do it for a program as short as this. If you've done any AUK or seen any AUK, you might see some similarities here. It's heavily inspired by AUK. So this program traces two different kinds of events. System call entries, syscall, colon, colon, colon entry, and the detrace system ending. So basically when you trace for a while and you hit control C, boom, that's the detrace end event will fire. So the colons here indicate a few levels of namespacing, essentially, and then inside the curly braces is the processing action that's going to happen when that event fires. OK, and in this example, we're aggregating all the system calls across the machine. So this is system-wide, not even just the system calls made by any given by a specific process, but by any process, which is kind of cool. And we are aggregating all the system calls and then aggregating them sort of by two keys. And one is the probefunk, which is the name of the syscall, and then the exact name is the name of the program that called it. And then when the program ends, it truncates the aggregation to the top 10 and prints it out. So I don't expect you're going to come out of this call out of this talk knowing everything about detrace and just being able to hammer out programs, but this is hopefully going to give you a good enough idea of what's going on that you can know where to go. OK, so that's the idea. And then when we run it and we hit Control C to end it and the trace, it looks something like this. And as you can see, Slack shows up a lot here. It's probably spending a lot of time rendering animated gifs. Actually, I remember when I ran this, and that's exactly what it was doing. It was a bunch of animated gif rendering, which is great. And these syscall names over here on the left might be a little opaque to you. And they certainly were for me. Even after digging into detrace for a while, I didn't know what half of these things were. So I looked at the man pages because I was interested in seeing what they did. You can Google them, et cetera. This is going to be a theme throughout the talk, that you don't have to know everything about systems to get value out of tools like this. You can use it as a learning tool. OK, so stepping back up actually a little further down from the system call sort of application to operating system boundary, you can actually look at lower level resources, so things that are further down downstream. And system resources like the disk or the memory system, you can actually trace events on those. So an example here is we can see how many page faults requiring I.O. are happening. So that gives us some idea of what memory usage is like and what processes are actually causing these page faults to happen. So in this case, if we run the trace, Google Chrome and MDS stores are the two programs, the two exec names that are triggering the most page faults. We can aggregate in different ways and drill in further into these things to investigate more. OK, and then going back in the other direction, away from this back towards our application land, where I am happier, you can add your own probes. So developers have already and probes are essentially like names for events that are going to fire when they happen. And developers have already done these for some widely used applications and dynamic runtimes. So Postgres has probes built in. MySQL does. Java, Erlang, and of course Ruby has these built in as well. And so there's also libraries out there for Ruby and other languages that allow you to write your own probes yourself and do even more sort of domain specific events and trace those without even having to drop to the C level or anything like that. So think about this sort of statically defined tracing, these user defined probes as almost like adding log statements, except for that you've kind of got a log aggregation and search system, like the Elk stack or something like that, all set up and ready for every application on your system in de-trace. So it's pretty cool. I mean, there's not the persistence, it's not a perfect metaphor. But logging is a decent way to think about it. OK, so an example here, you can trace things that make sense, as I said, for any application or language runtime. Here, I'm tracing all the events whose provider, which is that first piece, starts with MySQL. So, and we do similar truncation and printing as before. So the MySQL developers wrote some code in the MySQL application that lets de-trace see these sort of internal events that happen when I run a few simple queries. So things like query execution, parsing, the select statements starting and completing, and then reading and writing from the network. These more domain-specific things can be interesting to trace. OK, and here's where things get really cool and almost mind-breaking for me anyway. So you can trace arbitrary functions, so even in the kernel. So without the original developers of an application writing any tracing code at all. So de-trace can dynamically instrument the running code.  There's zero performance cost from de-trace. You essentially flip a switch, spin de-trace up, and dynamically insert instrumentation to running code on the fly in the already running process. And this is possible because de-trace is running in the kernel and it's able to sort of manipulate stuff as it sees fit. So this is really sort of magic stuff and it's a thing that sets de-trace apart from many other sort of systems of observability tools. The D in de-trace, incidentally, is for dynamic. It's for this sort of magic dynamic tracing bit. Okay, so here's an example of that. This code traces and times every kernel function that's being called while the de-trace is running. So there's two probes here. There's any kernel function being entered and any kernel function being returned. So the first one sets this thread local variable, the self arrow in. And the second one checks to make sure that thing's present with a pattern match. It looks sort of like a reg X there. It's a guard statement. You're able to filter events by pattern. And the idea there is that if the return event fires first, like if you happen to start your trace and it'll return fires first, you don't end up with these weird numbers. So we make sure that the function has been entered and then we check when it's returned and see how long the duration was and aggregate those durations up. Okay, and we aggregate here with this quantized function which is a bit of a cool thing where it lets you see a frequency distribution of how long each of these things took. And so this kind of blows my mind. We can actually look at how many nanoseconds different kernel functions take to run. And here's just an aggregation, but with any of these, we could drill in and see what kernel functions taking the longest to run. And if we're interested, we could even download the kernel code. I mean, the Mac OS 10 kernel code is available to download online. The Mac ecosystem isn't completely open source, but the kernel is, which is pretty cool. Okay, it gets even wackier in that you can not only trace function entry and exit, but also all the way down to the instruction level. So like two or three assembly instructions down, however many bytes offset from the start of the function, and you can trace that event happening. So personally as an application developer, I don't have a lot of use for that, but I can imagine that if you're further down and you're reading and writing a lot of assembly language, this could be really cool to look at. Okay, so DeTrace gives you a lot of ways to get insight into what's going on under the hood. So what does it look like actually in practice to investigate performance problems using it? I think all of us have probably had experiences where code ran way slower than we think it should, and we just didn't understand why. The thing's slow and it's upsetting and it's frustrating, honestly. Jeff Hodges has this great quote in this article where he says, it's slow is the hardest problem you'll ever debug. And I think it's a really, really good point. His scope, he's talking about his broader, he's talking about distributed systems. So there's a lot more going on, but this stuff's not easy even on one machine, I think. Okay, so we're gonna look at an issue for the rest of the talk that happened to me in real life, but at the time I didn't know how to use DeTrace at all, so the solution would be just terrible, it would be me just stumbling around in the dark if you were to watch that. So we're gonna look at an idealized version of a way you could investigate performance problems with DeTrace. Okay, so here's the deal. My team has some tests that are intermittently really, really slow, like 30 seconds. We expect them to be milliseconds, maybe even less. Computers should be fast. They should be really, really fast, not 30 seconds to execute a single test. And we don't know what's going on, it's not clear what's going on. What is going on, what is it doing? We rely on fast feedback from our tests so that we can make sure we knock out bugs before they get shipped to production so that we can ensure that we fix them quickly, and then we don't have to wait two hours for feedback, right? Okay, we know that it's not just one test that's affected, it's not just one person that's affected. We know that there seem, you know, anecdotally to be a few people who have it worse than others, but we're not really sure. It's not clear if it's the features they're working on. It's not clear if those people are just more vocal about the problem. We think it might be worse in one part of the team, but we're not really sure. Okay, so anyway, this test suite is taking way, way too long. It's making us demoralize many minutes running for just trivial changes, and we rely on fast feedback. Okay, so let's make a quick mental note and even feel free to yell some out of some ideas. What could be going on here? What are some hypotheses? What things could we check for? What are some things that could cause really, really slow, like 30-second level wait times? Database lock, is that the same thing, cool? Running out of memory, great, yeah. And just yell them out, no need for me to acknowledge. N plus one query, great. Sorry, two at the same time, but maybe I made a bad idea. Network calls? Cosmic rays. Cosmic rays, excellent. Cashmasse, right, yep. Slow desk IO, connection pool timeouts, yeah.  There are a bunch of things that could be going wrong, that could cause really slow programs, but we don't have enough information yet to do much more than hypothesize, and here's where I'm sort of trying to make that shift from immediately jumping to let's implement some solutions to, like, using some other methodology to understand the problem better. And you can probably guess that that methodology, based on the context clues here in this talk, the next step is gonna be to use de-trace to gather more data and to understand the problem more fully. And, you know, that data we gather is either gonna support or detract from our hypothesis, hopefully, but either way, it can generate new ideas, we can generate new ideas based on what we learn. But we could start with a hypothesis and see where it leads us. It's not the worst idea in the world, because we have the tools to look at. Okay, so what are the usual suspects?  Both in Ruby and the JVM, I always think it's gonna be one of those two things. So that's my instinct, that's my gut reaction. And a lot of times, that would be right. So let's test that idea. Let's start with garbage collection. We can calculate the duration between garbage collection begin and end events. Here, we're using the Ruby static tracing affordances to check when the garbage collection mark begin and end phases occur, and then aggregating on, or actually not aggregating, but just printing out the duration every time it happens. We could do the same thing for the sweep phase. In fact, in the output, you'll see the exact same thing that's going on in the sweep phase. So this is not the only way to learn about how garbage collection works. We could add Ruby code to do it, and there are a lot of other tools, but this works. Okay, and with the code here, there's not a ton super interesting and new, except that since we're dealing with nanoseconds, we divide by a million to get our nice, sort of human readable millisecond level, or column readable at any rate. Okay, so, and here's the result. The problem definitely surfaced here. The 30 second pause definitely did happen, so I'm not gonna try and trick you with any of these traces we see from here on out. You can assume that every time I show you a trace, the 30 second pause, the problem did actually occur. But the slowest GC pauses that we saw, or the slowest GC phase execution runs, were like on the order of tens of milliseconds, 38 milliseconds I think is the biggest, no, there was a 71 millisecond one there. But still, tens of milliseconds, nowhere near the 30 second culprit we're looking for. So, this is sort of an interesting thing. Let's back up for a minute, and think about this problem at a higher level. When something is slow, it often means that some resource is limited. In fact, it may always mean that some resource is limited, whether it's hardware or software, but I'm not prepared to make that argument for you today. But resource limitation is an interesting way to think about performance issues. Brendan Gregg, terrific performance analyst, he has this thing called the use method, utilization, saturation, and errors that elaborates on what resource limitation means, and some other stuff, which I definitely recommend checking out. But it could be a software resource like a lock or a connection pool, could be a hardware resource like CPU or memory or something else. So, let's back up and think about resources. Think about this problem in terms of resources, figure out what resources are limited. So, let's start at the system level. What system resources might be having issues? So, here are a few system resources, this is in all of them, but these are some that we could take a look at. We could do that with eTrace. So, CPU is a pretty important resource. We can sample the on-CPU processes 997 times a second and aggregate the counts of those executable names that were caught in essence on CPU, aggregate the counts of those things, like how often they happen. And it turns out that we find that a program called kernel task way down here at the bottom is by far the most often on CPU. Kernel task, we might not know what it means, we probably can surmise that it has something to do with the kernel. We could drill in more and figure out what that program's doing. By grabbing kernel stack traces, wherever the kernel task program is, the one is on CPU. You can also grab user land stack traces. There's some caveats that we don't really have time to get into right now around symbol resolution and compiler flags, but you can also get user space stack traces. So, we're looking at kernel stack traces here and if we drill in and see what they are, we see that it's pretty much all idle. We've got this at the very bottom, machine idle is the name of the kernel function that's being called and we can pretty much guess based on the good naming here that nothing's going on. The CPU is not burning, this is just the stack that's on CPU when nothing's going on in CPU. And if we bounce back up to the full list of programs that were on CPU, our Ruby process actually only has up three times on CPU and all the time we're tracing. So, there's really not much going on CPU-wise here. And to me, this sort of discovery feels a bit like a let down or it might feel like a bit of a let down. But it's not, I wanna shift everybody's head if you're thinking this is a bummer that we haven't found the issue. This is not a bummer, this is an awesome discovery because now we can rule out any hypothesis that has anything to do with Ruby being on CPU. So, this is like tight loops, infinite loop sort of stuff. This is too many threads competing for CPU. This is garbage collection. And with any of those things, if it's not obvious to you that those things are CPU intensive, you could write some test programs and see how they behave to guarantee it. But those are all CPU intensive when we've just ruled all those out in addition to more. And it's even better than that. It's even better than ruling out Ruby being on CPU. This rules out anything CPU intensive. This is like database result set sorting. This is anything at all out of process that would use CPU heavily. So, I think sort of meta lesson here is that starting early with these broader resource questions is a good strategy. It feels a little bit similar to a binary search, right? Where you have this wide search space and you want to trim out as many options at a time as possible. If you investigate closer to log in of the possible performance issues, you're gonna be a lot happier than if you have to investigate all N of them. Okay, and so we can rule out memory and disk slowness with other detray scripts. So, those aren't the issue. So, that rules out tons of possibilities. So, maybe we take a look at networking next. We know that we definitely do a lot to call to the database like any good Rails app. We know that, yeah, networking is a thing that happens a lot in our app. And if none of these pan out, we could investigate more system resources or we could go over to the software side. So, let's start with networking. Let's trace all the socket connects and see what's low. So, there's no code on this slide. There's just the URL. And the reason I'm doing this, it would take me a long time to write something like the thing that's in the script. It would take me a while to get in there. But what's in the script works. It works well and it uses syscall tracing to give you socket connection latency. And I show you this because I wanna emphasize you don't have to write everything yourself. You don't have to be a systems wizard or even invest a ton of time in detrace to get a lot of use out of the tools that it provides. A lot of scripts come with your operating system and others you can pull down from GitHub and elsewhere. Okay, and so the result when we run this, we see that our database connections to 127.0.1 are pretty fast. The longer latencies we do see are like 20 to 40 milliseconds which is longer but it's not like there's thousands of these. And I see that I've got some typos here where I have the wrong executable name process. But on the other hand it is interesting that we've got these connections to this external host, the 7<IP_ADDRESS>. I don't recognize that IP address. I expected only local connections to my local database. That IP address we could grab around our code and our configuration files. We find that it's not anywhere. It doesn't appear anywhere on the code base. So it makes us sort of like wonder what is this IP? What is this host? Okay, so in order to figure out what host name and IP maps to we can do something like reverse DNS lookups which aren't always reliable as we found out, this didn't end up mapping back to anything that was in our code base either. But DNS lookups are typically over UDP. We could see if there are DNS queries that might end up mapping back to this IP. So it's essentially like running the process backwards to trace back from where we end up. So DNS lookups are over UDP and after digging a while into UDP tracing I got a little overwhelmed because there's a lot of UDP stuff going on in my machine and at any given time there's some asynchronicity things happening. But after doing a little more study and thought I learned that most DNS calls go through this C library function called git adder info. And so we can use that fact to use dynamic tracing and figure out what hosts are being looked up and how long they take. So we can dynamically trace this git adder info for the process ID that we're interested in. Grab the timestamp and the host. There's a bit of a dance and this copy inster is about copying a pointer from user space to kernel space and so that it can persist through the calls. It's a whole thing. If you wanna write a bunch of scripts you'll have to learn a bit about it but don't worry about the details here. But then at the end you print the duration in the host name when the call returns. Okay, and then when we run it sure enough we find this someplace.com URL's getting looked up and even better we find that it takes around 30 seconds to resolve and if we do the DNS look up ourselves we can see that that does map in fact back to the IP address that seemed to be seemed to be the external one. So it takes 30 seconds to resolve. So not only is this a solution to our why are we making external connections requests it's a solution to our entire problem. Incidentally the someplace.com it's not just me making up a fake URL for the talk this is actually literally the URL, the domain that was in our code base. It was in the test code it was not in production code don't worry this is all completely isolated to tests. So but this is the underlying issue this is the whole thing. This is the game over. The people getting the slow tests are getting slow DNS look ups for this host. And we could drill in further and learn more about networking but we know what to fix now. We have the slow DNS look up for this particular host. So we can complain to our ISP we can switch DNS providers we can replace the someplace.com references with example.com which is always fast or we can do what any reasonable person would do and fake out the network calls and tests to just make sure we never connect to external URLs and guarantee that across the system. But and it feels really good like now we fully understand the issue. If it's super like I don't know freeing to have this ability to go from this very specific you know this sorry from this vague idea like the tests are slow the tests are really slow sometimes to this very specific DNS look ups for someplace.com or slow and they're intermittent due to people being in multiple locations flaky wifi DNS caching stuff but it feels really really empowering to have made this jump down to the details. Okay so let's take a step back here and review what do we learn here in this process. So one answer is we learn not to do these two specific things and it's absolutely true right. Don't connect to external services in the test suite and two don't have a bad DNS situation make sure your DNS is fast right. Those are absolutely correct that and that turned out to be the solution to our problem. So those are correct sure but on the other hand if we you know look for that specific problem or problems in that space every time something's slow we're only gonna be efficient in solving performance issues if those are the only problems that we run into and for better or worse you know we've got as devs and certainly my intuition tells me to focus on whatever happened the most recently and or whatever just sticks out in my head the most based on a blog post I read or what have you and so those are the problems that I'm able to come up with solutions for you know very quickly but I think the bigger better takeaway here is that by doing performance investigations in a more disciplined and principled way we can make ourselves more efficient in solving any problem not just the ones that come to mind or happened recently. Okay so DeTrace is absolutely not the only tool we could have used to track this problem down but it's versatile there are many paths DeTrace gives us the ability to ask really broad questions like what's using CPU or really really focused ones like how long is the specific low level C function tick to run and DeTrace is programmable so it means we can use it to build our own custom systems tools. Most of these scripts I wrote for this investigation but of course now I can just copy them and then in my bin directory and reuse them again on future investigations to answer that same question if it occurs again in the sort of binary search or it's not really a binary search but some sort of search space. And I want to encourage you that it doesn't take years of study to get great use out of DeTrace or tools like it. You can get started with a few one liners like the ones I've shown here or the ones on the websites I'm gonna link to you later but you can get a lot you can get pretty far with just some thought about what questions you wanna ask and you can get a lot of insight. Learning about how systems work at a lower level than you need day to day can be really helpful for avoiding problems down the road and of course obviously when you're in crisis mode trying to solve issues. Okay so where do you go? What do you do next if you're interested in learning more? Well first I would definitely recommend reading the free online resources. Brendan Gregg's blog in particular is a gold mine just everything he writes not just the DeTrace stuff and the DeTrace guide right below is really thorough and I think it's pretty well maintained. It is focused on a Lumos so if you're using it like me from a Mac perspective then there may be some things that don't work or some things that you have to sort of like infer differences. There's also a really good print book available that does have some OS X, OS X, I don't know. Sections available and if you're like me and just really like having print books or if you get through the examples above that I would definitely recommend a print book. Okay and then of course as a developer I always learn a lot by reading other people's code so it was really helpful for me to sort of dig around in the scripts that other people have written and see how they work, tweak things, see how things break or change in their output. A lot of these are totally approachable.  There's a bit of a caveat here in looking at de-trace examples that people have up on the internet or on your system is that depending on how any given script is written it may be directly useful to you or it may be out of date and it may just not work and that's sort of a fundamental issue, right? De-trace gives you this enormous power to trace arbitrary functions even down the instruction level so if there's a script that you're looking at that does some of this dynamic tracing stuff and let's say the function that it's dynamically tracing is removed or renamed or changes semantic, changes the way it's called in the next version then your script is gonna break or it's not gonna work as you expect and this is sort of a fundamental issue but there are sort of more stable de-trace probes that you can count on working from one version to the next as well and as you get into it, you'll learn more. So I think it's also worth extrapolating in two directions here like using de-trace for other things besides perfect investigations and also using other things besides de-trace. So for the first one, I've personally used de-trace way more like as I said, I'm using it on my laptop. I've used it way, way more for just general like learning about how my computer is working than for performance investigations but it's taught me a lot in that respect and it's taught me stuff that I can apply to future issues. And then secondly, if you're on an OS that doesn't have official de-trace support see what other tools you've got available to you to get your questions answered. And of course like many of us deployed to Linux and on Linux there is a de-trace version for Linux. It's not sort of an official thing unless you're on the Oracle one but there's also a system tap which is try and solve similar problems and there are some really exciting things with the latest Linux releases that if you see Brent and Greg's blog you'll know what I mean. There's some really exciting things coming out in Berkeley packet filters in like four nine kernels. So that's really cool stuff as well. The IO visor GitHub organization is where a lot of that lives. So de-trace has taught me a lot about performance operating systems and problem solving in general and I hope it helps you too. Here are a few resources to check out. These slides will be on speaker deck and I'll tweet when they're online so if you follow me on Twitter you can get notified when they're up. And I'd love to take any questions you have. Looks like we've got some time left and please grab me anytime also that you wanna talk throughout the conference. I always love meeting new people and talking about technology and especially come say hi if you're interested in talking more about AtheLight and how we help companies to build and maintain reliable and flexible software. I'm in Chicago, London, LA, New York and we'd love to talk. So thank you very much and I've got a little time for questions. Right, integrating something like de-tracing into the testing framework. Yeah, I can imagine doing so but I guess there are a few issues that would probably make me shy away from that. One is that typically the de-trace program that you're running is external so it's almost like you'd have to shell out and then run it. It also needs root permission so you'd have to shell out and pseudo out which means your test suite would have to be running as root which I probably wouldn't advise. But it's definitely, yeah, I can imagine that sort of thing being useful. Like maybe you wanna say there should be no DNS calls ever made period and you wanna run that trace. The orchestration would be tricky, right? Cause you'd have to spin up two processes now like one, your de-trace thing and then the other test suite. Learning about which syscalls are relevant. Yeah, so I mean my approach so far has been to essentially spy on all of them. There's this program that's built in de-truss that's based on de-trace that gives you a picture of all the system calls that are being made. So if you're, like what I've done so far has just been to kind of like dig into, run a trace against a program that I'm interested in how it's doing, what it's doing, see what syscalls are being made and then just look at the man pages which are pretty extensive for each syscall. Yeah, so that's most of it. And de-trace will also, for like file opens it'll, I believe it'll give you the name of the file that comes out. It might be another program besides de-trace. So yeah, a lot of it's just been like observing what's happening on like workloads that I understand that I've like made some small program to do a thing and then make sure I understand the stuff that I want to understand on the other end when the output comes out, when the full output comes out. Yeah, man to name of the syscall. So the man command gives you like the option of passing two arguments in this. The first argument is like the section of the man pages and two is the syscalls section. Okay, I'd be happy to take questions or as I said talk with any of you afterwards up here. So thanks a lot for coming and I appreciate it. I appreciate it. I appreciate it. I appreciate it.