If I speak here, you can hear me? We're about to stop. So if you're closer to the mic, you'd better putt it. If it's for Hal, we'll be able to get it closer to the mic. Can you hear it? Yes. Hi, everyone. We want to start. And I don't have high quality sound. No, you're perfect. That's much better. I would like to sing the blues now. Please do it. No, no, no, no, no. Good afternoon and welcome. I'm going to read my notes because my brain doesn't work anymore, so I have to read my notes. I want to say happy convening, everyone. It's out there last year. It's really great to see all of you in the same room. Yes. Yeah. These gatherings and these conversations are invaluable, and I hope that you appreciate them as much as I do. I want to begin by acknowledging that we are standing on the ancestral land of the Oloni Ramay Tush people. May their spirit guide us through our work this weekend and propel us towards collaboration and deeper understanding of our own shared humanity. A lot of time and effort goes into putting something like this together. Let me thank everyone who has and is making this gathering possible. Who, Golden Thread staff are in the room. Stand up for the Hewlett Foundation, who is the only foundation that supported this convening financially. I'm grateful to them. Our fabulous interns this weekend happen. Just say it that these sessions, the public sessions are being live streamed on HowlRound. If you don't want to be live streamed, just sit behind the camera. Also, I don't know if we are getting comments on Twitter and Facebook. All right. All right. Well, Julius is managing that, and if something exciting comes in, please share with us. We should have said this earlier, but I'll say it now. If you have turned off your cell phones in case of an emergency, there's a door here and there's a door there. And already today, we've had two sessions, a fabulous conversation early in the day about how to get more of our plays on U.S. stages and a really productive conversation just now about mentorship and intergenerational support in our community. I'm hoping that these conversations will generate actionable strategies that we can then as a steering committee or as representatives of our community implement. In a few minutes, I'm going to invite members of the steering committee to the stage to provide some context and background information to how we got to where we are because as you can imagine, years and years of work and community organizing has gotten us to the point that we are here. Where am I? On my page, I don't know. Including what I wanted to say, including the MENA survey, which Kate will talk about in our work with TCG Theater Communications Group, which Andrea will talk about. As I was thinking about gathering this weekend, I was reminded of a turning point of my own thinking about Golden Thread and Golden Thread's role, which is actually surprising, so I thought I might share it with you. A few years ago, we were doing a strategic planning process for Golden Thread and we were at a staff and board retreat. We were having conversations about what distinguishes Golden Thread's work from other theater companies, and we were imagining a world where Middle Eastern plays are being produced by many, many theaters across the U.S. In that environment, would Golden Thread still be relevant or not? As we were digging deep into that conversation, we have always acknowledged that we kind of came up with the term Middle Eastern American theater. We made peace with the term Middle East, despite its colonialist baggage, just because of practicality and the fact that it's recognizable, but always being aware of, super aware of the connotation. So we've always thought about the term Middle East a lot, but not the American part. So it was interesting at this retreat, the conversation was about, but we're not a Middle Eastern theater company. We're an American theater company, and you may be surprised to hear this, but that came as news to me, right? That took me, like, I was taken aback by that statement. And then I started thinking about, well, what does that mean? What does it mean to be producing this work in the U.S., in this environment? And this retreat was in 2013. It wasn't like during the 2000s while Iraq was being invaded and destroyed. We survived through making theater. We survived those years through making theater. So we've always been very aware of our relationship to the Middle East and what's happening there. But maybe I think that turning point was when I actually began thinking about our community here and our existence as a growing immigrant community in the U.S. and what that means for us making theater in San Francisco and in other cities. I think that discovery, you know, then we called ourselves the first American theater company devoted to plays from or about the Middle East. That's where that sort of statement came from. Before that we were a theater company exploring Middle Eastern cultures and identities. So it's a subtle difference, but it's also a profound difference and it's a way of, I think, looking at your work differently. So it made, I think that discovery, that statement made me take greater ownership of Golden Thread's role in the U.S. as an organizing force in the country. Before that I've always had a great relationship with Jamil Hury at Silk Road Rising and we had, with the LARC, we had launched a play commissioning program Middle East America, which Jamil called it Middle East America, and it was an ongoing program. So we had kind of a national network or partnership, but I contacted Jamil after this retreat and talked to him about what I thought was the need to actually gather our community. Because I don't know, I thought if I hadn't realized that I'm a Middle Eastern American or that this company is an American company, then there must be so many others that are in that same situation, right? And I don't know how it is for you trying to tell these stories in your community, but it can be a lonely experience, right?  And so Jamil and I began talking about how do we gather our community. The LARC had been a partner to us. Catherine Coray at that point was becoming more involved with the Middle East America Initiative, but also reaching out to artists in the Middle East. And so we began talking about hosting a convening and Catherine was able to host a convening at the LARC in 2016. Jamil was able to organize what TCG calls affinity group meetings during the national conference at TCG. And we began kind of meeting our community, right? Like the artists that self-identify as Middle Eastern American theater artists. And we learned a lot in that process. I think one of my specialties is sort of to sense what's missing and then try to, I don't know, try to address it if I can. So at the affinity sessions, we began sensing the absence of leadership and representation in our community. And I don't have to tell you that representation matters, right? Being able to tell our stories in our own words matters. And I'll just say this and if you want to, you know, question me about it later, do that. But the reason why the wars in the Middle East are happening, one reason is because it is justified in the popular media, in the popular imagination of Americans, right? And that's what we're fighting. That's what we're trying to counter. That's not an easy thing. So that requires leadership. That requires national collaboration and partnership among all of us. So the affinity group meetings and the convening at the LARC generated some momentum. A lot of positive things came out of that gathering at the LARC in 2016. Mona Mansour, fabulous playwright, started a Middle Eastern American playwright group at the LARC, which just recently had an evening of readings, very successful. The Maya directors actually, that idea came from that gathering at the LARC. And now there are four directors. I don't know if everyone's in the room, but, you know. And then the idea of us needing a bill of rights came from that gathering. Because the trajectory of the growth of our community, like we had gone from questioning our right to tell the stories of people in the Middle East or our right to even tell our family stories to claiming and demanding space for our stories, right? So when you make that jump, then you need an awareness of what your role is when you're in the room what is appropriate for them to ask you and what it isn't. So the Bill of Rights came out of that need and what I heard a lot of people say about constantly being put in the position of being the cultural consultant without getting paid for it or someone in the room sort of taking the role of being the cultural authority or the political authority on the situation. And so we developed the Bill of Rights. We sort of tested it out at affinity group gatherings at TCG, got some feedback. And then the big tonal shift happened which we made two documents. One was the Bill of Rights for Middle Eastern theater artists and then one was an open letter to producers in U.S. theaters. So both of those kind of addressed the same issues, but from different perspectives and the tone is different. So slowly over the years, a lot of people have already put in a lot of sweat equity into this movement. People have been working for more than 20 years to get us to this point and we're slowly moving from individual ideas to collective action. And collective action requires generosity, respect, openness to people's individual experiences and a space where we can have conversation and this gathering is about making that space. Conversations this weekend. I want to invite you to visualize yourself as a theater maker. Visualize yourself in 10 years. What kind of theater maker are you? What kind of environment do you want to be making theater in? Who are you collaborating with? Who else is in the room? Who is running that theater company? What other kind of theater are they putting on their stages? Who is on their board? What about in 20 years? What about in 50 years? I want you to visualize that. And then let's think about what can we do today? What kind of ideas and recommendations can we come up with today to bring about that vision? Our work today is about Middle Eastern American or Middle East North African community of theater artists. But I also want to offer that as we define the future of Middle Eastern American theater, we are in fact redefining American theater. Visualize that. Let's talk about that future. Yeah? Thank you. And now I want to invite Kate and Andrea to come on stage and Kate's going to talk about our community survey and some of our findings. And then Andrea is going to talk about our work with TCG. Can you all hear me? Yeah. Okay, hi. I'm Kate Morhini. I'm the artistic producer at North Theater in New York City. Oh, sorry. You want me to lower this? That's great. Thank you. Yes, so I work with North Theater in New York City. We support, develop and produce the work of artists of Middle Eastern descent. And as part of, so continuing the work that was just discussed, we were asked to be part of a group of theaters of color at the latest TCG theater communications group conference. And as we began to sort of prepare for this, and Andrea will talk a lot more about our time there. But as we began to prepare and think about sort of how we can best serve the community, we thought, well, we need to first get a sense of who is this community. We have a sense sort of of who, these are some artists working in New York, these are some artists working in California, Chicago, across the country, but who are we as a group and how can we best represent this group and come together as a coalition? So we've created this MENA Theater Community Survey. And we thought that as the community grows in the U.S., this would be a way to build a national network to advocate for our community as a whole. And we hope to use this survey to better understand our needs and help advance our cause and best represent the concerns of the community as a whole. As was mentioned in one of the earlier sessions today, I just want to say it again, we decided to define Middle East and Middle Eastern very broadly using a definition developed by Golden Thread and Silk Road Rising. The definition is that we embrace the multiplicity of ethnic and religious identities that span Southwest Asia, North Africa, Central Asia, the Caucasus, parts of Mediterranean Europe, and our diaspora communities. We understand that our respective backgrounds in terms of rich pluralism and interconnectedness. So I just want to say that that was really important in coming up with this, that it felt like an inclusive document. So in the initial survey, it's still a live, active document. So if there's anyone here who hasn't filled it out, we can maybe post the link. The goal is to get as many folks as possible to fill it out. We have 100, as of June, we had 191 participants from both the U.S. and across the world. There were 16 theatrical disciplines represented. And this is my favorite. Collectively, we had in this group 2,492 years of experience in the U.S. The most important takeaway, there was some demographic information that we collected. We found that the top five theatrical disciplines represented were performers, playwrights, directors, producers, and administrators. But there were, of course, folks in many other fields as well. Those were just the top five. We found that of the regions in the U.S. that were represented, 8.7% were from New York, 23.6% were from Chicago, 11% were from the San Francisco Bay Area, and 8.9% were from LA. Again, this was just our initial survey of the folks who we were able to reach out to. We also had 18.8 folks from other regions, including a number of folks from outside the U.S. We measured age range, whether or not folks recognize a person of color. We found that 27% of the people we surveyed identified as a member of the LGBTQIA plus community. So we just wanted to see who exists in our community. We found that a number of folks were members of professional theater unions or organizations. 66% were a member of actors' equity. 5.7% were a member of stage directors We found that 14.4% were members of theater communications group. So this data is helpful and it'll help us to sort of chart the growth of our community over time. The biggest takeaway, though, from the survey was we wanted to know what were the top most important issues facing our community. What did people think were the things we should most focus on as a network? The first was more of our plays produced at major theaters. So that makes sense that that was the top thing that people thought we should focus on. The second was greater national visibility followed by more seats at the table or greater voice and impact followed by access to funding followed by creating a MENA community network which is some of the work that we've been doing. So with that overview of the survey I also have, we have hard copies of it if anyone wants to take a look later on. I'll hand it over to Andrea to talk more about our work at TCG in the theater communications group conference. Hey y'all. How you doing? What a beautiful room. I'm so excited now really. What a beautiful room. I'm so excited to be here. And so this is quite off the top and I will do my best to appropriately represent. For those of you who don't know me, my name is Andrea Asaf. I'm the artistic director of Art to Action, Inc. which is based in Tampa, Florida. And I'm Lebanese-American and a third generation. And we all come to this. I feel like this is a moment I feel like we're in a delta right now. This is a moment of many streams, many rivers conjoining and converging, convening. Yes, thank you. And so I'm going to talk about a little bit of some of those other streams. Perhaps the main one the largest body of water leading us here is what Terange began with. But also so TCG Theatre Communications Group has had over the last several years an initiative under the framing of EDI or equity, diversity and inclusion. And I want to give props to TCG for really listening to artists of color in the field many years ago that had a lot of feedback about how we were feeling served or not served well enough and the leaders the leadership in TCG that really took on this initiative to make substantive change in the organization in a way that would impact our experience going. So if you haven't been to a TCG conference or you haven't been in many years go back or try it out because it feels quite different now and that's partly due to this cohort work that TCG has been doing led primarily by Emilia Ketchapiro and Elena Chang. And so what's unique this year is that they're in the fourth cohort and they decided to make the fourth cohort focus specifically on theaters of color and particularly networks of color within the theater community because some feedback that was received earlier on was that from artists of color who were participating was that this feels like white America is still the center of the conversation. And how do we talk about equity diversity and inclusion and even go further and talk about justice in a way that doesn't center the historical center that is truly as complex as we all are and as our field is today, right? So they made this commitment to bring together theaters of color and networks of color that serve theater artists. Also, multiple streams leading to a Delta, I guess, also I want to lift up the work of Leslie Ishii and East-West players and the consortium of Asian-American theaters and artists that also was convening networks of color in 2016 and doing that work prior. To start to talk about how could the Latinx theater commons and the multiple black theater networks that are in the field and the consortium of Asian-American theaters and artists which is relatively new and trying to understand how to be inclusive of and deal with calling the Middle East. Understanding that a large part of it is Southwestern Asia and also North Africa and all of our complexities, right? So Leslie Ishii was leading organizing that work under the auspices of Akata convening in 2016. TCG was working on co-port four and developing that and we're working on figuring out how to convene us together. And these multiple streams have led us to this moment and I also want to make sure that we acknowledge and lift up other organizations that aren't necessarily theater organizations but that have had a big impact in our ability to come together and so I want to name Mizna and Raui, the Writers Network and the Arab American National Museum and their convening currently called MOVE and make sure that we recognize that all these things bubbling in all these places in the 21st century are leading us to having greater collective voice and power and input and vision yes? And that we cannot do that in isolation. We are in a historical moment in which we must I feel and like Taranj said, talk to me later if you don't agree but I feel we must embrace our status as a community of color in the United States. Now I know that's complex for us in the way that it's complex for the Latinx community, right? Because many of us pass as white, many of us live our daily lives as white, many of us can't pass and can't live that way, right? And yet we're all trying to have we're all trying to share our cultural and artistic experiences within this highly racialized society that we live in called the United States of America which is itself a problematic term of representation. So, if we are trying to build voice and representation and collective power around cultural representation and around, you know the direction that this country is going in and who we are and what we do in the world that we're all a part of if we live here like it or not, we're all part of what the United States is doing in the world, yeah? That's a lot to reconcile for a lot of us, right? But we can't do it alone. As complex as our communities are and hard as it is for us to get on the same page and to get together about who we are and what the Middle East is and where we fit in it and what it means to be that in this country in this context, we are doing that alongside communities of color that have been in this conversation and leading this conversation and doing this work and fighting this fight for a very, very long time and we don't have to reinvent the wheel, right? We can learn from each other. We can learn from the Latinx Theater Commons. We can learn from the Black Theater Networks. We can learn from the struggles for civil rights in the south. I was born in Virginia. I live currently in Florida. I've lived throughout my life south of the Mason-Dixon line at different times and it is different. It is a thing and it is extremely underrepresented. The U.S. and artists in the south continue to be extremely underrepresented which led to the creation of an organization called Alternate Roots, which I'm a part of. Which is a tremendous resource for us about how to organize how to collaborate with organizers about how to have these difficult conversations about race and equality and justice as we're trying to do our artistic and cultural work. It's a lot. All of that is a lot. But that's our charge. And so I want to give thanks to and lift up all the other folks who've been out there in the field doing this work and painting the way and creating models, which we're going to talk more about tomorrow. And I also want to acknowledge that we're not the only ones trying to figure out how to do this now. That Native American theater artists, first nations, first peoples artists and indigenous artists in this country are in a similar place as Middle Eastern communities in terms of there is no infrastructure there is no national network for contemporary artists in theater for them. And so they're also in this kind of early stage how do we get together all of our nations that haven't always been friends let's be honest all of our communities that don't always get along all of our histories that have such complex and different relationships to the United States national government and how do we build solidarity without erasing any of that but really creating a vision for the future as Terrence was saying. So I wanted to plant those seeds and offer some of those questions and offer to you that we'll continue to explore them and unpack them and talk more about coalition building tomorrow. And it is a tremendous really tremendous honor for me personally to be here with all of you as colleagues and I'm so excited for what it is that we're going to create together. Thank you. Alright, so I want to thank you Andrea thank you Kate. I want to invite the rest of the Mina Steering Committee to please join us on stage. Do we need more chairs? There are a lot of us. I'm assuming we're going to do this right? That's Jamil's chair. There you go. And if you're watching we love you. You you to get to know who we are we are self-selected group of people volunteers who basically decided to call ourselves Steering Committee of the Mina community because we have nothing else to do now. You can add that's the one that came to me. And I thought that maybe some of us you've met already some of who introduced ourselves and just say something about I'm actually personally curious to hear everyone's reasoning for wanting to be a part of the Steering Committee given how busy everybody is. Is that interesting? Would you like to know that? No. Basically I wanted to be Who are you? Yousfell Gindy Leroy I just we've been talking about coming together and trying to create a presence everything Andrea was talking about and it's been a long time coming. It's been a long slog and a long struggle and trying to create a presence of the thethedocene here and I just wanted to have my two cents and you know your question and I'll pass it along in two seconds you shouldn't have given me the microphone because I'm going to this can become a therapy session and really unload the anguish and struggle of trying to just say you know we're here we have a voice and your question where might we be 20 years from now, 50 years from now I keep trying to stress that we're not a conduit for the Middle East we are yes and no this is part of our identity but we're also part of the diaspora we're also immigrants and we have this we have a voice that partly emanates from the Middle East but also emanates from our experience here and I sort of want to keep emphasizing that we are you know at least Americans and that has value and that's a voice worth hearing from and if tomorrow all the wars and all the conflicts stopped and ceased and there was a wonderful resolution and we all walked away happy we still have something to we still have something to talk about we're still relevant in that we are a community we have particular experiences and I don't think we should be shy or tentative about expressing that identity on the American stage Thanks, I'm Toranji Ghizaryan, artistic director of Golden Thread I wanted to become a part of the steering committee because I always think about what do I wish existed that I could benefit from and so if it doesn't exist then who can I get together with and make it happen so I think the steering committee was that I was excited to know that other people were interested and were willing to put in the work because the idea has been there but I think the individuals who are willing to do the work has not necessarily always been there so this moment felt rich and rich with possibilities and great potential Hello, I'm Denmo Eberheem I'm an actor and a writer My inspiration to join the steering committee I came from a place as an artist feeling that the discomfort of the stories I want to tell they don't always have a place from an interior perspective and I think the steering committee and the presence of this gathering started to pose these larger questions where I think really for the first time I sort of felt like these are producers and directors these are we're coming at this from a larger perspective and I think I've personally had this question of like how what are the resources we need to be able to tell the stories we want to tell all the way from the production but also the process of wanting to be a part of that conversation very much so with nothing else then to make sure that those questions have a place and a space to be heard and that together we can find ways through I'm Tracy Francis I'm a director and producer and I've been involved with the Middle Eastern a theater community for a long time and like trying to these ideas have been coming up again and again and nothing's ever really happened with them I feel like the conversations have kept going and for me I'm just really interested in actually doing something finally I think this kind of the kind of have coalesced in the last couple of years we're ready to take some kind of action and just to find a way to bring community together and how we support each other as a community how do we make sure we're better represented in the national theater scene and how do we advocate for ourselves so you're online Hello I'm Debora Eliezer she occurs I I'm a theater maker in the model of ensemble theater so I'm very interested in how a thing gets done as much as what it is that gets done the product that is made so for me being in the process of community asking very difficult questions or passionate questions and being part of dialogue with the objective of creating change and uplifting the complexity of the identity of this community and the identity that I feel both from a diasporic position of always feeling othered in some way and also this idea of belonging and where do I belong and asking those questions kind of all comes home here and I love that we're talking about this being American theater because it is no accident that we are in northern California having this conversation as a native San Francisco like that makes perfect sense to me that this is where the delta meets here and I guess that cannot be de-emphasized is how we have that agency to have these conversations and what we can do with that agency it's very exciting to me I want to be on the forefront of that and that's why I'm here Hi, I'm Everon Luchkin he and his I'm a theater director, producer translator and writer now that's new I keep forgetting I guess for me this idea of giving back is a big part of steering committee involvement I think because the reason why I have a career is because of the Middle Eastern community every opportunity really that has catapulted me in any way forward has come from folks here in some way so it felt really important that as this moment started happening that I raise my hand step forward in whatever way I can and just help out and the other thing that was important to me is I started this journey Tarnash doesn't remember this he always gets annoyed with me when I tell it but I'm going to tell the story anyway and I avoided Golden Thread for four years because I didn't want to be boxed in by my immigrant Middle Eastern identity and finally she like forced me to have coffee with her she's like everybody keeps telling me there's a Turkish director who are you and why aren't you talking to me and I said as young men, artists, strong-headed tend to I was like well I don't care about Middle Eastern theater I only care about good theater and she said good we care about we agree on one thing and then welcomed me to the space anyway and then 15 years later I'm up here and I'm a Muslim Middle Eastern theater maker like I own this thing so if I could have that journey because there was a space and I got lucky that that space was in San Francisco when I arrived to give me space to figure that out and to give me resources so that I could make my mistakes and move forward in the way that I needed to I'm hoping that and I've seen so many artists not have that in other cities because Golden Thread doesn't exist there so I hope is that hopefully I can help make a national thing so that no Middle Eastern emerging artists fresh out of school or in school feels like there isn't someone like them or an organization for them so that just feels really important to me Hi I'm Pia Haddad I'm an independent creative director I guess my reason is I'm Lebanese but I'm identity confused like a lot of people maybe I was born here I grew up in Europe and then in Lebanon anyway but so when I started producing I'm interested in intersectional work because that's when I I believe I thrive and have different voices but then I realize in order for work to be really intersectional that needs to be everyone has to have the same same seat at the table or same opportunities to have seats at the table and the fact that the Middle Eastern community doesn't have a coalition and also the first play I produced was Middle Eastern and seeing all those challenges and opportunities that came up while doing that so wanting to be part of the conversation and owning my roots in a way also well since you kind of already heard from me I'll give a really personal answer which is I grew up in places in the United States where there was no Middle Eastern community around I didn't grow up in community right I grew up totally with that identity confusion and I could just kind of be that exotic racial enigma that kind of like passes a lot of things and I didn't always have to talk about it until 9-11 when I lived in New York and the public perception of my identity changed overnight really overnight like shockingly so and that was also when I was really coming into my own as a creator of original work and as an adult artist and so really since then my adult life artistic journey has been about understanding what this is to be Arab American to be Middle Eastern American and in the meantime I've done a whole bunch of organizing and convenings and network building with a whole bunch of other people and not my own community then I am still finding and discovering right and making making up and creating by virtue of being here and coming together and so it's really personal work for me and I'm really glad to be in it we've already heard from you as well Kate Morchini from North Theater in New York also a freelance director and I joined Mr. and can be really for two reasons one is that similar to whatever it said I was really lucky to find an artistic and artistic home at North really early on after moving to New York and feel like the Middle Eastern theater scene in New York feels really vibrant and an exciting beautiful community and I wanted to be part of creating a larger a larger community across the country but I feel like I hadn't met a lot of the folks making work in different parts of the country until joining this group and so for me it's about wanting to create a larger coalition across the country so that we can all be engaging and all be part of a larger vibrant community an exchange of ideas and art so that was the one of the reasons and then I think the other is just a desire to as other folks have said a desire for our work to become part of the American canon but it needs to feel like it's an integral part of the tapestry of the American theater not one thing in the season but an integral part of it but it feels like it's at the forefront of the American conversation so those are the two reasons so I thought that this could be an opportunity for you all to ask us questions specifically as your steering committee but I want to open that up by so first of all let me ask are there questions in the house okay where are you right now yeah I'm right this is right now thank you I'm curious since you're from all over the country how do you make this happen and what is your collaborative process like Kate makes it happen we should pass the mic to Kate to talk about Zoom and scheduling but one of I don't think we should underestimate the value of an organized person who is on top of technology notes, reminders all of that please speak to that I don't know how on top of technology I feel like in fact well that's another thing but I do feel like the biggest thing in terms of just logistics and planning has been figuring out how to get us all in the same room digitally it's hard for us to all need a person I think this is maybe the first time that this amount of us have been together in person but you know face to face communication I think has been key I'll be interested to hear what other folks think but the time that we've been able to have Zoom calls to discuss ideas and then decide on an action plan and then move forward with that I think has seemed household I would add that it helped that we had very concrete agendas the TCG National Conference was happening and the question was how do we prepare for that because we were Kate, Andrea and I are representatives of our community in cohort 4 TCG's EDI cohort 4 so we needed to prepare for the first cohort meeting that was going to be last June that was last June where was that? Miami so we had to prepare for that and then we had to prepare for this convening so we had very sort of concrete agendas other people want to speak to us? I just also want to add probably jumping to the end here that this is the first steering committee it's going to all change and that other people will become the next steering committee and when we determine what the next phase is what our next goals are there will be a new steering committee and hopefully as many of you who want to join join and so we can take it to the next level this is just the first iteration we have committed to serving two years on the steering committee we did I think the thing to say because I was on staff I read from the time the first convening for the Middle East America commissioning project happened at the LAR and we've had gatherings like this around the Ori and here I feel incredibly lucky and privileged that I've been at a lot of these and it was really interesting experience and maybe it's a part of my understanding of myself in the space as well the conversations started from questioning the space what does this even mean we are here you have to listen to us we actually have to compromise to there's been a real evolution of the conversation and our relationship to each other and it was frustrating at times because it constantly felt like two steps forward one step back and I must say there was a moment and it really the seed was planted I think 2016 with the CATA and the TCG and the convening there was like a year of convenings for us where all of a sudden it went from so what are we doing and it feels like we're here finally a bunch of people raised their hands and now we're doing something and that has been really interesting in the sense that there's quite a bit of trust at least to each other and that because of life I wasn't at the last call but I knew it was fine there was this sort of because it's national because it's so ambitious that you have to put in the work when you say you're going to put in the work and then you assume that the other people are going to take care of other things and we've had leaders and I'm looking at Andrea and Kate and Taraji and Jamil in absentia and we've also sort of been really beautifully setting course forward and vision of what the conversation might be like how the conversation might be structured so it feels like I just want to say that this feels like a major step forward for me as someone who's been in some of the like a lot of these conversations along the way and it made it so much easier to step forward and say yeah, I'll do that because it was so much clearer now and I think we should that's not a, you know, that's 13 I went in 2008, so it's, you know, 11 years it took 10 years to get clarity enough for a bunch of people who will probably disagree about a lot of things to get in the space to disagree I think that's pretty awesome Tracy, you were going to make a comment and then we have a question on that side I just wanted to kind of frame some stuff so tomorrow we're going to do some sessions about really getting nitty gritty of organizational structure and names and all of that but I also wanted just to kind of pose to you all what you guys feel a national coalition like why that might be important to you or not and what it could do because we've all been talking about it but I think part of this convening is to hear the rest of the community of what this organization could look like and what it could do and so I'm interested in your thoughts Thank you all I think it's partly answered my question but I would like to know I can see that you all have put a lot of time and work into making this happen and you've offered something so I want to know like really what do you hope to take away from this convening what do you need from us what are your expectations and what are you hoping to get from it looks like it's a great turnout so I wanted them um so um like on Sunday, tomorrow the end of the day it is so we'll be doing some more working type sessions and then we will have kind of a smaller kind of working session at the end of the day and we hope to come away with actually a plan moving forward of what we're going to call the organization what we're doing online about it what the organization structure might be this might be overly ambitious but that is some of our goals for the end of this weekend your guys input as that develops so we are making sure we're reflective about the community and I want to encourage us to come up with um not just like the first impulse is to think about what's lacking right so then the next step is what can be done about it right so I want us to come up with what can be done about it what can each of you do about it in your own sphere right um uh everyone mentioned the 2016 convening like the idea of creating a um a group for middle eastern American writers in New York like that came out of necessity and one playwrights willingness to organize it right and large Karim pardon me also Mona and Karim Mona and Karim and then the LARC gave them space right um or you guys like the Maya directors you started talking that year and then in a year or something you had like a partnership or an organization so it's it doesn't have to come from the steering committee it doesn't even have to be done by the steering committee each of you are fully empowered to lead your own little movement or your big movement right and we'll follow you so come up with um actionable what I guess on that really quick yes just to add on to that talking about you know solving a problem or what's lacking I'd also like to think about what we want to have not necessarily a problem we're trying to solve but like what are we dreaming about what are these bigger ideas and helps and things we have that are just awesome that we want to do I also just want to give a shout out to those who can't physically be here in the space with us this weekend if you're following on Hal round please like make comments if you know folks on the steering committee that you want to have longer conversations with or you know send a manifesto like whatever whatever way that we want this to be a truly national conversation and we don't want it to be limited to the people who have the time and resources to be in the room which is why so much of this is being live streamed so please make sure that you connected and find ways to be in the conversation and also I just want to acknowledge that you know nobody really elected us this is a volunteer effort you know in alternate routes we have a saying that is like it goes who comes is like if we show up to do the work then we're taking leadership then we are we are leaders right and so many of us and have the capacity to be in leadership and be in this process and be a part of it and so like a lot of times at alternate routes if somebody says well I think you should do this then the facilitator says that is a brilliant idea you should definitely do it how can we support you because this is all of us putting in our extra volunteer time to make this movement happen and it needs all of us so the door is open I was just going to say one of the biggest worries I had I feel like I'm channeling Leila now Leila Buck is the that we must represent everybody and everyone you know like the last thing I want is someone here or in the world to be like why wasn't I asked you know what I mean and I think when we're creating a national movement there's a certain jet like you know you weren't asked this time and now you know so next time as you said so I just want to say this is an organic movement that really came out of the gatherings and connections that these three or four organizations happen to have we are certainly more heavily representative in the Bay Area, New York, then Chicago, then other places just because those were the places that these companies were located so I just want to say that there is this stepping forward as you get inspired and as you want to is a big thing for me because again, we weren't elected it was a it happened and I think there can be great things that come from when things like that Thank you so much for doing this and my question is particularly about the two last comments which is like people who aren't able to make it here I'm curious even like with the survey how are you able to compile the people that you said that to and who is missing and how can we as a community somehow support this community to ensure that the work is out there and then that I guess my question is really around communication how can we ensure that all of these different initiatives like Gariman Bamba's writer's group and The Night at the Lark or whatever it is happening somehow is all confined in a place so that people know that it exists and it's sort of a big attack to that whether that means like a newsletter now we've set up you've compiled these five things that we want to work on how do we know if the next community is in three years what's really working, what's been improving in these three years do we have like a buy monthly buy monthly every couple of months type of newsletter where we learn what's going on etc how can we make communication really streamlined in the community so that we're all supporting each other and we're asking that that's such a good question to address the first part of the question about how did we compile the folks that we reached out to it was honestly a mailing list of the different theaters that we all work for and it was about trying to have as wide a reach as possible but knowing that there would be no way that we can reach everyone and so I'm so glad that you've asked this because we have everybody in this room and the goal I think is to compile as many people as possible and I think your idea of a newsletter is a beautiful one you know the idea that if everybody in this room takes all the contacts that they have in the community and we compile a massive list and we find a way to be in touch with as many people and it's an ever growing chain that you know that person says oh did you get that email you didn't oh let me add you to the list and I'm going thanks so I'd be excited to talk more and just forwarding the questionnaire you know this is one of the let's just do a survey I got the survey and I just forwarded as many people as possible do you all have a Facebook group? not yet well there is a yes there is a Middle East America Facebook group and there's in the same room Facebook group so those exist but we are not religious we'll invite them to take the survey too I just want to look up something that Tracy said which is part of our charge our time together here is to answer some of these questions it's not that we already have answers but that we're coming together to say what do we want this to be do we want it to be a coalition that has a kind of organizing model do we want it to be a service organization that we all become members of do we want it to be an organization at all as in a nonprofit or maybe not maybe some other kind of something like the Latinx theater commons model so I think that those are all living questions that we're hoping to get your voices out of the conversation so that we have more clarity by the end of the weekend about what's the group vibe what direction does this organically want to go in I want to answer the question as to why I'm here and first I want to start out by thanking all of you guys for all your amazing work can we just I feel it's it takes leadership like yours to get us all together I have been working in theater for 10 years but really my only connection to the Amina community has been to Ryan Everett and a couple of actors that I've worked with but I always felt outside or didn't make the effort so just being here with everyone and similar stories and hearing stories about actors being rubbed in to do free work it feels really good because we all have a lot of commonalities so that's why I'm here and I really want to thank you all for doing all this work there's a question here hi I was just going to ask a clarifying question because this is mentioned that you all had come up with these ideas during TCG conferences are we affiliated with any organization or would this convening be a TCG specific initiative I guess I'm asking about organizational affiliations that exist or would we be something independent we are certainly TCG adjacent I would say the TCG affinity space was one of the many places where conversations took place we really believe that a lot of the stuff that Alina and Emilia are doing at TCG Latinx Commons was part of HowlRound other places so we're sort of in that space I think some of the conversation will be about does this make sense for it to be launched as under an umbrella organization or do we feel like we're going to do it ourselves and set up a whole new model so I think some of the models that have happened with larger institutions like TCG and HowlRound those will be discussed tomorrow currently we're I feel like Kata has been we basically have used the resources of every organization that's already doing this that would help us and have us there and pay for things so that's we're crafty that's what we are so we've done that but in terms of where we take it from here I think that's going to be part of the conversation I had a question I'm not sure if it's going to where it lands but just thinking about your group as the larger as community members for forthcoming agendas and given that we've talked about identity in every single conversation so far is there any conversation or can there be any conversation about the census and how artists are highlighting them in a community so that we get the resources we need yes let's do it are you taking notes or recording right recording right recording right don't forget that idea I was just going to comment I think there is other theater community like Arab American discrimination but they have been lobbying for that for years to get Middle Eastern and African folks on the census so there is a lot of efforts around that there's a lot of efforts I think speaking to integrated community are you speaking in a mic no I just wanted to use your words lift the idea around it's going to take multiple community actions and if we as a community of artists are just coming around with the idea of our own identity and the stories we want to tell identifying the group giving a number to it a size to it so that you can talk about how to finance for it all of this is actually connected to census so you're going to bring board members based on who you serve and how you serve them it's a very large topic and I think as many folks who can be engaged in it that would be great I think that I want to take your question and put it under maybe even a slightly larger umbrella I keep doing the political thing which is nice, it's appropriate I guess but I think it's also as we talk about what this organization is activism art making I think I can say every single person on this panel here carries both of those titles in different ways and I think this specific organization where does activism both within the theater community which seems more obvious and then outside of the theater community in the way that it we interact with larger movements around equity, census Muslim ban, whatever it is you know how we engage with those issues is I think a conversation and it's really a conversation about where we want to focus and where the group feels like which one way is we can engage with everything and I think that's all within range of this conversation so I'm really glad you brought that up because I think that's a question that's going to continue to come up I mean Andrea you have brought up like is this an organization that makes statements about horrible political actions our government takes is that I think we want to do or is that we just only do that when a theater company does a horrible thing you know like where they're playing with us I think it's just a conversation but I really appreciate you lifting that very specific question because I think it the specificity of it is around the practicality of defining the population so that you can address that and I just want to say the artivism of it you know I have a thing about the census in my show just I put it in there so that just to remind people I have a show about Jewish Iraqi identity and it's like you know and about being American and sort of like myths about what it is the fact that you can't take the Middle Eastern box yeah so and I and I support what you're saying Sarah because it's really important that we keep identifying and defining who it is so that then we can have more agency as a group actually that's exactly what I was going to say so it's just we need to identify who's I mean for me what's happening now is huge the fact that we've all gathered together over this weekend we're creating visibility for ourselves we're hopefully empowering ourselves by coming together and by doing so we hopefully placing ourselves on the map so funders look at us and go oh look there they are and maybe we need to direct some of that money towards this group that's assembling identifying themselves and saying excuse me we need to have a voice fund our voice and fund our projects so yes it's just being a little more aggressive about saying we're here you know being visible about it and not hiding I think our past has been characterised by wanting to be kind of hiding the background and just what we're doing now the fact that we're stepping forward is huge so this is very encouraging for me yeah add on to what everyone was saying about more of the political work organising and again like these are conversations we can have if we want to take it in this direction but the idea of advocacy for our community and how do we organise that because I can remember certain instances where a theatre might do something that was not favourable to our community and there were frantic emails sent all over to people we thought about and were all signing petitions and it was you know there was action but it wasn't in the most organised effective way and so I'm interested in having a conversation about if there are more organised ways to advocate for our community or to have that kind of more unified voice when something comes up that we need to address I also want to say that I feel personally that it's important that we don't put the burden of representation on this in the sense of trying to force everyone to agree on everything because we know that's not going to happen what this can be though is a place where we come together we identify things that we are passionate about and other folks who are passionate about working on that too and then those folks have a platform and an infrastructure and a way to connect and move something forward together so that you know if there's an issue like the census that I really want to work on maybe not all of these folks want to work on it but maybe half of us do and we can and whatever this gathering place is can be a place that those initiatives launch from having to say we all have to agree on everything so I think that as we refine what is this structure and process look like we want to make it flexible enough that it can continue to be inclusive and push our own comfort zones and make things possible that just simply wouldn't be possible if we weren't organized other questions in the house comments other questions are we creating a wasn't there a website idea at some point yes there's going to be a website is there did I see a hand go up no okay I think that's a good place to end because after this so we're taking a break after this we have a session on representation or increasing representation of plays in academia yes so that's so you'll see we've talked about mentorship we're talking about I can't pronounce is it academy or academia or whatever and then tomorrow we're talking about coalition so those are like three aspects of the I guess outward aspects of our work and then the working sessions in the morning this morning and tomorrow morning are going to be sort of the inward elements of how we work together this morning we talked about our top priorities we're talking about them and tomorrow morning we're talking about what Andrea well first we're talking about naming defining and what we call ourselves and then moving into coalition building and structure and I have kind of a random question that I just want to throw at you show of hands is there anyone who is concerned about our national coalition political for those watching there are no hands going up I just want to ask that because I don't want to assume you know we I don't know what you think what would be concerned I have no idea to me does that mean we should be endorsing individual people races throughout the country in every community that we're in that might be too political but we're going to start saying hey here's the list of the council people and all the men we want to elect that might be too political maybe but that's why I don't understand I don't know what you're exactly asking we're talking about advocating for individual political races where you're saying this group can get super out there talking about advocacy well as an advocacy organization if we're advocating for ourselves there's a range of action that can be taken so I think one of the questions in my mind is what is that range of action for an advocacy organization and that's something that we can talk about more tomorrow because we are at time no mic for you something to think about and I want to invite you to come to the those of you who self-identify as Middle Eastern and North African theater artists to join us for the first session tomorrow where we talk about the nitty-gritty of naming our coalition and what kind of organization and decision-making structure we want to have and Andrea is going to be leading that with Tracy and Leila and Leila great thank you take a break for 30 minutes and we'll see you back here for 30 minutes thank you


Alright, so hello everyone, welcome to the CNCF end user launch where we explore how cloud-native technologies are adopted by end user organizations across different industries and sectors. The CNCF end user community is formed with more than 155 vendor neutral companies that use open source software to deliver their product. I'm Ricardo Russia, I'm a computing engineer at CERN. Today I have Andy Bergin as a guest speaker. So in these live streams we bring end user members to showcase how the organization navigates the cloud-native ecosystem to build and distribute their services and products. You can join us every 4th Thursday at 9am Pacific. This is an official live stream of the CNCF, so it's subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct, basically please be respectful of all of your fellow participants and presenters. If you have any questions for us during the stream we will be monitoring the chat, make sure you ask the questions in the live stream chat. So this week we have as I mentioned Andy Bergin, here he will talk us about platform evolution and 5 years of Kubernetes at skypatting and gaming. Before we dive into the questions, Andy do you want to briefly introduce yourself? I think I should. Hi everyone, thanks for joining the stream or watching the recording. Hi to you Ricardo, nice to be here today and thanks for inviting me on. So I am lead platform engineer within the infrastructure and platform engineering squad at skypatting and gaming. I have been at skypatting and gaming for over 7 years now. I recently started as a DevOps engineer in the vet tribe. Moved to work with Duke in the data tribe and for us 3.5 years I have been working with the Kubernetes team which has been great. Before that I did lots of things with digital marketing, many different hats, many different skills, from dev, from ops, from production management, finance and all sorts of things. But I am really enjoying being back in the tech now. Outside of my day job I run the local DevOps meetup in Leeds in West Yorkshire in the UK and I am also part of the organizing team around DevOps days London which is a conference that happens supposedly annually but obviously over the last year things have been somewhat difficult around that. We all know but we hope to be back next year so looking forward to that, looking forward to just going to conferences in general and including KubeCon in that list right there. Awesome, that sounds pretty exciting, lots of things. I agree for the conferences, it has been pretty good to have the first one physical after all this time with North American. But I guess we can dive into the questions. I would start, maybe you can tell us a bit more about the infrastructure setup at your company and specifically maybe you could explain a bit what are the specific technical hurdles that bookmakers have to face. Okay, let me see a little bit on that. So we are online bookmakers. We've actually been around for over 20 years. Initially the betting arm of Sky Television in the UK so we were to be the red button on the remote you would press that and be able to place a bet was the idea behind that but that was a very long time ago. And since then obviously things have evolved. SkyBet started its own technology stack over a decade ago and that's been growing steadily. We offer a range of product services around sports betting and also gaming as well so kind of poker and sports spots and also all sorts of other entertainment products like that. I think the main thing with our industry where it perhaps differences from a lot of others is really the nature of the traffic patterns we get and the technology stack we have to have to deal with that. Coupled with the regulatory stuff we have to do as well. So make sure that we are looking after our customers and we are compliant with the regulations. It gives us a number of problems which we have to use technology to solve. So particularly to do with load I think many people who work in retail will be familiar with the busy days of the Black Fridays etc. Well we tend to have at least one of those a week in our industry and we have to deal with unpredictable demands I think perhaps on the gaming side of things this is sweeping generalisation we know kind of the patterns and can plan around promotions for things like that but with sports betting we really are the the whim of what happens in the sports game so typically in the UK the soccer games kick off at 3pm on a Saturday afternoon and there's a large spiking activity of people placing bets up to that mark and then what would have happened several years ago is that would have dropped off immediately and we would have really been quite quiet until the end of the day when we were settling the results but now we've got in-play markets etc so we don't know quite the demand we're going to have on the services depending upon what events happen in the sports games so you know we've got a very spiky traffic pattern which is kind of unpredictable as well so we have to have systems which can deal with that sort of scale and to be able to obviously make sure that they're available for our customers so they're the kind of challenges we face and I suppose to answer your questions the traditional stack that we had in the pre-communities day it would have been very much vn-based running our data centres and building applications with capacity to deal with the load obviously things are all different now but that's kind of how things were when I started all those years ago with the business. That's super interesting actually I guess one of the questions or one of the points I'll save for later maybe it's also understanding how you manage these spikes and maybe over provisioning of resources or I'm actually interested if you're running on premises and probably cloud but maybe maybe we can start with your transition to Kubernetes you just mentioned virtual machines and can you tell us a bit about your transition to Kubernetes and Cloud Native and how did you get that going? Okay yeah good question so yeah I have set that one nicely haven't I so yeah we're about to start a Kubernetes journey and this is back in 2016 and it wasn't meant to be a Kubernetes journey it was going to be a journey of what could provide the next generation of hosting platform for the back tribe and you know what platform could we put together that really made it easier for developers I think as kind of like operations engineers I think we maybe approach problems from thinking about the problems it can solve for us but really this whole journey started on about how do we get our developers to go quick we're in a very fast change in market a lot of companies in competition with us how can we get products to customers how can we make it easy for our developers because unless codes in front of our users it is kind of worthless so how can we make that easy and how can we address some of the problems that we were having with the more kind of traditional infrastructure we had you know the quality gates the bottlenecks how can we enable those but still do it in a safe way so the objective really was around creating a platform which had as few human interactions between somebody pushing code to a repo and having an automated process to get that onto the service in front of people that that was the objective and to do that we we set about building out some some pox first of all obviously you've got a technology choice and back in 2016 kubernetes was still relatively new and wasn't as mature as some of the other container stacks that were around then so there was some technical evaluation done with that also at the time this was one of the first pieces of work which wanted to run in public cloud as well so although traditionally a lot of our stuff of running date centers we did have some stuff in cloud but we wanted to get more stuff in there so we did the initial kind of proof of concepts to check out the technologies settled on kubernetes i'm very pleased to say it was before my time in the team but i'm glad they chose it but after that it became a how do we build with the developers a platform that they need so we worked with the team that has one of these spiky workloads i referred to earlier so what we call our push team so they are the the updates in the typical sports website there are lots of events happening not just like football games but things that happen in football games or netball or whatever it is and these events can change the prices of markets that people can gamble on so there's there's literally thousands of updates a minute going through that all need to be reflected on the user's device what then which they're using to interact with us so we worked with the push team to build out an mvp first of all building it out using container linux as it was at the time on aws provisioning sort of the cloud storage and cloud load balances as we needed for that and what that allowed us to do was on the platform that for the for the updates that when there weren't many updates we could scale it right down and when it was busy we could scale it up and that platform was very successful and it went on to become the kubernetes platform which was fairly widely adopted around the business and here we are now five years later with a whole bunch of people around the business from different departments using it super interesting i think that the early adopters went through the same process as well of like deciding which orchestrator and container platform they should choose that's also very nice to hear um that maybe maybe you can dig also you just mentioned that you deploy on aws um maybe you can mention a bit about the stack do you use like managed kubernetes or is it like your own um deployment and yeah sure so obviously we're talking 2016 and i think gke was available back then but in the um in his very early days too um so we decided to do things the hard way as was the uh the way to do things back there so um we we didn't we do things completely the hard way we based i think mentioned before we use contained on the next so core os as the base for our solution and we provisioned that through a bunch of terraform which would provision in our kcc2 instances which ran core os we took a slightly um novel approach which has worked really well in the sense that we wanted the whole thing to be ephemeral so effectively when we reboot or create nodes in our cluster um the pixie boot into core os work with some um container linux technologies called matchbox and ignition to pull down pre-rendered configuration um all those uh for those nodes and then effectively boot from scratch they're going to the early user space where they um take these uh the matchbox and the ignition configuration and they uh they they apply that to the os and before it goes in the proper user space so it's kind of like a a pre-boot thing inside container linux so we use that to provision to set up the node with all this specific um settings and configurations mainly system defiles and then we boot properly and that's when the operations operating system spins up so that means that if we reboot a node we kind of start from scratch we do have some persistent storage on there which are volumes amounted off the file shares to store things like docker images because that can act as a cache we don't want to pull them all the containers every time we start them up um but other than that a lot of stuff is held in uh memory disk um so slightly different set up to some other um some other clusters but that's where it really when it means things like upgrades are uh a change of a version number in a repo and then we republish all the uh matchbox and ignition stuff through terraform and that means that when the node reboot they can pull in a newer image of core os so that that works really well um we run the control plane in high availability we run that across a couple of nodes the scd database is backed up very regularly and yes we have tested it to make sure we can restore it as well um like i said we use a lot of terraform to actually do the provisioning and based on um other system components we should expect like a monitoring stack based on prometheus uh we use some of the other services which were already running in the supply for developers around the business so we don't run our own login stack that goes into our elastic stack that's run by one of our other teams in the business so we kept that familiar um stack of tool language developers new um and now we don't just run aws we also run on-prem using um the same terraform and um ignition scripts although they are slightly customized for different provisioners for things like storage and for and for the virtual machines as well so we run those on on vmware but it's essentially the same configuration regardless of which environment you're running in apart from the nuances of the uh of storage and load balancers and etc essentially we keep the same stuff and um yeah that's allowed us to keep parity between all the environments and we run about five clusters uh we don't run a lot clusters we run them independently as well we don't have like a a a cluster mesh over the top of that although we do run the service mesh it's the over we run that locally on each cluster all right super nice um like one question i have is just before we jump into the coronary details you have a bunch of small socks behind you so um yes on the wall behind me um this is a we we recently moved officers um during the pandemic so um we uh we we we we we i left the office in march 2019 and i've only been back once to take like stuff because we moved officers so we're now in a uh a building that's entirely owned by the company which is really nice and it's completely custom set up um but in the office we had um a few bits of uh kind of uh customized stuff we had around the place just to make the place feel like home so the kubernetes sign behind which says platform engineering they used to hang up all of our desks and basically when i went in to collect my stuff i stole it i don't think work know that so i probably shouldn't have said that out loud but the socks behind me are from conference swag many of those as will have been from a cube con or two um and at the time we were going through our socks audit um so there's kind of a pun there so we uh we we uh used to have those in the office with a graph of the number of socks we had and the number of socks we were going to tell the audience the auditors as kind of a joke so yeah that was our that was our official socks audit for the kubernetes platform beautiful cool i'll go back to the not be geeky mechanic that's pretty good all right so um i guess like digging a bit more into the kubernetes part um like you mentioned you started um and eventually you you had to manage i guess growth as things picked up so i guess i had two questions here one is just uh the growth of extended usage once things get popular the other one is kind of related to what you mentioned that the beginning for handling spikes do you have some sort of autoscaling and how do you manage that yeah sure so so yeah um yeah great the growth of the cluster so um yeah we we started off with as i said we we built it for one customer in one use um and that gained popularity really really quickly um and um with that come some challenges because you've not only got to scale tech you've got to scale people you've got to scale um the way you work as well um so um i think when we when we moved to on-prem there were some changes we had to make around the code base so we made some optimizations at that point to handle some of the some of the growing pains we'd seen in the first iteration of the cluster so we for example on an aws we could use a cluster autoscaler to deal with workloads so as things got busy we could pop up more ec2 instances to run more workloads on and obviously as it got quiet we could scale that down as well um so that was great on um on aws but on-prem that's not something we can do we have to kind of over provision for on-prem um the bits we had to swap out were i think we're on the dimension of just before we turned to like storage provisioners so if you want to slice of storage on the aws base clusters you get an ebs volume if you're um on-prem you get a slice of net app provision through uh through our in-house storage arrays load balancers you get a lb in aws if you're on-prem you get a slice of f5 configured and of course what we wanted to make sure we did with that was we kept the same developer experience so although the uh the the um provisioners for for cloud were fairly well you know understood we had to write some custom stuff to do that on-prem but we didn't want developers to be slowed down by having to configure f5s and to be requesting storage so we used the same um obviously we kept the um the persistent volume storage we just changed the provisioner there which um makes it sound really simple that was a lot of work went into that and the same with load balancer provisioning um obviously we didn't necessarily want our developers to be logging into f5s and configuring those when they could just declare the state of what they wanted there network connections to be and the cluster should do that for them and and obviously we we put that together as well so um yeah there were there were some of the um challenges we faced from keeping that parity as we changed environments um in terms of growing um we uh i think when we were working more closely with certain teams we hadn't necessarily anticipated the challenges ahead with that particularly multi-tenancy and i mean i think the initial year of the cluster was without our back because it wasn't there um i know our back was added shortly before i arrived to the cluster but that that presented some challenges because um how do we manage that for both environments we've got a solution based off vault and uh LDAP groups which allow us teams to authenticate and get access to um to the cluster so from that they're restricted to what namespace they can do we've done a lot of work with putting insane defaults and these privileged security when those namespaces are created so that if you get on the cluster you're kind of locked down to start with until you unlock the bits you need so you have to set all your network policies you need to set up quotas and stuff um and by that we've kind of managed the expectations of the customers getting on um we've got a support channel where people can raise support requests and tickets and ask questions and we can help them there uh but i think the the main thing we found was in terms of that growth was our users didn't always understand the line between what was the kubernetes thing and what was our kubernetes thing and there's an expectation from ourselves now that we um expected our development teams to learn how to build apps for kubernetes and also how to maintain and manage those um and off the back of that we've um you know we put in a lot of training uh we've trained over 400 developers on a couple of different courses on how to build and write apps for kubernetes so they can get that right uh but of course there's still you know um i've heard it said that kubernetes isn't a developer tool i'm not sure whether i agree with that definition but um i think that there's definitely a barrier to entry there but um whether or not it's it's massive or uh small i think largely depends on the developers we're working with as an example we've we've got developers who would gladly be given root access on everything and would uh love to uh insert records directly into the xcd database given the opportunity to do so in the control plane but obviously at the other end of the spectrum we've got people that just want to put a few lines of yaml together and they're not that interested in what that is because they've got you know quite rightly developers have got a whole lot of other stuff to deal with um you know the main knowledge of their actual problems they're trying to solve the code they're trying to write uh the business logic you know there's the you know i think the uh expectation that go away and learn kubernetes as a like an apathor i think is something that that doesn't really work um and i think we we got a little bit by that and hence we got to retrospect what we do um quite a bit of training around that to uh to bring um what to to help developers uh easily understand what they need to do on our clusters now how clusters work all right now that that's very interesting maybe maybe i have maybe another question uh about the management of the clusters but maybe building on the developer experience that you were talking about uh is there like a streamlined or recommended way for people to manage and deploy their applications you mentioned that they they have access to the clusters but but is there like a recommended way to manage lifecycle of their deployments or or the upgrades to do like there's all this talk about github to embrace this kind of thing or use some other tooling maybe can yeah it i think um given a time machine we would have put more developer tooling in place or encourage the teams that we work with initially to do that um i think um if we were starting again from scratch now we would certainly have some opinionating ways of building apps for kubernetes and what we're supported on there but um as with all ecosystems that evolve um we now have uh particularly the bet tribe have put together a standard way of building applications so after a couple of years of people going off and doing their own thing or being influenced by what other teams have done um there's now a pattern evolved of how things should be done and we have a team that are building a built sorry an application helm chart which allows developers to build applications based off a set of base images which are regularly updated they can take their applications there's pipelines built for those to deploy onto the clusters they're going to set standard dashboards and they get a a bunch of tooling and references to where they can find the logs etc all of that kind of thing which you know you need to run an application but that wasn't built by us that was built by another team and that's kind of becoming slowly the de facto way and we're seeing lots of our developers migrate to that way of doing stuff which is nice to see um I think as I say if we could start again we would um perhaps have done things a little differently and I think what one of the things we've done over the last three years and certainly for my day job as well is we relied heavily on this kind of um use of developer experience to kind of like solve a lot of the groan pains we had with the cluster we got a lot of users on there fairly quickly and I think we suffered with groan pains sort of internally of how we were working with the clusters so we've done that we've done a heck of a lot over the last three years to to like to like smooth that out starting we're just basically talking to more and more of our users about what they want from the cluster how they're going to use it understanding who was actually using our cluster was was quite a big uh a big undertaking we did trying to understand um which workloads belong to teams because they can move around as well so we basically tag all the workloads on the clusters now with metadata so there's labels which indicate who owns the stuff and that's allowed us to do a load of really cool stuff it's allowed us to shard the logging so rather than just having one logging pipeline well we can do that per tribe now there's a lot of work gone into that I mentioned we go out we speak to teams we we talk about requirements we take that feedback back we can do that and understand about the workloads which they're running but it's enabled other stuff away around others as well around things like best practice and standards so we put together a whole bunch of um ideas and call them best practice call them standards the principles of how you load and run an application let's go back and engage them on a containerized platform so we've got standards around build run deploy now um and we've got that that was built with input from everybody um who was using our cluster so we've got like a collective uh mindset on that it's not just our opinionated uh version of what looks good um so we've got that and then we built tooling around that to kind of like check on that as well and provide dashboards etc to indicate where things aren't following the rules and some possible solutions they could have to uh to fix that so um we've done a lot of work on that and it's um you know that's evolved further into things like um understanding costs and education on resources and things like that so that we can um you know run things efficiently as well yeah yeah I think like you you covered a lot of the challenges yeah it sounds sounds very good but one one thing like maybe you already mentioned but if you would say like the main problem you would have today while running your clusters well would you highlight something you mentioned a bunch of stuff that that is tricky to handle and yeah well I think from the technical side you you're always going to have um you know and this isn't a kubernetes problem this is just a you know a running computer systems problem running distributed systems uh you know you've got you're going to have face problems with problem workloads and with components of the system not behaving there's you know constantly keeping things up to date even if it's ever green a management of that um and then of course the probably the big one which um I think um whichever system you're running is going to be capacity particularly in an on-prem environment you know do you have enough storage you have enough network bandwidth um is your monitoring able to scale with your workloads um and then coming down to like right sizing workloads to to have the right requests and limits on them trying to support teams to to to get that right we find that particularly challenging because I don't think there's a great range of tooling out there to to to help with that we built some in-house tools we're building more we know this is a problem and we're you know in order to get our development teams to understand and to set their requests and limits correctly we need to help them to do that we can't just you know uh you know produce graphs and then point out inefficiencies or you know things get in infilled or uh cpu throttled etc that's not going to help so we we need to put better tooling around that so that that there's some of the day-to-day challenges but many of them are you know just keeping things up to date making sure we maintain enough time keeping things reliable sounds very good very good I'm just checking if there's question I don't see any so maybe maybe we can switch slightly the topic and less from the the technical or tooling part but maybe you can tell us how what's your experience as an end user in this community is there I don't know what's your feeling interaction with other end users or with the tools well you mentioned there you've been to a couple of cook cons from from the socks as well so I guess you've been involved yeah I mean I mean I think from the the end user community I mean being a member of that it really um that's a real boon when you're at the conference I think um the attending you come is um suddenly the team have really enjoyed I've not actually been a one yet I've got to be honest about that so but I am hoping to get there but I do like the physical conference I've been to the virtual ones but I love the whole hallway tracks etc but they're again I am a conference organizer so I I'm a little bit of opinionated on that but yeah I know cubecon is certainly something which the team have have been to and have come back full of ideas full of different approaches to do stuff I think I think the the main takeaway I take from the team when they've been and they come back is they say they had a plan of what they were going to see and obviously as you know cubecon is a massive conference with with many many tracks there of talks to see and I always come back waxing lyrically about the things they didn't expect and I think when I think almost they've said when they went to like the popular talks and they couldn't get in actually the ones they went to because it was near or or it looked interesting they're the ones where they picked up these little tidbits and these little interesting bits of knowledge which have come back and have been used I think like OPA was was was a great example of that no one had heard of it before can't remember if they went to I think it's Copenhagen I think that was the one they went to before they went to Barcelona and they came back from that like like this is brilliant we have to use OPA it's it's obviously like like the you know something we can we can use to help our teams on the cluster you know but without actually ending up with that talk they would have obviously we would have known about it eventually because obviously it's like a huge topic you know but I don't think we'd have had that kind of early visibility of it I think a lot of our early Istio adoption was based around talks and examples and demos and talking to other people at cubecon which is which we've seen so yeah I think even more than just the conference which of course is you know great and important I think I think supporting the CNCF is important because you know we rely heavily on the projects which it looks after so supporting that is super important to us so yeah I think the you know the the end user community is is really important and so is that's brilliant and yeah we're all hoping that normality will come back fingers crossed it looks like it's happening so you actually mentioned a lot of a lot of the tools that you you are relying on you mentioned of course Kubernetes you mentioned Prometheus you mentioned Helm OPA just now I'm kind of curious because you have a pretty large deployment and interestingly you have both on premises and public cloud deployments so it's multi cluster you mentioned that you don't do any kind of communication between the clusters which is also kind of common I think from what I hear are there any tools you also mentioned challenges in costs and things like this are there any tools or technologies that you're particularly interested on integrating in the near future or that you're looking forward to to look at yeah I mean we're I mean there are a couple I can mention so for example I mean we're we're heavy on our Prometheus adoption and we have had constant requests for long-term storage of metrics so Victoria metrics is something we're we're heavily looking into now obviously we want to manage that carefully because we're aware that long-term storage you know means different things to different users and we were particularly careful on how we manage our Prometheus instances as it is based on you know things like the amount of cardinality the metrics have and startup times etc so you know Victoria metrics is something where we rolled out and we're starting to roll out to our customers now so that there's long-term storage but we want to do that in a manageable way so that's that's one of the things we're doing we've used Gatekeeper which is a tool which allows you to basically report on OPA states we've used that for our kind of like standards and best practice dashboard we wrote an exporter which takes our data out in the format we want because we've also got a lot of metadata tagging in there which can identify workload ownership and stuff so then in the dashboards we can visualize that by ownership as well so that's been a very useful technology there are various updates to the networking stack going on updates to Istio at the minute the 122 upgrade is not without its challenges I don't think so we are working closely with our users and I suppose the great thing about already having those community those communication channels with our users in place has actually made that fairly straightforward we're able to identify workloads and go talk to them they're going to have problems when we do the upgrade so we're hoping to have everything in place in the next couple of weeks so we can get a 122 but I suppose the overall thing with our cluster is although we're looking at different we're always looking at new bits of technology and replacing existing functionality with newer bits I think the thing we we're more fans about is just stability and updates to things you know we've got a lot of operators that we run you know we want them to be stable we want the underlying Q&A system to be stable we want the monitoring stack to be stable we want all the things that send data from the cluster to all of the other services which ingest our stuff to be stable you know stability is a big thing and you know it's a dull answer and it's not a very exciting one because it's not like the you know the new shiny tech but we kind of like things just just working I think one of the things we're really pleased about with our cluster is the stability of it and we like to keep it like that and nobody likes getting paged and we want to keep it like that as best we can I think that's pretty fair and yeah I think the interesting bit here is also that you have a pretty like large deployment and it's interesting to see how you scale things like primitives or metrics and things like this and you're looking at these new new products to handle that I think for other end users it's extremely useful this feedback so I think I don't think we have any questions one one thing maybe I would put here is do you have something else that you would like to tell other end users or the community that we didn't cover here yeah I mean one thing I haven't really covered which has been really important to the team is how we use kubernetes not just to run workloads we also use it to provision infrastructure obviously I mentioned things like low balance of provisioning I mentioned like storage provisioning but they're kind of you know they're with the basic built-in primitives for for kubernetes you know you want a pv you will get some storage if you if you want a low balance that I mentioned that that's configured for you based on that work though the the automation really hasn't stopped there so I would give you some examples we were a team that did the f5 automation they're now trying to automate more things if you want more if you want to configure an f5 now for a virtual machine usage you can do that through through a code base where you where you commit yaml definitions for the load balances you require so even if they're outside of kubernetes we can use our provisioner inside of the cluster to actually configure load balances for things that aren't in the cluster so obviously the pull request approval on that but it means that rather than going into an f5 and configuring that for teams it's now all done as code that is obviously you know a massive benefit the same with dns entries we've done a lot of automation in the cluster and if you want a dns entry you can create a dns object in the cluster that's got an operator behind it which will provision you a dns entry in our dns provider through their api and we'll handle all that and we'll tear it down when you don't want it etc but equally we've got another repo where people can put those dns definitions and they'll just get created even if the dns record isn't used by something inside the cluster so we're starting to automate bits of infrastructure through kubernetes even if it isn't kubernetes so two more examples of that firewall automation has been something we've been you know every organization is one and you know the software defined networking is something that the organizations have won obviously we have you know over a decade's worth of network configuration in data centers in offices etc we're now starting to build tooling which will configure some of the firewalls through things that are provisioned from kubernetes so again we've got a repo where these rules can be defined and they can be pushed out to kubernetes and they can be configured through the tooling that's available within kubernetes another example that is a cert manager which other people are very familiar with we're using that with our certificate provisions now to manage certificates and we're hoping to offer that outside of the cluster so there's lots of bits of automation that we run inside kubernetes to to manage the resources we need or you know teams or developers need and indeed people in infrastructure but we can also offer that as the way to manage this stuff in an automated way outside of the cluster so that I think is something we you know we we're building on and building on and I think a lot of the automation we're going to be doing over the next 18 months for things for infrastructure are going to be powered by kubernetes as well even though they may never have a workload related to it on the cluster I think that that's a trend that has been going maybe in the last two co-cons maybe we see a lot of projects like cross plane that seem to be like starting to look at managing things that are not related to containers or containerization at all they are just relying on kubernetes as a platform I guess for for all this so it seems that you've been you've gone pretty far already yeah I mean of course we're also doing that for things in the cluster as well so you know operators you know things like you know we've got an in-house mysql provisioner which will you know with a small chunk of yaml creating your namespace a you know however many node or container based replicated mysql cluster you want and the amount of resource and tooling to back it up and restore it built into that operator so we've done a lot of work on that kind of thing we do offer some operators which we didn't write so obviously the Prometheus operator is one which we use a lot to manage the Prometheus instance it's on the cluster but we've got stuff for redis and a few other bits and pieces as well so that means that you know obviously like developers they don't have to go and you know ask for that provision it or you know or manage it themselves you know you want to mysql it's about six lines of yaml yep that's brilliant I actually we still have a couple of minutes so I just thought of something also because you are mentioning managing things that are not in the cluster you also mentioned that you have like multiple clusters multi-tenant just out of curiosity how do you handle this setting of external resources when you have multiple clusters are users like allocated to a certain cluster or do they see these resources everywhere or yeah so I think the LDAP groups are obviously shared in the organization so they're in the LDAP groups you're in and they obviously define which set permissions you get and then those are bound to access on the various certain amount of points within within vault so we use vault on a per environment basis we don't have anything we have a vault in every environment I think the environment configurations are set within the same vault instance so they're not shared but they may be managed on the same one all right very interesting I think I think this has been fascinating thanks so much for all the information I think we will wrap up here and I guess if if there are any follow-ups people can reach out either either finding you or or meeting you at I'm on the LinkedIn I'm fine I'm findable on there I'm Andy Bergen on Twitter if you want to tweet me please yeah or hopefully like share some drinks or a future cup of corn I will afford to that that will be great I like to say I organized I spent eight years organizing meetups then a few years organizing conferences and I haven't done any of that for it for getting off for two years now and I miss it now I'm looking forward to getting back to doing that then yeah talking to people about stuff and finding out what they're up to it's going to be great okay okay super cool so then thanks everyone for joining the this episode of the cloud native end user launch it was great to have Andy talking about sky betting and gaming and how they use a cornice and cloud native again I remind that the end user stories are happening every fourth Thursday of the month at 9am pacific don't also forget as we mentioned a couple of times already to join us at kubcon cloud native com you yeah it's may 17 to 20 and we'll have a lot of latest information from the cloud native community also if you would like to showcase your usage of cloud native tools as an end user then you're welcome to join the end user community with more details at cncf.io slash end user so thanks again everyone for joining us today see you next time and thanks a lot Andy for the great you're welcome thanks for having me it's been it's been good fun nice to share with the people what we've been up to so yeah it's been great thank you all right