Ευχαριστούμε και ευχαριστούμε για ένα άλλο κέντ 10 καρδιά. Αυτή τη στιγμή θα σας δημιουργήσω στο κέντ του Ζιούν Καρδίκου by Simon's Stump και να κάνω 10 καρδιά και ένα άλλο. Τώρα, αυτή η κέντ είναι ένα καλύτερο σχέδιο, υπάρχουν όσο πολλές πρόδειγες στην κέντ. Υπάρχει το κέντ για 29.99 εάν είσαι ένα μονοσυγραμμό, εάν είσαι και μπορείς να μεταξύσεις για 34.99. Αυτή η κέντ είναι ένα από τα πιο καλύτερα που έγινε. Θα δείτε ότι δείχνουν μεγάλο πρόδειγμα, πρώτα από όλα, δείχνουν μεγάλη μονοσυγραμμό από τα νέα πρόδειγη της σχέδισης, by Team Halls, για να προσπαθεί. Υπάρχει ένα σχέδιο από 3 Ζιούν Καρδίκου by Daniel Smith's Watercolors. Αυτές είναι οι πιο καλύτεροι εκεί. Είναι ένα πολύ καλύτερο επίπεδο για εσείς να προσπαθεί, να δείξεις αν θέλεις. Και να δείξεις τι χρώματα θέλεις να δείξεις. Και αγαπηθώ να δείξεις έναν νέο και χιλό, επειδή αυτά είναι οι πιο καλύτεροι χρώματα, που δηλαδή μπορείς να τα μπράξεις και να δηλαδήσεις πιο καλύτερο χρώμα στον Ραμβό. Τώρα στον κέντ θα δείξεις δυο αμυλόμπες σε πολλές χρώματα, για όλα τα συμβουλή. Και τι είναι το καλύτερο για αυτές οι χρώματα είναι ότι θα σας δίνουν έναν πολύ μεγάλο 8 x 6 στάμψε, που είναι called BEAUTIFUL FLOWERS με αυτά's BEAUTIFUL FLOWER ARRANGEMENT. Μπορείς να δείξεις πόσο μεγάλο είναι το καλύτερο για να κομμάσεις πιο καλύτερο ένα στάνταρτο χρώμα. Και θα δείξεις 3 χρώματα δυο αμυλόμπες, που είναι πραγματικά the same as the ones in the FLOWER ARRANGEMENT. Λοιπόν μπορείς να τα κομμάσεις και να τα μπράξεις στον κέντ, αν θέλεις να χρησιμοποιήσεις να τα χρησιμοποιήσεις. Of course, it is back to the sentiments, so I think you will have lots of fun with this stamp set, creating cards for pretty much any occasion. Now, another amazing product is one of the new 3D embossing folders by CISIX and this is designed by Tim Holtz. I'm super excited about this product since I don't have it at my stash and make sure to hold on to the package since there are instructions on how to put together the sandwich. Now you also get this watercolor paper that you can color and create a card out of it and you get a pack of beautiful papers. You will see these designs. These are 6x6 and they are from Moda Scrap. This is an Italian company and I want you to check them out. They have beautiful, really beautiful designs and you can see how wonderful these are with those loose flowers and watercolor backgrounds.  They are so beautiful on their own. Now you also get cardstock, a mirror gold one. You get also a white and 3 more colored ones. I usually cut them in half and score them to create my card bases. Instead, I'm going to save the mirror one so that I can use it on top of my panels and I'm going to just add one more cardstock from my stash for creating all the 10 card bases. So here we go. Let's start with the first card and for that I just had to stamp this beautiful flower arrangement. I stamped it with a black ink that is alcohol-friendly and now I'm coloring everything with my alcohol markers by Nouveau. If you love coloring, this is a beautiful stamp set for you since you will be able to practice your skills with many different mediums in this beautiful flower arrangement. And I'm going to color all the flowers in the same color. I'm going for a birthday card here that I wanted to keep quite elegant and I'm using the Corals set now from the Nouveau. If you want to see a product review in these markers there is one in my channel. And as you can see, I'm using the classic technique. I'm working on one petal at a time, starting with a lighter color, adding my shading with the middle and the darker one and then go all the way back to the lighter color. Before I stick this panel on my card base I decided to add a strip of black cardstock on one side just to add some more interest there. I'm keeping it simple so all I have to do is to just stick this panel on top of my card base and stamp the sentiment. I went with the Happy Birthday sentiment since I'm always in need of birthday cards and I stuck this panel on top of my card base with a foam tape at the back. Now I'm adding a little bit of clear Nouveau drops. This is actually morning queue and I'm adding little drops here and there just to make those flowers look more fresh. And this is going to complete the first card for today. And I went to the second card and again I'm going to use the same stamp. This is so gorgeous. I decided to create four different cards using it just to inspire you. This time I'm going to stamp with VersaMarking and you will notice that instead of stamping totally clear it's going to leave an impression there and that's just because I didn't clean my stamp well from the first card. But that's not a problem since I'm going to apply golden bossing powder on top and it's going to completely cover it up. In fact it was better because I could see that everything was stamped nicely. So here I'm applying my golden bossing powder I'm going to tap off the excess and heat set it with my heat gun. After melting the embossing powder I went ahead and stamped the sentiment and of course you could have done all the stamping and embossing in one step but I didn't know what to I wanted to do in the beginning. So I went with a sentiment that says you are a treasure to me. And again I'm going to keep pretty much all of the cards quite simple and easy to recreate and I am trying to use supplies that are included in the kit except from some exceptions such as this embossing powder for example but these are basic supplies that you probably all have in your stash. Now have a panel of this golden mirror cardstock which is the size of a standard card and it's going to provide a beautiful frame around my panel and finish it off. I'm adding some nuvo drops with gold nuvo and all you have to do to finish it off is to just stick this panel on top of a card base. And again for the third card I'm going to use the same stamp and give you one more idea. I definitely suggest you to check out this stamp set I know you will all love it and it is still available so if you don't want to get the kit maybe you want to check out just the stamp set. So here again I'm working with the stamp I have stamped everything with my VersaMarking I'm applying my white embossing powder and by the way this time I'm working on a watercolor paper So I'm going to melt the powder and then color everything with the samples that were included in the kit. The idea here is to keep everything quite monochromatic and I want to apply blue color at the bottom where the stamped image is. So I'm holding the paper up so that I can catch the light and see up to where this image goes. I'm also going to spray lightly at the bottom of the paper with water which is going to help my watercolor to spread evenly and easily and the idea here is to have first a very light color only at the bottom that fades out as I go towards the top and as I go to the top I'm making sure that my brush is clean so that I can make this watercolor look as if it's fading out. Now that I have my base color I can go ahead and add more details on the flowers and the leaves just to help them pop a little bit more. But I'm keeping everything quite simple so all I do is just steeping my brush directly on the color which is quite concentrated as you can see and just because I'm working on top of wet paper just by touching the brush there everything spreads nicely and creates the shadows that I like. I'm going to add shadows on all the petals of the flowers as well as on the leaves. This is going to help the design pop against the background but at the same time it's going to have this loose watercolor look that I was going for. To complete the look with the same watercolors I'm going to add some splashes. I'm going to stick this on a card base that I created from the cardstock in the kit and you can see how wonderful this color the watercolor matches the cardstock that was in the kit which is one thing that I love about card kits they make all the hard work for you. I also looked inside my stars to find an ink that I had that matched pretty much the colors of the cardstock and stamped my little banner that says love and hacks and of course since I'm obsessed with NuvoDrops I'm just adding drops here and there. Now for the next card I'm going to stamp the three flowers that are stand-down flowers from the stamp set. Again I wanted to let you know that these flowers match perfectly the flowers in the stamp that has the big flower arrangement so you can color them you can cut them and you can pop them on top of the big flower arrangement if you want to have some dimension but for this card I'm just going to color them with the watercolor sample from the kit and again here is my quick method of coloring flowers so first of all I have wet the surface this is watercolor paper I just sprayed all over the place now I'm applying a first base color very lightly and I don't care if I go outside the lines because I will be using my scissors and fuzzy cut all the flowers. Now again just like I did with the previous card I'm adding shading just by touching the brush directly on the concentrated color and then dabbing my brush wherever I need to add shadows mainly at the base of the petals and where the petals meet you can see that as I add lots of concentrated color just because the paper is wet this is going to spread out nicely and it's going to do the hard work for you quick and simple and really pretty result now everything is colored and I really love the outcome I'm going to use my scissors to fuzzy cut everything I'm cutting everything directly where the line is without leaving any white space around it now I want to create some green stems so I'm going to mix the blue with the yellow and create my very own shade of green and then I'm just going to spread it all over my watercolor paper and out of this watercolor paper I'm going to cut out thin strips of paper which are going to provide the stems for my card so I have my white card base here I have some foam tape at the back of all those flowers I'm going to decide where everything is going to go and stick everything down now I'm going to double up the foam tape on the third flower so it's going to stand even higher than the rest of the flowers and I'm going to add some tape adhesive at the back of these stems and stick those down off-camera I created my sentiment that says love and hugs this is just a white embossed on a piece of black cardstock and I have glued down with some foam tape so it does have dimension too just like the flowers finishing touches again by using my white nuvo and my card is finished now I'm going to show you a different idea on how you can use this panel rather than just coloring and stick it on top of a card base you can see that you can separate easily those two areas or the sentiments and the flowers and I found out that you can either draw a line and cut it out with your scissors but this die works great as well so I'm using this die on this beautiful patterned paper I'm going to run it through my big shot and you can see here that I end up having a panel that converts up completely the area with the design so I use the same die to cut out a little piece from the gold mirrored cardstock that was included in the kit and now I'm going to put everything together so this gold cardstock is going to provide a little touch of shine separating the two areas I'm going to add some foam tape at the back and I'm going to stick everything together making sure that as I apply the panel on top it's going to cover up completely the design underneath and then I'm going to use my scissors and cut off the excess now all you need to do is to stick this panel on top of a standard card base I'm adding some finishing touches with my novel drops you know I'm obsessed with those and of course you can color in the words flowers grow just make sure that you use colors that match nicely the colors of the patterned paper now for the next card I'm going to use this beautiful patterned paper I believe that this can stand as it is on top of a card to create a beautiful card just stamp a sentiment but I'm going to show you a great tip on how to turn it into a dimensional card since you know that I love dimension and just can't keep anything completely flat so the height on this panel is five and a half so that's going to completely cover up the card front but the width is four instead of four and a quarter so you can see I end up having lovely borders on the sides now don't throw away what's left from this patterned paper I'm going to cut out the buildings using my scissors making sure that I cut them out nicely and now I'm going to add some foam tape at the back and just stick those buildings at the front of my card this is going to provide an extra layer of buildings at the front with dimension and it's going to turn this flat patterned paper into the next level so here I have one more building foam tape at the back and stick it on top and I was actually able to cut out four different buildings from the leftovers so you will see here the result nice dimensional and for the sky I'm going to cut out the extra hot air balloon and I'm going to stick it with foam tape on top of my sky covering up the other same colored balloon so you can see all the dimension for my sentiment I'm going with a tiny little thing that was included in the stamp set and I made sure that I attached it on my block on a curve so it follows the curve of the balloon and my card is complete and now let's move on playing with the patterned papers and show you even more ideas on how you can use those to turn them into cards now I have cut out a panel from this beautiful patterned paper and I'm going to stamp a sentiment combining sentiments from the stamp set that we'll end up saying wishing you happy birthday of course you can stamp your sentiment with black at the center you can also use embossing powders to add some shine I decided to look through my stash and try to find inkpads that match the colors of the flowers so I'm stamping every part of the sentiment in a different color just for the fun of it now you can call this card done or you can add an extra element of dimension like I'm doing here so instead of throwing away this beautiful flower from a scrap piece that I have from a previous card I'm just using my scissors to fuzzy cut it and I'm cutting out the flower along with the leaves making my life easy I'm adding a little white border all around it and now I can use my foam tape and stick it directly on top of my card and my card is finished now let's make a fun shaker card using patterned papers so I have cut out this beautiful patterned paper with the stripes to be four and a quarter by five and a half and I'm cutting out a circle at the center and as you can see I'm not leaving anything to go to waste I'm going to cut out this beautiful flower composition which was actually printed at the front page of the patterned paper as I cut it out again I leave a white border and the colors on this flower composition of course much perfect the colors on my striped paper so I'm going to put those two papers together and I have a lovely element at the bottom of my little window so now at the back I'm adding tape adhesive I'm going to stick on top a piece of acetate and I'm also going to use my foam tape to go all around that window to create a little pocket for my confetti to nest inside now for my shaker elements I'm going to mix three different products by Nouveau and these are the hot pink stars the diamond and the purple organs these are really awesome products you can get them in packs of three and you can also get them separately I really love those jars and here I actually mixed up three different jars just because I wanted to introduce colors that are actually on the patterned paper and I really like my handy tool because it allows me to add as much of that mix as I like without making a mess on my table I covered up the window with my card base and my card is finished all that's left to do is to add the sentiment for the sentiment I just created a little banner there and stamped the sentiment love and hugs from the same stamp set and my card is finished now I'm going to play with the resist spray I have placed my patterned paper inside the box and I'm slightly misting from one side to the patterned paper Make sure that you do this inside the box something to contain the spray this is glue and it's going to stick all over the place so I'm going to apply my gold embossing powder now and you will see that it's going to stick wherever that spray went I like the fact that with embossing powders this is the only product that you can end up with such an organic and random background and here is the finished result after melting all the powder and of course there are many more techniques that you can use with your Distress Resist Spray now have some of these mirrored cardstock left over so I'm going to matte my panel with that which means that I am going to trim the edges making sure that I end up with a lovely thin border all around it I'm adding tape adhesive at the back and I'm going to stick this panel on top of my card base and I already love the look of it just as it is but I'm going to embellish it even more and I'm going to turn this card into a next level by adding of course some dimension so from this pattern paper I'm going to fuzzy cut three of the balloons and then with foam squares at the back I'm going to stick them on my panel and if you follow my videos my one kit 10 cards features you probably know by now that I always try to make the most out of the products in the kit making sure that I use up pretty much all the pattern papers so here is another little thank you card ready to go and now let's play with the embossing folder included in the kit I'm going to make the correct sandwich so make sure that you have your standard platform then place the embossing powder and on top just one cutting plate so here I'm not using two cutting plates only one at the top and you get a beautiful really embossed deep embossed result and I repeated the same process on a white card stock and look at all that detail that you get so here I'm going to use the black embossed one and this time I'm going to apply some Nouveau mousse I am working the mousse with my blending tool and I'm very lightly going over the design making sure that I touch only the raised areas and you can see how quickly and easily you get this gorgeous effect of course you can go ahead and add some more color at the edges to frame your background nicely it looks really gorgeous and I consider this my plus one to 10 cards today so I'm not going to turn it into a card I just wanted to throw you the idea of how gorgeous it looks when you do this technique on top and all you need to do on the other card is just stick a die cut sentiment on top and you have a beautiful card there now for my finished card now I'm working with the white embossed one I have cut it on an angle I am embellishing it with just the thin strip of the golden cardstock this is going to be very clean and simple I have stamped my sentiment and white embossed it directly on my card base and the sentiment says love and hugs from the same stamp set so I'm putting everything together just adding some tape adhesive at the back and you can see that I add that very thin strip of golden paper were those two colors meet I'm using my scissors to make everything look nice and neat and I'm going to finish it off by adding some dots of white nouveau in different areas on this embossed paper you can leave it as it is of course but I always like to add little details like this they really bring everything together and they add an extra element and different texture on my card and finally here is a quick look on all the 10 cards that I made for today making the most out of the June card kit by Simon Says Stamp and don't forget to try out the extra background that I show you where you use the nouveau mousse over the embossed cardstock here are some close-up photos on all the cards that I made for today there is also a giveaway running on my blog so for a chance to win this card kit make sure to visit and leave me a comment there don't forget to hit the subscribe button if you haven't done so already so you won't miss another video Thank you all so much for watching and I'll see you all tomorrow with a really fun art journaling event wish you all a lovely day


Oh, and welcome to Discovery, Knight Foundation's weekly program on the arts. I'm Priya Sarkar, director of arts at Knight Foundation. And today we'll be discussing resilience and the importance of adapting during and beyond the pandemic. Thank you for joining us. As we all know, this year has been one of rapid change uncertainty and in many ways loss. And all of us have had to adapt to these changes to find a way through. But what does it mean to adapt? And how can we get better at doing it? Late last year, Knight Foundation supported trainings in Detroit and Philadelphia for arts organizations on adaptive capacity. Those trainings were designed and facilitated by EMCRs who has been doing trainings of this type since 2005 and we'll drop the link into the chat if viewers would like to check out EMCRs for more information on adaptive capacity. Our guest today is John Jarbo, founding artistic director of the Bearded Ladies Cabaret in Philadelphia who participated in that training and have really run with it.  John himself has many distinctions to her name for her own performing roles, including having been commissioned recently by the Guggenheim's works and process to create a piece that was produced during the pandemic. The Bearded Ladies promote art that is intellectual, accessible, entertaining and meaningful all at once. So what does that mean? Well, before I bring out John, let's take a look at a piece recorded by the Bearded Ladies during lockdown. Social distancing. Oh, yeah, of course, yes. Do you want some hand sanitizer? No thanks, I've washed, scrubbed, and I'm wearing gloves today. Nobody wants my hand sanitizer. I'm sorry. Well, what do you do, Cookie?  Smoke? Binge? Netflix? Everyone does that. So you just sing beautifully.  You know, one thing I admire about me, I admire about you, one thing you admire about me is that when you sing, you really sing out. You know, you really, really belt out at the tune. I mean, there are not many of us left. So who are you anymore? You know, me and Eliza. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Come on. Come on. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Oh, my God. Oh, my God. We heard you from the street. Well, of course, I was belting. Were you belting? Of course, were you belting? Of course, I was belting. We're always belting. We're always belting. We're always belting. We're basically belting. We're belting. We're belting. We're belting. Do you need any hint? No, I sanitize at home. Nobody wants mine. Well, we should do a number, shouldn't we? I think so. We just need an arpeggio. What's an arpeggio? Notes going up. I think I had one of those last night. Oh, yeah. Okay. One, two, three, four, five, six, seven, eight. It's like show business, like no business, I know. Everything about it is appealing. Everything that traffic will allow. Nowhere we get that heavy feeling, and you're stealing that extra bar. There's no people like show people they smile when they are low. Even the turkey that will not go far leaves you standing in the cold. So you wouldn't trade it for ours. Let's go, let's go, let's go. Fine, how are you doing? Good. Okay, thank you. That was Cookie Diorio, Martha Graham Cracker, and our guest, John Jarbo. John. Hello. Hello. Thank you so much for joining us today. Thanks for having me. It's wonderful to be here. Thanks for showing that very silly video. Thank you for letting us. Yeah, Christopher Ash is the filmmaker on that video, and Pax Whistler did the piano. Very silly little experiment we did. Wonderful. And that was, I believe you said that was based on a, actually a video that was done from the Judy Garland show. Is that right? Yeah, the Judy Garland show when Barbara Streisand was on the Judy Garland show, and then Ethel Merman stood up. And then we've done a file. It's Portion Trilogy, so we're trying to do three, but we've only done two so far. We've done one with Cher, Diana Ross, and Beth Midler. And it's really a pleasure. The whole purpose of that project is just a pleasure and giving some joy in the midst of all this craziness. That's great. Well, thank you for sharing that joy with us and our viewers today. So, so I'm really excited to have a conversation with you about how you and the bearded ladies have been adapting during this time and just kind of what that journey has been like. I wanted to just kind of start with a bit of a, a provocation from EMC arts is moniker somewhere becoming rain, which for me helps me think about a mindset that maybe is required to be able to adapt. One doesn't discover new lands without first consenting to lose sight of the shore for a very long time. And I feel kind of like that's the moment we're in right now. I wonder if you could just tell me a little bit, you know, the Beards have been on a journey. But if we were to rewind, say to about this time last year, how, how would you describe, you know, the bearded ladies and, and maybe how, how you thought of yourselves or your community, maybe how you thought of your mission or purpose. Yeah, I think exactly a year ago we had, we were going through a pretty big change. We started in my living room 10 years ago. And with very little resources. And we attached to kind of these larger companies that needed, needed content, wanted some queer energy and, and develop shows in my living room. But a year ago we had, we were doing late night snacks, which is like, which was a new cabaret festival that we did 30 performances over the month of September featuring over 75 artists and groups, all in that kind of cabaret nightlife, some from opera. But a bunch of, I would say, Queerdos and I, this is a very positive term for me, like, like arts misfits, people that work in the liminal spaces and between things. And we were creating a platform for those artists. And hosting those artists, both in terms of having them into our space, but also being there to host and welcome them. So that was actually a big, that was a kind of turning up of the volume on something that was happening in the Beards a lot, which was that we have a strong vision. Sometimes we like homemade goods, like we'll make a whole play for you with music. And it's our ensemble, it's people that we know. And then there's this other thing that was starting to develop where when we first started, we created the Beards and I created the Beards to be a space where I feel like I could be seen and heard. Because I wasn't finding spaces like that in traditional theater and doing mediocre Shakespeare musical theater. And I think we've realized since then that the Beards can be that space for other people. And so that late night snacks was kind of the first, well, not the first, but I think the largest manifestation of that happening. And what was so nice about that is that right after late night snacks, we went into this EMC arts training. So that's where we were. I think we were actually like in a shifting notion of our identity last year. Interesting. So then when you went through this, this training, the workshops and just kind of became, maybe introduced to or more familiar with this concept of adaptive capacity. And some tools that folks can use to become better at adaptation. I'm just curious, having gone through that, what does adaptive capacity mean to you? How would you describe what that, that is for the bearded ladies? I think it was really helpful about the EMC arts training and that process. I mean, the first thing is just having time to think, to think in a more meta way about, about how you're doing the work, like just having that time and also that time among colleagues. There were many other different arts organizations of different sizes. Having that was a gift, but also the fact that adaptive capacity, it's about us. It's a spectrum of like how structured and planned are you, how adaptive are you? Are you, and I think thinking of it as a spectrum, because the beards actually like went into that being maybe very adapted, maybe too adaptive sometimes, because we've been very responsive to, we're, we're cabarets of responsive art form.  You know, so I think what was so useful about us, and I think we also came into that, that, that session, thinking of ourselves as this like tiny little barnacle that would attach to larger whales in the city. Sometimes I even say not to be blue, but we've like sleep around a lot. We're like, Hey, can we use your house for a little bit? I know we slept over last night, but can I like make my breakfast here? So we, we've, we were like this little barnacle that uses the resources of these larger institutions in the symbiotic relationship and then detaches. And I think what some of the work with the MC arts was helping us realize is that we are also our own whale of sorts. And there's a whole community of artists that are also working with us in symbiotic relationships. And that we, we have a responsibility and values around making sure that we're the most robust, like thoughtful whale in the, in the, in the ocean. I'm really going with it. Go, go, go as far as you want. But that we can, and that doesn't mean that we stop doing the kind of barnacle work that we've done. It's just that we, we can do that for others as well. And that's our, that's really our community. That's the people that are feeding us and we're feeding them. So that was what was really interesting about that workshop. I'm not sure if that fully answers your question. It does. It does. I mean, I think, I think that's a really interesting and useful illustrative analogy for, for it is for me. And I'm curious as you think of yourselves now as a type of whale. What are there things now that you are thinking about differently, or maybe that you're doing differently since you sort of came to that way of thinking. Or that realization. Yeah. I mean, and this was part of that process with the MC arts was taking a little tiny risk basically. Like doing a little experiment.  And asked them about their relationship. To us as an organization. We weren't actually there. We had them facilitate it themselves. And that was another part of this risk is like, like, let's, let's put together this community of artists that, that have been functioning as barnacles in our lives. And see, like, how would they want to design. How would they want to design this relationship? So we're in the process. Now of really solidifying what our relationship to our stakeholders are, which I think in the light of COVID as well, becomes even more important is how do you take? How do you, how are you thoughtful? Like, who were you taking care of before and who was taking care of you and how can when things get extreme, do you not pull away or silo or not communicate? How can you continue that expression of care? Even when you're, when the world wants you to be in a scarcity model. So that was one of the ways that we're really adapting is, is, is putting language. And time and thought and community. Conversation into how we design. How that weird barnacle whale identity is.  So thank you. That, that is really helpful to, to think about. And what I take from what you just shared is intentionality, which is something that I know has, has been, I've heard or read and talked about as maybe a distinction between being reactive to changing things and adaptive where they're, not that folks or that you and the bearded ladies weren't being intentional before in certain ways, but, you know, having the time or taking care of, to think about this and, and sort of being open to thinking coming at it from maybe different angles, then allows you to be even more intentional. And I feel like that's also a theme that has been coming up throughout the pandemic. As you said, thinking about caring for others, others who are caring for us.  And so in that way, perhaps the ability to be more intentional is aligned well and well timed perhaps with, with the pandemic and what we're in right now. Yeah, I actually think for many of the things that we encountered in the pandemic, the beards are well suited to exist in that, in that environment.  You don't know if you're going to have to, but you, you can. And, and I felt like we were particularly in a, in a position because of this training and the way that we think and because we're a bunch of queer people and, and a lot of that, whatever the trauma that's involved in making that happen, instills you with survival tactics that allows you to navigate many different, many different situations. So, so I actually think that beards can be quite good in crisis. Yeah. And that training was super helpful and kind of prepping us for that, but I would say that it hasn't been a huge, a huge impact. Right, right. Well, and, and I would, you know, love to, to hear and for folks to hear more about that impact and sort of what you're doing with it now. And I wonder if maybe we could kind of take as a thread through, through the, the next, you know, part of the conversation, something that you, you know, you just mentioned trauma. And you also talked about listening to and sort of, and really giving over sort of de-centering, I think is what I heard, de-centering yourselves and listening to the other folks that you work with and serve, that you feel you're in service to community and listening to how they would design their relationships with you and, and their activities. And I feel like that has also been a theme, especially with regard to equity conversations, racial equity and other equity and, and a lot of discussion that's been taking place in the arts and elsewhere around how we can, and systemic racism and, and, and pursue equity. So even though you, you didn't say that word specifically, I feel like that's a theme that's coming through. And I wonder if you could just talk a little bit about maybe how, how, yeah, just how, how the Beards are thinking about that or working toward that or just how you view that view, that idea. Yeah. I appreciate that question. And I will say like the context, the context that we're currently in when we're recording this of like having a conversation of art and adaptability also feels really hard because we're on the streets, you know, like people are on the streets and it's a really grief filled time. And the pandemic has been full of that pandemic. And I think the, the heavier thing, which is like the, the uprising and making sure that we're in support of the movement. Noelle Diane Johnson, who's an amazing artist does warm weather, or warm weather Noel, a kind of healing practice in the city who also works as a curator with us has been asking the question, like what is worth it? What is worth doing right now? Which I think I'm trying to think about our current context as an invitation to be more thoughtful about what is worth it. And what we do and like when we post and what we post about and when we leave the house and why we're leaving the house, trying to use this all as an, an invitation to that. I think we were already in a conversation. We've been through a process with a brilliant, with a brilliant artist and kind of thought leader and consultant, Matthew Armstead from Philly. We were actually doing a kind of review last year of equity. In the Beards. And we're actually deepening that relationship right now. We've created a position called the, um, um, Buds beard on Buds person. Which is a silly name. And so Matthew's joining us on throughout the year, checking in with both the staff, the team that we have, and we have an amazing team. I'm, you're only seeing one person right now, but, uh, Brandy, Matthew Thompson, Sally Olive, uh, uh, our production management team. They're incredible. So it's a whole group of people. So Matthew's checking in with the team and with all of our artists throughout the process and has the ability to pause a process and stop it and say, let's have a conversation. This is not the comfortable time to have the conversation, but we're having it now. Um, to try to make sure that we have systems of, uh, harm reduction built into our work. Um, uh, that feels very, very important. Um, Right now. And I think also, uh, You know, you know, working with our BIPC artists that are like in our community and centering their work. Um, so we did a video with cookie D Oreo from, uh, for visit Philly. That was all dedicated to black artists in the city. And as I say, we sleep around a lot. Um, and we think that there's a lot of power in partnership. So we're in the midst right now of, uh, We also partnered with spiral Q, which is an amazing, uh, Puppet, uh, street theater company and girls rock Philly, which is run by the brilliant Sam rise. Um, on that video and we're partnering with them and other organizations right now to, uh, work on this truck idea that we'll, I'm sure we'll talk about at some point to have a live mobile performance unit that goes around the city and can be used for protests and performances. Uh, so it's really, uh, I think it's deepening the way that we're doing our art. Um, Uh, and at the same time, I think, uh, The Beards and our artistic community are constantly shifting gears and, and asking the question, you know, is it, is it worth it? What is worth it? Where are we putting our resources? So definitely in process and especially in process on that, um, Uh, today and in the past week. Yeah. Yeah. Thank you. Um, I, I think actually it's a great time to learn more about the truck and, and some of these other efforts. Um, I'm just going to mention, I want to, uh, let folks know you, we will, uh, be, um, Doing Q and A in just a couple of minutes. Um, so please drop your questions for John into the chat. If you're on zoom or into, uh, The, um, Facebook live and, uh, we will, We will get to as many as we can. Um, and before we do that, uh, John, I know you have some images available, uh, That you can share with us as you tell us a bit more about some of these Activities that you just mentioned. Could we go to that? Yeah. I'm going to share my screen. Get excited. Um, I try to close all my million tabs. Okay. Here. And I'm gonna press present. Look, a drag queen doing technology. It's incredible. Um, uh, so this was us in 2019. Um, this was a late night snack space. I was thinking about that EMC arts metaphor about like leaving the shore, uh, and going to new lands and, and, uh, We are traveling a lot in the city. And I think also the thing. About, uh, Being adaptive, I guess, is asking what question, Like what questions you have to ask. And when you're going to new lands, those are not new lands to everyone. Uh, So like what is the history of the location that you're going to. Um, which we're thinking a lot about. And with our late night snacks process, we build this in conversation with a community. Um, we build a cabaret space with a community It lasts for a month. And then ideally that space will transition to a local arts organization or have a future. That is designed by our collaborators in the community. Um, It's really cool. So we were quite happy in our live space and then, uh, COVID hit and we had to adapt, adapt, adapt. These are animals. If you don't know what animals are, It's pretty clear from the woman turning into a starfish. Um, Uh, so one way that we're adapting, and this is the way that many people are adapting right now is we are going digital. How we're doing that is we're doing a 12 hour. Um, marathon of cabaret from around the world. So we're taking that project that you saw in live space, which incorporated international and local artists. And we, we decided, well, one thing that we can do really well is resource and connect to our international community. So each hour is curated. Um, by a different artists from a different city. As an expression of love. For their cabaret community. So that is feeling really cool. And that's it's 12 hours. Of course, people's attention span is about 12 seconds right now. So why not give you 12 hours of content? Um, but it's very MTV nineties. And we're trying to think of a digital cabaret is a different food group. So I myself have done a lot of the performing in my home vibe stuff. Uh, and that this is, this is higher quality, a little bit more focused and made for video, um, specifically. Um, and, and it's free and accessible. I think that that's important. It's on Twitch on October 17th. That is me and a banana custom. Uh, then, uh, we are. Oh, I skipped ahead. We're also like sending boxes to people's house so that, because cabaret is kind of like a space and a mode of performance at the same time.  So that's our version of digital. Um, we also bought a truck. We did a lot of work fundraising at the beginning of COVID. For these ideas. Um, and we've been really grateful to have support, uh, from our community, uh, because people are so excited about this idea in particular. So we bought this truck. It's, uh, it's a 15 footer. And we are in the process right now of converting it into a mobile stage. Um, we're working with producer cat Ramirez on this. They've been really wonderful. And, uh, and in our conversations with cat and our curation team, we've adapted this to be even like we've, we've gone through so many iterations of this project. And at first we were just like, Oh, we want to perform cabaret on a truck.  And we could use this. For protests. We could use this to go to your block and give you that party that was connecting your community that you didn't have before, or to go to your block and to celebrate. Um, to celebrate the, the person that's delivering groceries to everyone on your block or the person that with that great garden. So it's really expanding our notion of what artists are. Um, and so we're in the midst of, uh, a lot of the work that we're doing right now, like as I speak, we're painting and building this truck. This is a design by Joey Tony. It will be a bearded truck. Um, and we're loaning it out to different organizations. We're partnering with lots of people. Uh, so that once this truck gets going, uh, in the late fall and spring, uh, It can be a resource to the larger community. So really excited about this idea and also how it plugs into the projects that are worth it right now. That's the end of my presentation. That's great. Thank you for, for sharing all of that, John. And I, I love the fall season on the truck. Um, it feels very, very right. She's high. Um, so we have, we have a question, um, uh, from David. How has the transition to more digital content recalibrated your under understanding of boundaries and how to challenge them? That is such a great question. I'm in a box. I hate being in a box. Um, uh, I mean, there is something like delicious about seeing people's living spaces. Uh, the beard think a lot about boundaries because we're often in spaces where there's a lot of touch and intimacy and direct contact. And so we have all these tools like clutch your pearls. If you don't want to be touched, we have the audience clutch their pearls and that's like a nice way to build consent into things. Um, there's a different kind of consent that, uh, is present online. Um, that we are playing within our hosting. Like we're adapting the clutch or pearls. We're like trying to try and create versions of applause. Um, uh, and wondering like how far we can get you to do something in your own space to like, can we get you to open the window and scream out of it? Um, can we get you to dance? Can we get you to sing? Um, uh, I also think, um, I mean, I guess the boundary of, of who we can connect to and spend time with, uh, we, we felt really limited in our ability to present international artists. Um, and because we've traveled, uh, myself and Sally, all of our associated artists have traveled around the world, uh, making these connections to queer dose. Uh, and so that feels like the way that we're able to access those communities. Um, and be in dialogue with those for our audience in Philly in a way that we weren't before feels kind of amazing. Um, but yeah, it's, it's a great, it's a great question. Um, I, I made a whole video about cannibalism with Rose, which felt, uh, it's called Rose, um, for the works in process. Um, we'll drop the link into the chat. I think we have, we can do that. Awesome. Yeah. Uh, and that was about my gender queerness and the fact that I was told by my aunt that I ate my twin in the womb. And that's why I am the way I am, uh, lovingly told that. And so that felt like it was getting deeper, uh, more deeply. Um, and so I think that was a great question for me. About my, um, my life internally, I guess, crossing some boundaries internally for me. Um, yeah, that's a good question. I think we're thinking about that all the time. I agree. That is a great question. Thank you, David. Um, you know, I'm wondering as, as you're talking about these things and I'm just thinking about the, um, The difference for live performers. physical space where you can feel the energy. And it's not that you can't sense energy through a screen, but it's so different. And I just, I'm curious how that's been for you and for the Beards. Yeah, that's really, that's really, really hard. I did a whole lecture on what's critical about glitter recently for Temple Contemporary. And I sent them all these like little packages of drag to wear at different points in the presentation, which was really lovely. I feel like the more that we can cross the live space, the better. I think that that's why the Beards are doing the truck and the, are trying to do the truck and the digital portion because it feels like it feels like we're not going to get that kind of responsiveness that we need in the visual world. But we are really trying to maximize our use of the chat function as like a new place for the audience. It's really exciting on playing with Zoom about getting people to see each other, seeing each other, which feels like a central part of our work. Like we're less sometimes invested in actually the content of the performance in what the content does to the audience that's in the room. So a lot of what we're working on right now is trying to navigate making that happen in these sort of weird Brady Bunch worlds that we've been forced into. Yeah. Yeah. Yeah. Have there been any kind of, you know, interesting surprises that you've kind of come across as you've been trying to do that? Maybe anything that, I know, works in a way that you hadn't expected, something, anything that you feel like, huh, you know, maybe that's, I don't know if there's something that maybe it seems like it is, you'd want to continue with even when we can get back to, you know, experiencing Cabaret in person. I think that that is key about adapting, right? Is that we're not trying to invest in things that are short term. We are experimenting and we will fail and succeed and let things go. But everything that we're experimenting with in terms of feasts, in terms of having a digital international festival of Cabaret and in terms of the truck, which will both have a digital element and a physical element will always be living in both worlds. Those are both kind of long term solutions. We're not trying to think within like, what's going to work for the next month? Because as we know, everything's changing all the time. So we're actually thinking about like, what would be, what would give us pleasure and what would serve our communities longer, longer term, even if like tomorrow, things changed. And we didn't feel like we had to be in virtual space as much. So that feels key to adapting. Yeah. The drag, like giving people drag and encouraging them to dress up. And the fact that we're equally weighted in performance, like right now the performance space, you have the same box as I do, I think. So like the fact that we can pass the performance around feels really exciting. And I think also just taking time, which as I speak really quickly, it's like, slow your motor, take a breath, give people space. Because that is one thing that live performers can do that sometimes the digital content that we're inundated with on social media is not going to do. Like actually like tell people to turn off their cameras and lay down and coach them through something. Like give space and breath has been really successful for us.  And avoiding irony and things that distance. Because this is already a distancing form. That's really interesting. Yeah. Yeah. Thank you. So you mentioned about the box and how we are sharing kind of, I guess it's in a way, it's a bit, there is equity in certain ways in how these boxes online. And I think that kind of relates a bit to a question from Beverly, which is how is the creative process different when you can't be in the same room as your creative partners. So for example, rehearsal, I'm even thinking about, I was wondering that watching the piece that we shared at the beginning of the show. Oh, yeah. Yeah. I mean, that was, and that was really Christopher Ash emailing me and saying like, let's just do something. I want to experiment with, with this kind of filmmaking. And so I made my video and then Cookie responded to it. And then Martha responded to that. So that was really fun, because we can give each other prompts. And for Feast, I have all these hosts from different cities that are going to, you know, they'll be like, I don't, I have, they're going to like toss, toss a doll. And then, and then I'm going to have to see their video and be like, okay, I got to catch this doll, you know, I don't know why I have a Ken doll. But you can give each other prompts. So that feels really fun. We have a team that spread around not just Philadelphia, but our associate artistic director lives in Seattle. And so it's been wonderful for her to have more equal presence in rehearsal processes. So that's been really great. We do kind of meet on the porch sometime and social distance, talk through things, because I do think there's a kind of creativity that happens when you're in the flesh that you can't have otherwise. And we've managed, I would say to, we've managed to, to with testing and with very small teams have small rehearsals.  Christopher Ash is amazing and has been doing social distance filming. So Rose was filmed with like a camera far away from Chris, that was on a remote sometimes and it could move. So we've been lucky to benefit from his expertise and resources. Yeah, but yeah, just trying to keep an open heart to all of the new new ways that we're, we're discovering each other. Yeah, yeah. So, so I guess as, as we kind of wrap up the conversation, I'm just wondering, you've given us a lot of great ideas of things to think about or be open to any, any other kind of advice or food for thought that you might share as, as you all continue this, this journey of adaptation and resilience. Yeah, I think, I think work with one another as much as possible and communicate that like our truck idea is not a new idea by any means that's happening and has happened in many different iterations. I've been talking with Anthony Rothkastanzo, who has the New York Philharmonic truck, like we were talking with Clark Park and Curio Theater in Philly about our truck, the initial designs. I think that this is an opportunity to, to try to connect more deeply with who your community is and ask yourself what is your, what is your community? Who are you in community with? Who, who do you want to be in community with and deepen that? Yeah, I think we need to in times of scarcity try to be as abundant as possible and thought, which is a hard thing to say and is coming from a, you know, a place of privilege here. But that is what I'm trying to do with my team is try to work in abundance as much as possible. Great. Thank you, John, so much. Really, really great, great food for thought and really appreciate you kind of just letting us into the, the Beards, you know, world and journey with everything you've shared today. I want to remind folks that you can, you can actually share this with folks who you think might be interested and didn't get to tune in live. The link will be in the, or is in the chat. You can watch that on Knight's Facebook page or on our website. You can also check out what the Beards have coming up. I think, John, you mentioned upcoming performances and I think that is and I'm not sure if the October 17th one is part of Philly Fringe or if there's another one that's part of Philly Fringe. It's a little later than the traditional Fringe, but October 17th, it's a 12-bar marathon. You can pop in, pop out, it's on Twitch and if you want to buy a box, you can buy a box or it's free. So it'll be really, really beautiful, wonderful community of artists coming together and the theme of that is what kind of art feeds you. So yeah. It's very, very important and I think on the minds of a lot of folks tuning in today. So that link is in the chat. I want to just thank you again, John, for sharing, sharing with us, spending time with us, sharing, sharing your art and the work of the Beards and wish you well on your, all your upcoming performances and I can't wait until in person I can experience the truck because I am very excited about that. It's going to be, it's going to be really rad and thank you for having me and thank you for like supporting not only artistic projects but also artistic process and strategic thought. That is huge. That is huge and not everyone's doing that. So it feels really, it feels really like a blessing to be involved in that and to have the follow through to get to talk to you about it is really lovely. So thank you. Oh, well, we thank you. We are so pleased to just be able to participate in that. So for everyone at home, thank you for joining us. Please do check out these links and I hope you'll tune in next Friday for our next episode of Discovery. And until then, take care, wishing you safety, health, and joy in art. See you next time.


All right, welcome everybody. I'm so glad you're here. I'm going to be talking about adventures in data and leaning on Kubernetes storage to run hundreds of real-time analytic databases. As you'll see in a moment, I am a database person. Let's just jump into the intro parts. My name's Robert Hodges. I've been working on databases for over 40 years. And as a database person, we love storage. Kubernetes has made it very interesting because it's quite different from the storage the things we used to love about storage return things like battery-backed cache and firmware versions and RAID and stuff like that. There's a whole new set of resources that we have to think about. Been working on Kubernetes since 2018. My day job is I'm a CEO of a database startup called Altinity. And what I'm going to be talking about here is actually based on experience in that startup. We are a service provider for Klikas. Five years ago, we made a decision to move our processing, to make a bet, basically, that Kubernetes is going to be the wave of the future for management of data. That has turned out to be true in spades. And among other things, as part of that bet, we built an operator for, first of all, for Klikas, popular data warehouse service that I'll be talking about a little bit here. And then built a cloud on top of it. There's now, at this point, thousands of clusters running on this. We ourselves and our cloud are running about 240 prod clusters, data warehouse clusters, at this time. Some of them are pretty big, sort of 24 nodes, 40 terabytes of attached storage. These are not small. So that's the engineering. And in fact, as I go through here, I will be talking specifically about Klikas so that I know what I'm talking about. But just about everything we do here generalizes to other database systems. So let's just talk a little bit about analytic databases and introduce Klikas. So Klikas is what's called a real-time analytic database. And there's some kind of marketing words here that explain it, but what's simpler is to explain what it does. Imagine a security event, an incident management system. It could be a multi-tenant system, say run by some big company like Cisco. It could be reading tens of millions of events per second that are based on pulling data from various types of sources and looking for trouble. People breaking into systems, calls to servers, DNS requests to servers that are known malware sources, things like that. So very high rates of ingest. You also want to be able to react to things that happen quickly. If somebody does start making those DNS requests, you wanna know about it. Moreover, you want to be able to go through the whole history that led up to that event. So being able to, for example, be able to scan over billions or even trillions of rows, trillions of rows to find out what's going on and zero in on the cause. This is the kind of problem that databases like Klikas solve very well. So what's the architecture to do this? So this is a database that is optimized for reading large amounts of data very fast. So there's a number of features here. I'll just highlight some of the key things. It uses columnar storage. That's not really relevant when we think about the underlying disk or SSD that represents it, but that's a fundamental part of it. In modern analytics systems, we tend to have block storage. Using cloud block storage, for example, we'll talk about that quite a bit, or SSD. And also increasingly you see S3 or S3 compatible object storage used as backing stores for the tables, sometimes for the source data itself. We won't talk about that so much in this presentation. Databases like Klikas are complex distributed systems. So this just shows two nodes and with replication between them. So that'd be one shard of data, two replicas, but they could have 10 shards of data, three replicas each. These are quite large systems that communicate through networks. So that's the architecture. We're gonna be putting this onto Kubernetes and then figuring out how to make it access storage efficiently. So the first thing I'll talk about is mapping that database to Kubernetes. And since this is Kubernetes and it is a database, of course we're gonna start by writing an operator. So when we started this back in 2019, operators were not that common. I don't know when the model officially joined Kubernetes but it was not long before that. But basically an operator, if you haven't used one before, allows you to define new types of resources. We call this a custom resource definition and then supply the operator which is a special kind of controller which can basically carry out operations to make that resource definition become reality. And the thing that's really cool about them is that the resource definition allows you to take a complex system like Klikos, like Druid, like Cassandra and reduce it to maybe two feet of YAML which you can then kind of reason about change easily. And what the operator does is when it sees one of these YAML files come up, it goes out and looks at the corresponding resources that should be there in Kubernetes to make it reality. If they're not there, it creates them. If they are there but different, it changes them. That process is called reconciliation where you're looking at what the desired definition, desired state of the world is, the resources and you make it happen. So the first design question that we had which was relevant to storage was how to even represent a database like Klikos in Kubernetes resources. And I think how many people here are familiar with stateful sets? Everybody, yes. So stateful sets are the sort of standard abstract, low level abstraction in Kubernetes for dealing with data. They have a notion of a fixed name for each replica and then logic to make sure that you can connect that replica to storage and if the replica restarts, reconnect it, stuff like that. So the question is, just starting with a simple example like this where we have a database running in a container and some storage that we wanna talk to it, what's the right representation to actually result down on some worker node, having a process that's talking to some block storage? So what we ended up choosing was to use stateful sets. As I'll describe, there are some limitations to them but what you see here is a stateful set. It defines a pod that represents, in our case, a couple of containers that manage the data or manage the database and it has a persistent volume claim. In fact, it could have a number of them where, you know, because we often have things laid out with multiple volumes but a persistent volume talking to, excuse me, a persistent volume claim that then corresponds to a patch of storage represented by a PV. So that's the basic abstraction but then the question is, should you just lean on that? For example, I talked about replicas. Should we use stateful sets to do that? And it turns out the answer is no and this has to do not just with storage but with the way that databases work in general. And the thing is in clustered databases, even though, yes, these things should all be, you know, these things should be cattle, not pets. I mean, in the database industry, we've kind of resisted that. They in fact are more pet-like than you would like. Database replicas often have different resources. Like they could even have different, differing amounts of storage, different types of storage, different distributions of volumes. They could have different software versions. They could be in AZs. A very common thing in databases is to, particularly clusters is to have some nodes be dedicated to ingest the data and the other ones dedicated to reads. For this reason, you have the potential of a very asymmetric architectures. And as a result, we made a design decision, very early on, to go ahead and use stateful sets because they have a lot of good properties and they do map the pod to the storage in a fairly convenient way. But we just use one stateful set per database server. And this doesn't seem very exciting now, but you can believe we spent like literally weeks thinking about whether this is the right thing to do. And then once we did it, we stuck with it. So that's the basic, that's the basic map. And what that means is that the CRDs, and what we did further beyond that is that when we defined our CRDs, we actually imitated the good parts of stateful sets, which are the pod template that allows you to define the pods. And it's very flexible. You can do whatever you want with the pod definition. So when you look at our CRDs, you'll see we refer to pod templates. And then we also have, stateful sets have volume templates. We have a volume claim template, but it used very similar syntax, almost identical to a stateful set. What that means is you can configure the pods and the volumes virtually any way you want because we allow all of the relevant syntax. So this is the top level. I don't show the pod templates yet and the volume claim template, because I'm just showing in a couple slides from now, but we basically, in our CRD, the key point is we then imitated the stuff you know and love from stateful sets. So I'm gonna take a quick detour here because now we're gonna start to talk about storage in a little bit more detail. And if you were going to EBS, let me ask you a question for the crowd. If you were going to EBS, you're doing a rewrite performance test, and you were going to SSD, which would be faster? How many people would vote SSD? And let's say NVME SSD, and it's locally attached. How many votes for SSD? How about EBS block storage on Amazon? One brave soul. You won, okay, let's, so let's look at this. And this is relevant because it then leads us to some things like separation of storage and compute. So here's the basic architectures for a database that's gonna be talking to, in this case, mapped to a process running in an i3-4x large VM which is sort of an older Amazon VM that has attached NVME SSD, and then another VM with elastic block storage or EBS. And what's kind of interesting is when you test these guys, here's a, there's a popular test called Clickbench. It's used to do benchmarks against Clickhouse and also other databases. When I run this test and allow the storage to be cached, in other words, to run it multiple times so that the pages are pulled into the page cache, one really interesting result that pops up is that that EBS, the VM with EBS storage is universally faster, has higher performance than the NVME. This is a test where smaller is better, the red line is the EBS VM, and it's faster in every single case. And so that's just one simple example of how sometimes storage behavior in database is not quite what you think. And in this particular case, you know, we ask, you know, why is it faster? Well, it turns out since we're mostly reading out of memory, it's clock speed. The i3 is kind of an older Xeon. It's got a clock speed that's about two thirds of the speed of the M6i, and that accounts for probably most of the difference we're seeing there in these performance test results. Now you're thinking, I came here to hear something kind of obvious, but what's interesting is if we test it again, and even if we force direct IO, we actually get similar results, which is kind of surprising. This is another run of the same test, and actually we have the NVME direct, that's the blue line, the red line is EBS. You can see they're pretty much the same, not a lot of difference. We even threw object storage in there, S3 storage, that's also really fast. And part of this is because databases, like Clickhouse, are designed to dampen out the effects of storage. So that's important because it means we can use EBS, block storage for these database clusters. So that's a kind of interesting finding, something again was not obvious to us when we got into this, but then has major consequences going forward. Let me just dig into one of them. Separation of storage and compute. So in modern database systems, as much as possible, we want to break the bond between the storage and the compute that's applied to do things like calculate page bounce rates or time on page or sort of histograms, things like that. These are relatively compute intensive, and depending on how often you do them, you might wanna have powerful VMs or you might wanna have weak VMs because there's a huge cost difference. And what's cool about Kubernetes is if you use block storage in the cloud, you basically get separation of compute and storage for free. And so we lean pretty heavily on this. So this is, some people will say, oh, that's just vertical scaling of database nodes, but the fact is it works. And it means that you can choose, by choosing the node size, the most expensive part of many of these database systems becomes variable. So here's a simple example. What we do, and this is actually really important, is we in databases tend to like having a VM to themselves because they like, for example, they make assumptions that they own the entire page cache and that they have access to all RAM that they can see on the machine. When you're running in Kubernetes, Kubernetes does not hide the fact that if you're in a container, Kubernetes doesn't hide the fact that you may have 16 gigs of RAM. Everybody can see that. So somebody has an assumption that I'm gonna use 90% of that and you have a couple of them running, they will contend with each other. So we run them each on a VM and then of course we have the attached storage which we're gonna raise as they need more data. So how do we do that? Well, that just kind of, if you know how compute and storage are allocated, here's an example. We set up the Kubernetes clusters and first of all, for the VMs, we use a provisioner. In our production systems, we typically use node groups. Our cloud guys like them better. Personally, I use Carpenter because it's very flexible and you install it once and it works for all VM types. But the other thing that's important for storage is that of course you have the, in modern Kubernetes, particularly managed Kubernetes, you use a CSI driver, which then has a connector provisioner that can allocate EBS. What you actually get allocated, the properties of what gets allocated are controlled by the storage class. So the storage class will say things like, oh, it's GP3 storage on Amazon EBS and you can set the number of IOPS, you can set the throughput up to a gigabyte per second so on and so forth. So this is how storage gets allocated and if you set this up properly, it then becomes very easy within an operator like ours to use this well-known label, the instance type, and just say hey, give us m5.large. So this doesn't, again, it's not very profound but it has incredible effects for databases because it allows us, this could be a very large cluster. Like it could have storage that holds a trillion rows of data. When I'm developing, I'll run on this instance type and then what I'll do is I'll flip it to like an M6I 8xlarge when I wanna do performance testing. I just make that change and the operator will take care of basically reallocating the pod with a new definition and boom, in the background, old VM drops, new VM comes up and it now has extra compute horsepower applied to storage. So this is really, really nice. In Kubernetes, it just falls out of the box. I'm making this simple because normally I would do some other things. Like I have the node selector and you can see I've got a selector for zones to put it in a particular availability zones. I would also do things like add anti-affinity so that I'm not going to have two click house servers trying to be on the same VM at the same time. I would probably also, in a heterogeneous system, I would probably add taints and tolerations so that I could make sure I'm also not contending with other services that aren't click house. But this is all very, very easy to do and it's just a few inches of YAML. So, and then the volume claim templates which are the other part are equally easy. So we can just pick the storage type name. You can, if you're just playing around, you don't have to include a storage class but in this case I want to be sure I'm getting GP3 because this is good storage, very performant. I want it to be encrypted and then there's the size. That's again all I have to do to allocate storage. Moreover, let's see if I have the, I'll get to this in a second. I can now extend this storage while the system is running and I'll talk about that in just a minute. So, again this is kind of cool because these are things that are actually essential to making these large analytic systems work efficiently, particularly cost efficiently and they just fall out of running on Kubernetes. We do have users that are still doing this through Ansible on VMs and believe me, every time I work with one of those folks I get a syncing feeling because we're gonna have to help them get their Ansible scripts fixed so that they can deallocate VMs and then remount them on the new VM when it comes up, make sure everything gets properly installed, make sure all the mounts come up, it's a total pain and Kubernetes just makes it happen easily. So, with that, we can now map the servers, as I've said, precisely to the VMs and storage and as I said before, if I want to change it, I just change that value, resubmit the YAML and it happens automatically. So, there's another cool thing. I mean, it's just like all these little tricks and another really cool thing that drops out of this is because we're using the stateful set, do you guys know this trick about dialing the replicas to zero? Anybody do this? Okay, if you have a stateful set, it has this setting here, number of replicas and that's the number of pods that gets started, each of which is attached to storage. In our case, it's normally one and it was that way for a while and then we discovered this trick that if we wanted to just shut the pod off, like, hey, I've got storage, but I'm developing, I'm working nine to five and I'm going home now. Let's just turn this thing off. You can just dial it down to zero and at that point what Kubernetes will do is we'll just shut the pod down but it'll keep the storage alive. So, this is ideal. So, we use this to implement what we call uptime schedules. These are similar to what Snowflake does if you've ever used Snowflake Data Warehouse. You can have a schedule to run your virtual data warehouse. You can also do this in Kubernetes very, very easily. That's important because actually what we're doing with this open source database and Kubernetes with the operator, we're actually helping people. Part of our goal here is to help people build proprietary or sort of custom equivalents of Snowflake that solve particular problems and Kubernetes makes this work very, very well. We also, I mean, you can just go patch the replica set and do that but to make this convenient we actually added something to the YAML that we consume in the CRD. So, we just say stop equals yes.  So, and that's the effect. You'll just run it and basically what'll happen, your stateful set is still there. Your persistent volume claim is there but the pods, they just vaporize until you turn them back on again. So, but we're not done. There's still more tricks and we've got 12 minutes. So, that was, we were feeling pretty good about this but actually GP3 when it came along, how many people are familiar with the difference between, how many people use EBS? So elastic block storage, okay, about half of you. What are you, the rest of you on Google or Azure? Okay, I see somebody nodding when I said Azure. Yeah, so GP3 is one of these things where you wouldn't think this would be a big deal but people were probably lifting beers in, it's really good block storage. It's way better than the previous generation of elastic block storage, which is GP2, includes things like the ability to set the actual throughput that you can dial it up and down. So you don't have, it used to be that EBS had these kind of weird, sort of weird rules where for example, in order to get adequate throughput we would end up stacking EBS volumes. So, because for every volume you added to the VM it would give you more bandwidth. We don't have to do that anymore. The throughput is, you can just dial it up and down up to 1,000 megabytes per second. You can set the IOPS. And there's also allowing volume expansion. I believe that one existed in GP2 already. In fact, I'm pretty confident it did but the interesting thing is these can be set in the storage class now. So this is a storage class and this is the definition of the storage, the class of service of the storage that you're gonna get when you refer to this GP3 encrypted. There's just one problem. That is that these parameters, particularly those that list of parameters right there you cannot change them once they're set because CSI drivers that the CSI interface doesn't know about these things and there's no way to get them from inside Kubernetes out to Amazon where they can get changed and actually do you some good. And people do change this stuff. I mean we have people who'll allocate sort of the lowest level of storage performance and then they discover things are slow, they wanna change it. So we need to give them a way to do that. So, what we wanna do effectively is have a simple way of changing these parameters on demand and so what we did was we applied a kind of idiomatic pattern to use from Kubernetes, which is to build a controller for it. And so what we do when we make our persistent volume claims now is we add a special labels or special annotations I should say. And this is an annotation. You can see it has our name, it's called altinity.com on it and it says throughput 1000. This is a request on the persistent volume claim to go make this have higher throughput. Kubernetes doesn't really know what this does. Kubernetes annotations, you can do practically anything you want. So Kubernetes ignores this but our controller that we built does not. So we have a controller called the EBS params controller and what it does and this is kind of an expansion of the model I showed a couple slides back is we have the persistent volume claim. It points to a storage class. That storage class then is the EBS, the CSI driver sees that and it actually creates the volumes. That's, that gets you the initial volume. But what we have is the, we also have these annotations which are on the claim. The EBS params controller is watching and every time one of these claims changes, it will go out and it'll take the annotations it finds and it'll actually apply them to the EBS service directly. So that we can now make these changes to elastic block storage very simply from within Kubernetes. It gets better. So not only have we changed the values and this is totally simple. I mean it wasn't really, it's the, I think the controller is about this much go code. It's really pretty small. One other interesting thing it does is it reads the current properties back from the storage, or from the volume. And this is, turns out to be exceedingly useful because if you wanna make changes to EBS volumes, you can only do them, like expand it, for example. You can only do it once every six hours. So one of the things we do is collect the state of the current mod, you know, is it completed or not? When did it start? When did it end? And this allows us for management purposes, again, from within Kubernetes to keep track of what's going on out there in Cloudland and sort of manage it correctly. So I've used Amazon examples throughout here, but this controller we're actually gonna be using, we're now certifying Azure. We have users actually already on it, but we're certifying this controller to do the same thing as Azure. It turns out that Azure has similar need to control storage. So this is kind of a cool trick. One of the things that follows from it is that every now and then we have users who say, hey, you know, I've got 24 volumes, I need them changed fast. Well, you can now just by applying these labels, this is a kube-cuddle annotate command. We can apply that annotation, sort of lock, stock, and barrel to, for example, all of the persistent volume claims that have the label my-click-outs, and so we can apply this to all of them at once. So this is very useful if we need to do something quickly. Otherwise, we would have to go and sort of do this in a much more cumbersome way, so we can just get it applied instantly. Okay, so how are we making good time here? So there's a final trick, and this is kind of interesting. One of the great things, in addition to the fact that you can reattach block storage, one of the other cool things about it is that in Amazon, in Google, in Azure, you can also extend its size on a live system. And that's totally cool because the most common configuration problem that we have, a resource problem we have in data warehouses, is people are ingesting more data and they go up, they get 70%, if they disk allocated, 75, 80, 85, around the time it hits 90, then we're starting to get nervous because they could run out of storage.  So what you can do is, of course, you can fix this in, like if you're just using a stateful set, you can go ahead and fix this by changing the storage allocation in the volume claim template. And that is fine because it actually gets the job done, but there's a problem. And in stateful sets, when you do it that way, what it's going to do is it's going to alter your pod definition and it's gonna cause the pod to restart. And that's bad because why is it bad? Well, it could be really slow. Data warehouses, when they open up, in fact, databases in general, tend to have to do things like go have a look at all the files that they're processing, they may need to rebuild caches, things like that. That's slow, you really don't want to restart a database. So one of the things that we recently added back in May is we've actually taken the responsibility, we give users the option of taking the responsibility for managing storage away from the stateful set and we just do it for them. And so what happens is this option right here says, hey, the operator is gonna manage the storage for you, we remove the template for it from the stateful set that we generate and we do it in the background and as a result, these operations can be taken without forcing the pods to restart. So that's the tricks that we've learned so far. I think there are others, but these were the main ones.  I think we're right on time here. So learnings, Kubernetes is great for running databases. I want to emphasize that and operators are probably the biggest single reason for the greatness. But the Kubernetes, I think another sort of secret power or not so secret power of Kubernetes is this, it does work portably across many environments. We operate in multiple clouds. Our users want the ability to run these clusters, everything from mini-cube on a laptop all the way up to clusters with dozens or even hundreds of nodes in the cloud. Kubernetes does that. And so the things that we sort of got out of this experience are first of all, build on the existing Kubernetes resources wherever possible. We went with stateful sets. I think in the end, we're sort of taking things out of stateful sets and we're at a point where you might say, hey, maybe we should have just had a different, you know, just implemented our own controller, our own specific resource for this. Other operators have taken that, but we went with what was there and that had the advantage of making it pretty flexible and easy to understand. Performance is important. There is no cost that we know of to using, to running things in Kubernetes as far as performance is concerned. This was something we were, I should have mentioned this earlier, we had concerns that somehow Kubernetes, you know, maybe because it's using overlay networks or something like that, that it would affect performance, it does not. And certainly it has no impact on a process talking to storage. Because, and in the end, that's all that Kubernetes is managing. It doesn't insert anything in the way. But you definitely want to test performance carefully because how performance behaves with a particular application like a database may be very dependent on the use case and the way you're accessing it. This Kubernetes plus cloud block storage, that gives you separation of computing storage. That is such an important property for large systems. The ability to scale the compute, which is often the most costly part of these systems, to scale it up and down efficiently and using an operator to do so without taking apps out by using rolling upgrades. That's a really wonderful feature. And actually it's been one of the things that's allowed us to build a business on this. And then there's just storage, of course, is complex. There are these extra parameters, there are these tricks you have to, you know, things that you want to avoid like restarts. Fortunately, there's idiomatic ways to get around these within Kubernetes. It's a very flexible model. I think this, the controller pattern, that idiom of using controllers with, you know, the look for special labels is a very powerful one and can enable you to sort of hack around, excuse me, to engineer around limitations in APIs. So where are we going next? I mean, the current stuff we've got works pretty well. There's two things we're looking at. First is object storage. I haven't talked about it much in this talk. And it actually isn't really something that Kubernetes has that much to say about because object storage like S3 is just another application service. Runs on the same network as your applications are using to communicate. But it does have some interesting storage implications because object storage does require, it does require caches to use it efficiently. You don't want to, typically, you do not want to read everything from object storage.  So we will be coming back to NVME SSD. That will be more prominent in our architectures and this is why. The other thing is we'll probably be, we're back up as a big issue in all databases, in data warehouses. It's kind of an interesting problem because data warehouses often have such huge volumes of data that maybe you just don't bother to back them up. It just adds so much cost to your system. Maybe what you do is recreate stuff from source data if you have it. But we are definitely very interested in disk snapshots or volume snapshots which are available in Kubernetes. Other operators have applied them very successfully. So that may be another place, sort of a new place that we're going in terms of volume management. Some references, by the way, these slides are up on SCED. I want to give special thanks. You know, I mostly work with spreadsheets and SQL and stuff like that. Alexander Zaitsev is our CTO of LAD. Klamenko is the engineer that wrote the operator. These guys are great engineers and have figured out they and other people on the cloud team are the ones who really figured out these ideas. And that's it. Thank you very much. And we don't have another talk so if you want to ask questions, you can just step up to the mic or you can, and we can go outside after a while, but I'd be happy to stay around and take questions. Please go ahead. Yeah, my question is with the use of EBS or block storage, how do you deal with challenges around multi-zone support and dealing with failover? At least I heard EBS will be multi-zone in the future but at the moment it's not. Well, actually on Amazon, EBS is, it's interesting. So Google Block Storage actually does have the ability to replicate to other locations. We don't use it. It turns out that ClickHouse data warehouses are pretty good at replication themselves. So we just spread, we just use the pod definitions to force them to particular availability zones and just make sure they're spread out evenly. So we'll just, we can operate perfectly well over AZs. Thanks, if I could ask one more quickly. The data, the benchmarks you showed, S3 was almost close to EBS, which was surprising. So how do you explain that? Yeah, how do we explain that? Well, you know what, it's a really interesting result and there are two things actually to notice about that. I'm gonna flip back to it. One thing is S3 is pretty darn fast. This is something that people don't realize that, like for example, S3 and EBS actually compare pretty favorably and the network performance on Amazon is very good. So that's one of the reasons. Now one thing that's interesting, the yellow graph, this is something where you actually can see a difference. All the little queries, so this is a log scale. So the littler queries, you tend to see a bigger gap between the S3 line on the larger queries you don't. What you're seeing there is the latency to go fetch data off S3. There is a real cost because it's a longer hop to get to the S3 data and get back and you actually do see that in these test results. The other thing is that, yeah, so S3, where it gets expensive is that S3 doesn't have a buffer cache. I mean, you have to write application logic to pull the data down and cache it in order to use it efficiently. So there's definitely, there's no free lunch here, but these are relatively, queries on a relatively small amount of data. I think it's about 100 megs or something like that. And for that, S3 is very, very fast. Yeah, so a comment here, yeah, with S3 you get P90 outliers, yes you do, or P95 or P99, and yeah, anything that runs on a network. Moreover, S3 contends with other stuff your application is doing on the network. That's a key thing. EBS, I think this is certainly something I didn't understand well, but EBS is a SAM. And so there's a separate network. The EBS, you will, and Amazon in particular, is very good at allocating bandwidth. So you don't get contention with other things the application is doing. Do you wanna step up to the mic and just ask that? So we get it. Are these S3 numbers through a VPC endpoint coming from your VPC over Amazon's private back plan, or are you going? We didn't do anything special. I just set up a VPC and I'm going straight to Amazon. There's no endpoint in there. Yeah, S3 is, yeah, there's S3 has a bunch of, this is why I'm not doing this talk. So yeah, so this point, yeah, there's special costs around S3. I think one that people really underestimate, and this is another reason why Cache's API calls can just blow you out of the water. Normally people look at S3 and what's attractive about it is it's internally replicated. You only keep one copy compared to block storage where we replicate maybe two or three ways. So already that's a huge difference in cost. But then of course you add in the API calls and maybe it doesn't look so good anymore. Get another question? Yeah. Yes, I have a question. Does alternative cloud run on top of Kubernetes by using your operator? Yes. Can you repeat that question please? Does alternative cloud run on top of Kubernetes? Yes. Cloud, your alternative cloud runs on top of Kubernetes. Oh, does it run on top of Kubernetes? Oh, yes, I'm so sorry. Yes, completely. We do 100% of our processing is on Kubernetes. We don't do anything on raw VMs. And so if we want to use alternative cloud in AWS. Yes. And do we need to build our Kubernetes cluster or not? All right, you can if you, but only if you want to. So the question is do we have to force our users to build their own clouds? No, what we do is by default, if you use our cloud and there's other clouds do it the same way, we will build the Kubernetes in our own account and you just get an endpoint. And under the covers is Kubernetes, but you don't know that it's there. What you can do, and this is I think another place where Kubernetes is very powerful is because it's portable. We actually have a model where we can just, you can pop a container into your own Kubernetes and it will form a management connection, a secure management connection with our cloud management plane. At that point we can manage data warehouses in your Kubernetes. This is how we bring up people on Azure. So we don't formally certify Azure yet. We're getting there where we don't run it in our own account is what I should say, but we can allow people to run it in their accounts. Okay, thanks. You're welcome. Yeah, that's a great illustration of portability of Kubernetes. So now that we're here on the KubeCon, you mentioned all you have to do to go around the parameters that you specify in the storage class because you can not change the storage class first and they will ignore changes. So how do you think this can be modified to be able to do what you have done but without having like a... You know, what I would expect is, and I don't know the CSI interface very well. In fact, I don't know it at all. I normally work in the database, but I would expect it to be some sort of custom parameters argument that you could pass in so that you could in fact control, like have a way of taking those parameters and just pushing them through. That would, yeah. So, all right. I'll take one more question and then we'll go outside. Yes, so this is modern ClickHouse. I hope that's okay to ask. So one of the things we were evaluating and I wanted your input was one is, you know, ClickHouse being used standalone versus ClickHouse being used as a query engine with the data in Parquet files in S3. You know, these are fragmented partition data or chunking in S3. So your thoughts on one versus the other? Oh, you mean using it in Parquet versus using it in ClickHouse? One is used ClickHouse standalone for everything. The other is used ClickHouse as a query engine and data is in fragmented chunking to Parquet files in S3. Yeah, you know, that's a great topic. Like the difference, and this sort of extends into object storage, the difference between having storage, you know, using things inside a table versus external data. We're trying to merge those models. And in fact, one of the things we were even talking about yesterday was to have a Parquet basically part type so that ClickHouse tables would just store the data in Parquet. We can read Parquet very well. And what we're seeing is that in analytic systems, this is sort of a secular development across many systems, is that Parquet is the favored long-term sort of read-only storage type. And people put it out on S3 and then you wanna have your data warehouse be able to read it because it's fast and can do real-time query. But at the same time, you'd also like it to be accessible to, for example, your machine learning and AI. So there is ongoing work in ClickHouse. I think the most important thing in ClickHouse right now is just to make Parquet reads fast. And we have seen that things like predicate pushdowns, row groups are done better in Parquet than ClickHouse. No, actually, well, okay. So, see, Parquet is just a format. And so predicate pushdown is something that, which is, in other words, taking a condition and sort of applying it within the Parquet library. Yeah, it's just a matter of how well the database uses those features. And right now, there are some databases like DuckDB that are really quick on Parquet. Our first priority is catching up. Great, I'm gonna call it good because I think our AV guys gotta go home. But thank you so much, everybody. Feel free to come. Yeah, and thank you all for coming.