And you can see, there we go. And I'm not quite done with Chris and Deborah. I'm just about there, but not fully done with that one. So which order do you want to talk about them in? And now I've lost you. Is that me? I heard a little snippet of you. Maybe my phone would be more reliable. Do you mind if I call in that way? None at all. Now I hear you. I'll do that real quick. There we go. Hi. Cool. Technical difficulties here. OK. Here we go. So you said you're so completing the Chris and Deborah conversation. But I've got a bunch of stuff on them, so I'm happy to talk about it. I feel like there's just going to be a little bit more added to the conversation, but it was very, it was really geeky. Like, yeah, it was sort of a conversation between alpha geeks, it was interesting. Yeah, I know I think that not being especially familiar with the AI field, it felt like I wanted to have a to-do list of all of the articles that they mentioned to read and research papers before I could really re-listen to the conversation. Even though I understood the concepts behind it, and it was very fascinating, especially the idea that's around the feet of AI as a system, it wasn't being treated with regulation like software or even gave the example of cars would be. And so saying like, OK, we might not know how to regulate this. It's not going to look like how we regulate other things. But what is the structure for it? Because no regulation is not a solution. And the data is not just data. Data is still selective and tells the story. So I found that part really fascinating as a concept, even though I would love to read all the articles and research papers they mentioned in support of Fox. And also the thought that a car doesn't inherently capture your personal data. A car is just a car. In the worst of cases, if you have GPS, it tells you what addresses you stopped at. That's it. And maybe modern cars have open microphones and they're spying on us. But really, the degree of personal data that we're divulging inside of Facebook and Google is crazy. And I think Facebook is more than Google because Google is like, well, maybe you're Gmail, but they're not really looking at your messages. But I just learned recently that there's such a thing as a lawsuit for who searched for these terms. Sorry, not a lawsuit. It's a subpoena. So Google can be served by, I don't know who has the rights to do this, but law enforcement agencies can serve Google with a subpoena that says, give us the IDs of all the, I think the IDs, I'm not sure, of all the people who search for these particular terms. Wow. Not scary to me because I'm curious about something that you don't know about even if you're not. They wind up on the I list. Yeah. Yeah. So that was one of the kinds of implications that I think are really interesting here that they're talking about. And I actually went and found most but not all of the papers they mentioned and I didn't read them. These are really geeky papers. I mean, but I connected them into the brain. So I kind of have the nexus of things that they're talking about in a way. Yeah. And then you're on your phone right now, right? So you probably can't see text from my brain if I share screen. You're breaking up on me again. My computer talking. I don't know, can you hear me now? Yes. Now you're clear, but that last stretch was bumpy, bumpy. I didn't say too much. OK, oh, there you go. OK, so now you can watch it on your other device. Cool. Record progress. Good. OK, perfect. Cool. So are you seeing what I'm sharing now? Let me see. Can you hear me now? Yes. OK, I turned off my cellular. Maybe it's the Wi-Fi stronger. Well, that's probably true actually. Yeah. OK, I'm looking at your brain. OK, good.  But machine learning engineers often use the data that is most easily available, which I need to connect to that thought. It's really hard to recall a bad hammer, but you can just turn off an API. That was interesting. And then up here, they mentioned a paper called Critique and Contribute. That's this one. Then there's the 1993 paper called Greater or Lesson Statistics by the guy who was at Bell Labs. And they started with algorithmic bias and a bunch of things around that, which are really interesting as well. So I did not know about third party AI algorithm audits. So everything here is new. This is I knew about software audits kind of. I knew about algorithmic bias. But I didn't know that there were third party firms that were starting to do an audit. That was really interesting. Yeah, I thought that was so fascinating. I put that in. And then one thing I know, because I have some background in AI, I've never been a coder, but I used to be an analyst in the field.  So 5% of the time when somebody shows up with sniffles and a cough, but if they also have this other symptom, you might have tuberculosis. That those are rules. Neural networks are just trained on data sets. And they don't actually have rules inside. So when they make a recommendation, all they're doing is pattern matching. And they can't tell you why.  Well, if those cities are woodlining or if those cities included bias against minorities for lending, then guess what the algorithm is going to do, right? The pattern is going to replicate the bias that humans already exhibit every day in systemic bias. Basically, systemic racism and all those things just show up in the data. So that's interesting. And I think I'm going to pursue that when I story thread this. I'm going to talk about that some. Yeah, I thought that was very fascinating when Deborah said she said it's something around the around the lines of there's rather than considering which data would actually best support this decision, you're just pulling the data that's available or what has been the standard thus far. And so not thinking about could this data have bias? And that was such a strange eye when Chris said, yeah, like the minds years ago was just its data. It can't have bias like its fact and to think it's still in the hands of people. It's not even the facts are recorded or formed by people. So and also the people sometimes have bias. So the place I think I was going to start because they started with they started kind of with some of like it's not camera bias, but image like a lot of image processing algorithms don't perceive black faces well. And that's the problem. And that kind of bubbled up a couple of years ago. But the thing I was going to show people is that in the world of image processing, the default, the standard image that they moved that they moved around is this one. This is a Playboy Playmate image. Her name is Lena. Her name is Lena, actually. And her name is actually Lena Söderberg. This is a standard test image. And this was when people were when researchers, almost all men were doing image compression. That's the image they were working with just to show you like what a boys club this was kind of forever. And so there's clearly bias baked into lots of different places, lots of different corners of what's happening here. Yeah. And then I've got this connected to how can we keep AI from being a threat? Which is kind of the bigger the bigger piece of this. And then that could also bridge over to one of the so I've got the thought about what are the common threads across the different conversations across these different sessions. So here's how I've been storing the common themes across these sessions. Hold on a sec. There we go. So deep listening, coping with crises, algorithmic bias, healing from trauma. And what should we do right now to create a better future? These are the threads that I've got so far. Mm-hmm. Yeah, that feels that feels really that feels very true to a lot of the different conversations that we listen to. And it's I found that another another interesting pattern I saw throughout I was reading through some of my different notes from from talks. And it was kind of I think because of the the doom, the like the impending feeling of midnight, there was this conversation around like staying in motion and being in motion to not be stuck and to keep energy flowing, not just through us for healing and like things like dealing with trauma. But it was also this like if we're in motion, at least change will happen versus feeling stagnant or stuck or depressed or things like that. So that kind of motion of energy. And I don't know if you saw my Tracy Ryan's map, but it was the water currents through the ocean system, the hot and cold currents. And that idea not just of like his conversation of Qi and moving energy through the body and processing out of trauma, but also it was like the the way to find hope or or a glimpse of hope was was in processing, slowing down processing personally. It ties into the deep listening because it's all about kind of how do we slow down our systems to actually be aware of what we're doing and how we're acting and how we move forward. And that even feels relevant to the Chris and Deborah conversation around like how do we slow down and see what harm is the system doing? How do we regulate it? How do we move forward in a better, more thoughtful way? But that thoughtfulness and that idea of deep listening and dealing with trauma, it all is like looking inward and facing ourselves before we can help others, which it's so much easier to problem solve for other people. But I think, yeah, you're going to have even more difficulty with finding true solutions and listening to others if you aren't listening to yourself. I also just passed the part in the Chris and Deborah conversation where they talked about how these algorithms are always in motion as well. And so, if you're in an accident, we can figure out which model Pinto you were in and there's like hundreds of thousands of that model Pinto. But here it's like the algorithm is always updating, always changing and may even in fact be very custom to you because of your own variables and all that. And so figuring out what to fix and when you make a claim about something that broke, oh yeah, but we fixed that in Rev, two Revs back. So how do you find it? How do you fix it? Who are you going to sue for what part is really interesting? But then also just back to your theme of motion, I think there's like eight different kinds of motion that have come up in these different talks, right? Including kind of in the Buddhist session more about the like the Pasana, what happens when you stop all motion and when you try to slow your brain down and what shows up there, right? That was really interesting. And I think that ties in perfectly with the idea of deep listening because what you're doing in that you're intentionally pausing motion to sit still for a set number of hours, but by sub-doing or by halting one act like physical action or even speaking, you're, I feel like you're becoming more tuned to others. And so that brings up kind of all this internal motion, which is why we often run into high speed to avoid that internal motion. So yeah, I thought that that was really fascinating because that it was a pause, but it's actually just activating a different motion within us, which can truly lead towards some of that healing and fresh perspective and processing of things. But the one reason we avoid that is because, as you said, you're literally taking everything out of the closet and trying to just put back the good things, but it's like the closet falls in on you when you slow down that much, you know, everything is out. And, but yeah, I think that the interesting thing of this, especially that conversation left me with was that so much of the work is so often we're not doing the internal work before trying to help others. And that sort of thing. Yeah, absolutely. And that, and that kind of felt like some of the Roblox that Virginia was running into because people were just pushing their ideas so much, they couldn't pause and listen. But I think if you pause and listen, then you can kind of tell maybe the core of some of these ideas is actually the same and we're just going about them different ways. But it is complicated when you bring in people with all different opinions and thoughts. So yeah, what were some of the other types of motion that stood out to you? So what you just said reminded me of my old boss, Esther Dyson. So Vanity Fair did a piece on her once, a profile of her and they interviewed me for it and they quoted me as describing her kind of as a shark. And I didn't think like Shark Tank and that was before Shark Tank, I think even was a thing. Yeah. What I meant was Esther was in perpetual motion and really couldn't stop. She booked her days where she didn't drive. So she always would approach somebody to drive her from or to airports and she was always traveling. And so those were briefings. Like somebody would offer to pick like, okay, good. I'll take the ride out to DFW and then I'll get to pitch my startup. But her days were very, very tightly booked. And then she, you know, she'd booked time to write a piece and she had this incredible focus. But it felt like if she ever paused for introspection, she wouldn't like that. So she didn't make room for that, I think on purpose. There's a piece of that, that accompanies what you were just talking about. And I thank you for reminding me of the metaphor of pulling everything out of the closet and then putting back in only the things you want. That was, that was special. Yeah. And then there's also all this other, there's also this general motion that changes happening so fast that we're in an era where everything is moving so quickly. That, you know, do you need to run fast to keep up. And April just wrote a book where the first of her eight superpowers is called run slower. And she quotes, she quotes the Navy SEAL saying slow as smooth, smooth as fast. Right. And one of the, and in terms of the dangers to our democracy right now, one of the bright spots for me is what's called a deep canvassing. Have you heard of deep canvassing? Yes. Yes. You mentioned this earlier. Yeah. So Stacey Abrams and others are basically saying, Hey, don't just knock on the door and leave a pamphlet and like, like ask who you're voting for, slow down and have like a 45 minute conversation. And that's much more likely to change somebody's mind, which is great because it, because relationships and trust building to actually take time. And so there's this weird, there's this weird feeling I have that things are really, really, really urgent that we don't have a lot of time left on the midnight clock. Talk if you talk to people who are way deep down the climate rabbit hole. They will tell you that this thing is hard to turn and may not be turnable, et cetera. Yeah. And, and then I think, wow. Okay, good. That means we have to do something fast and a lot of it. But if it involves humans, chances are we have to do that slowly. Yeah. Right. If it involves putting iron filings in the ocean or sulfur particles or sodium particles in the atmosphere, fund it, put it up there and try not to destroy the earth. You can do that at the pace of engineering. But anything that involves humans and voting and whatnot is just going to have to slow down in the face of crisis. And that's going to be really, really hard. Yeah. That's going to be super hard. I know, especially when some of the population has such a sense of urgency around it. Absolutely. I know what the exacted. What it was in Italian, but have you heard the phrase? It means make haste. That reminded me. You were saying just a minute ago with that idea of like, how do we. Yeah. How do we keep pace and also be thoughtful? Exactly. I think so. Exactly. Yeah, I feel like that's that idea of. I think it, I think the, the pasana and the like sitting meditating and slowing down also. In that conversation, it was so fascinating how they were saying like so much of that actually like reminds us that we're not alone and shows our connectedness. When we slowed down and what, I mean, not just through like Tracy's vision of his mother, but even through healing with trauma and really working through things that brother fulfillment was talking about. But I feel like that idea of. When we're running at such a speed, we can feel really alone. Because. Because there isn't that time to pause. And you feel alone in your, in your rush or in your speed of what you're doing. So. It's interesting that actually in slowing down, even though we're in. A mid night crisis state. Is probably the best way to actually truly connect, which resonates with the deep canvassing and everything as well. Yeah, it's so interesting to figure out how, what does that look like on such a big scale, especially around the climate. Yeah. And when you're, when you're in motion quickly, when you're moving fast, when you're in motion for a while, when you're in motion, when you're moving fast because everything's blurry, you're hard to pin down, like all other things are hard, harder to do. So. Yeah. The thought of how each, well, some talks didn't directly discuss midnight as a topic. hopeful about it, but some, some did and thought of it more as like a new beginning, but I really resonated with the Justin Ida conversation around it because it was like this time zero like this. It could be the beginning, but it's also kind of feels like the end. It's just darkness and you're not sure when the light is going to come but you think it will come and that is a believing that the light that the light is coming feels abstract and like a willful exercise. It's so fascinating because then thinking about that as the first talk that we listen to, and then the Tracy and Aaron talk was kind of a practice of a willful exercise to be present and take note of those things so it was neat to see some of those overlaps and people who are like, their discussions were actually about very different topics.  The light right now. Exactly. As the light as the sun is setting on you. But only today, only today this is not the best metaphor. It makes me think also of the piece that I was story threading in about the new story like we're missing a new story. And one of the things people light is a hopeful story. Yeah, right. And part of the problem here is that we're missing. It's not a uniform story but, but a reasonable story that paints a future that we might actually be able to like survive into together and thrive in a few sounds a little bit better than, than we expect or then then realistically people are coming now. I'm, I'm, I'm perpetually interested in what are the stories people are offering that do engage that kind of hope. And in contrast, the different kinds of crazy stories that people are weaving that are destroying all of us like you know like that. There's a really good article. I don't remember if I mentioned it in one of our calls but Q and on a guy wrote a gamer game designer wrote a really nice piece that said, is Q and on an alternate reality game. And if you look at Q and on there's a mysterious host who keeps dropping messages, which have plots and things to do and then and then it's like, holy bean dip it looks just like me. Right. Yeah, except this is this is a life and death ARG that's actually affected a whole bunch of people and shifted the behavior dramatically. Right. Yeah, that's really really interesting because that's a story to it's a it's a set of nested stories that are busy, like causing chaos in the world. Yeah. I know. Yeah, the stories we tell are important. And that's what I think I think in the version e conversation I think I was looking for and we discussed this earlier but looking for like a new direction or a different approach or next step towards how do we bring people together over climate change and actually take action. And I think that she was her approach felt much like the make case slowly like she had, she had great hope that something that change could happen quickly once the story is agreed upon. And the, once the story is a wet is vocalized in a way that people can connect to grasp on to feel a part of, and in so sometimes it's just you want there to just be one quickly and it's like how do we find the right one before we're even approaching the public with it which is so, so alter to so much of our every idea you have you put online immediately mindset right. So that like slowing down and thoughtfulness, but also with a sense of urgency is just it feels and in pole with each other you know. I also felt like vision the. And I may have this wrong but it felt like her audience or her posse or her the circles she was busy thinking about these important issues with were all policy abstract level. They were, they were kind of in this policy dance this long term policy dance that wasn't yet resolving things that is like having a nice week this weekend cop 26 and Glasgow. But, but that that wasn't the, the native story that ordinary people are going to pick up and use and do right. She felt to me like she was operating a couple levels or layers removed from normal people's lives and stories. Yeah. Yeah, but it felt like it was a building block, like that was all of these people who are maybe running a lot of the organizations that are going to be involved in some of the major steps of climate change mitigation is is kind of where the story needs to start and then we can explain how everyday action is involved in that. But I don't know. Yeah, those bigger, the bigger concepts like that are just so fascinating because it's everyone is like trying to approach it in their own capacity and how they can and what makes me think of the Tatiana and Deanna conversation with their ideas of, well, we are living in one world, even though we're trying to build an aspire to a better world. But there is this push and pull where we can't totally step in or out of one and it's not just they mentioned cowardice but I don't think it's as much cowardice as like one system we're actually most of the world is functioning in. And it's like being an early adapter. So it's, there is going to be tension just how do you have a healthy tension where you are moving forward and not staying in a system that is not, is not progressive working. Yeah. Yeah. Yeah, I'm looking at, I'm looking at the quotes I wrote down from their session. We don't know as humans where to put our collective trust. Yeah, the idea of trust was really fascinating and trust almost has a currency that that we individually have the highest value of so like there's so to think about every so much of what we're advertising and in life is actually like they're asking for our trust and it's our choice what we're giving our trust to. Yeah, I thought I don't really know where I was going with that thought but I just thought that that idea was really fascinating as trust as a currency that we actually hold the power to you because I think sometimes saying like, well, but if we have more power than we ever think and sometimes I feel like my only power is voting and recycling. And I'm like, what, what is what else is there to do and I, and I think thinking about it in that way. Like, how do I, it is often challenging but how do I, how am I proactive in like every decision that I'm making is actually supporting a company and a cause and things like that and integrating that more that thought pattern more fluidly into every chance.  And here they both come and saying trust us, you must do this. And that doesn't, that doesn't, that doesn't go over very well that that kind of stock me. And then, and then back to the post and Tracy and all that.  Finding your way back to things you could trust in you or about you, something like that. Yeah. Yeah, and now that was really fascinating because brother fulfillment Aaron Solomon who was speaking with Tracy, he, his conversation about stepping from behind the robes and his position as a monk was really fascinating because he. He said that I need to be known for my action, rather than for the symbol that I represent. And I think, and he made a good point to say like, we often need a physical representation of something that we're aspiring for, or a mindset, or vision of how we would like life to be, but being in that position he actually had to step out of it to like, for a simple term it's like practice what he preached you know like how do you actually just live and be known for the way that you live and that that call that leads into all the internal reflection that you're talking about to was there a piece in there that he thought that the saffron robes might misrepresent him as he became more of an activist. No, thank you. No, because a lot of his activism was in was when he was. Yeah, before he enrobed so well. And was aligned with it well with with being the Buddhist monk. Mm hmm. Yeah. Yeah. He wasn't fully like nonviolent protest that they were doing when they were bringing it was, it was around climate change actually but bringing like the ship into Times Square and all of that and walking in handcuffs and, and he actually got arrested during that protest. But yeah I think that that that aligned with what he was passionate about as a person in the healthcare of our world and everything. But yeah I think the, the idea of stepping from behind the road was more of a personal step how do I, how do I just be known for what I believe rather than. Yeah, I thought I thought his story was really interesting. It was only touched on, you know in that first part of that conversation where he really spoke, but it was also around the idea of our. Our position and the way that we can help in the world and be show up for our own lives can shift so much right and have seasons. And I think that that shows a fluidity that some of that motion that we're talking about where you don't have to stay like locked into one identity. And then even in the item just conversation where you, you grieve even like expectations about yourself and who identities to then move into into others. It was interesting because I think a lot of people can get stuck in kind of a rigid, a rigid expectations and that actually is not only limiting to yourself but also the potential of who you can be and in his position of stepping into this role of not only relationship but kind of a whole new phase of life and also being in a position where he's surrounded by people not in a monastery and able to like to take a lot of his learning to process people through trauma. It's really fascinating. Yeah.  So, to reach more people. Yeah, stepping outside the robes, allowed him to be like a normal person.  Yeah. I know because we all have expectations of an assumptions and bias based in our own histories. Yeah, that was really fascinating thought. Cool. I also, I also felt like the idea of it has to do with the motion that we discussed earlier but the a the conversations around neuroplasticity I feel like that actually came up in a few different discussions. But I feel like it perfectly aligns with what we're what we're saying which is that that openness, the like willingness to process and work through things, which is deep listening is a part of in the rewiring being being open willing and even intentional to change. And with that. And that there is a hope in change because, because of the motion and the way that our bodies can heal and reconnect so I thought that idea of neuroplastic the neuroplasticity like popped up a few different and a few different conversations and that is interesting to be a common thread, but it does align with a lot of the other topics we've discussed. Me too. If we broaden it a tiny bit to be social neuroplasticity. Is that work well too because, because I think part of your knee looking for what are the what are the right messages to change us etc so this this kind of we're busy trying to figure out what to do next about these problems. We're trying to figure out what are the new patterns of behavior and belief. In a sense, and neuroplasticity is usually referencing one skull and one set of neurons, you know, trying to rewire. But here I think we're also like both like definitely the ones inside the skull but somehow also all these, all these other things and there's a there's a book I haven't finished reading that's a favorite or friend friend of mine. It's a handbook or the field guide to interpersonal neurobiology. And it's kind of about like what what happens to us electrochemically when we talk and when we're in relationship and when something. And it feels like we need to figure that out at a global scale. Yeah. I know how do we rewire before midnight or after midnight midnight so he happened. And then there's people. There's people like Jim Bandel who are saying hey, we need deep adaptation because we've already messed things up so badly. That is going to be far less hospitable than it has been so we need to do we need to take larger measures to make sure we survive and manage to, you know, goodbye. Yeah, and that's what that was a in kind of a, this is the end of civilization conversation that Doug rush cross mentioned. It was very much like what drastic next steps are, are what we need to do to move forward through this like, how do we deal with the the crisis of immigration and people being having to move across the world and country and who is going to take the people who are moving from uninhabitable spaces. And then also the idea of how do we move towards food sources and kind of recent ourselves in a way that will work for whatever is coming next. And I think that the, the resistance against change out of fear of it shifting normality is like the, the biggest roadblock, because because there's a comfort in rhythms that we're familiar with, but it's clear that that is that those rhythms are unsustainable so how do we shift our mindset personally and as you're saying socially to adjust for becoming dim. On a happier note. And we just really love our ruts and we love our familiarity right. Yeah. And part of the problem here, or part of the opportunity is reframing these new worlds as as positive as opportunities which is what Virginia did with her kids. They plan their next life, right. Yeah, involve them in the planning. You're like crowds, crowds were so playing for, for the future. Yeah, what, what do you want and how do we have what are our steps to get there. And I do think that the, I think that the hopeful aspect of this is the connectedness. And it is like, and I think that the shift from into like the globalist mindset that we've been in. In the, in the past time is just it's it in that way it's positive because there is still some very fierce nationalism of course, but the idea if we can view others with with care and or at least with an ownership over like global health and the people that the other people that live in the world. And I think that that actedness is how we actually move into any, any next phase of, of growth and future open a positive change so. Yeah. Yeah, one of my friends he's in a lot of OGM calls. He's doing a lot of what he calls positive cartography. And he's trying to bring people together to do that to draw maps of positive futures. That looks like and he's working kind of at all ages and all that he's, he's in Holland. So Northern Europe and all that but it's it's interesting it's and and you can see how drawing positive maps of a wish desired future might actually be helpful. Yeah, I know and I've actually I've actually done that in. Or had like therapist tell me to do that and done that in different stages of my life, life too when going through something difficult and feeling hopeless and saying, Okay, what are 10 positive futures and let's write. Let's write a short story let's write a paragraph about each one, because I think the sense of opportunity and the sense of like, there are forks in the road that lead to more forks in the road is because it's not. There's no change in path and unless I completely turn around, there's no changes coming up. And to remember that we are like in this forks path that will continue for key and there are continued opportunities is to shift and change and. I think that I think that's very fascinating I have the word Tim shell tattooed on my arm because it's around the power of choice from Steinbeck's book but, but it's also like every choice we make sets our life course a different direction so every day there's a way for that versus, we've made one decision and that's, that's where we're going and that sense of motion and positive motion is was a thread throughout all of these. So, such resonance I just had the conversation the sentences you just uttered, I just had that conversation an hour ago. So, about the process we make in our lives and you know, the fork yeah whatever it's really it's really cool. Repeating motion see these conversations are happening everywhere. It's true. October them. Midnight is in the air. Yeah. Oh, well, I've really enjoyed working through this process with you. Same here. Let's go look on the conversation boards and see how this is resonate with other people, because I think that a lot of these topics are so they are so communal and they are. As you said like so many people are thinking about the same things and we're just eager to to discuss and figure out what maybe what are those, what are the possible positive cartography features for ourselves. Exactly. Exactly. Yeah, thank you for all this. This has been a gift. It's been. And just, just the chance to slow down and talk about the sessions afterward with you has been super helpful. Really great. Yeah. Yeah. Well thank you for putting up with the dogs and birds and other things in my current environment. So, there you go. That I know it sets early here. Okay, well I hope you have a good evening. And tomorrow. Yeah. Yeah, let's talk more. And then I'll definitely play into maybe meet up and have a conversation when I'm in Portland to sounds great. Okay. Thanks.


What I would like for us to do is get our top five players currently in the NBA. Right off the bat, like no explanations needed, just whatever. Just let him go. Let's start with number one. For me, right now it's Jokic. Two, Yanis. Three and four are close for me, but three put Luca for Tatum. And then five, Steph Curry. I just feel like he's still the best shooter in the game.  My list is very similar, but you're not going to like the one that I have on my list that you don't. I have Jokic, Luca, Yanis, Embiid, Jason Tatum. I mean, the reason why I don't have Embiid in mind is like, I know he's reigning in repeat and everything. For as big of a human being as he is, the way he just flails for foul calls, like, is embarrassing.


team as software engineer it's going to be about Ansible Automation to custom your own industry that is to build your own atomic host but this talk is not going to be about introduction on an atomic host because Dusty already covered that thing so composing of poetry so it's blind server architecture so you have a set of RPMs in the server based on the architecture the requirements would be I need a machine that would act as a server that is going to host the web server to host my oyster repository and the atomic host line that is going to compose the oyster and just configure it through our workstation as a server that that is going to host the oyster oyster package oyster packages and so you will need all the chemo for Fedora atomic they have a script on pack your .io so that is used to build atomic host for that as well in order to add your packages since atomic host is immutable you need to if you want to add your own packages you just added the manifest file there there is a file named Fedora atomic Docker which is a JSON file it has all the list of packages that atomic host is going to have so suppose you need to want to add VM team or server whatever just go on adding the packages there to start the web server that is going to make the oyster available for the consumption of atomic host you can't imagine that so I'm going to create VM or VM from it that is going to act as client and you just rebase it and so that atomic host can use that composed oyster so these are the steps when you want to have your own customer is your own atomic host so I just so I just explained all the steps the first step was my server machine and the requirements so the here is the atomic build script that Fedora atomic has so you will need to clone this repo that has the metadata and script to build the atomic host and this is the file inside that repo where that whichever you have to edit to in order to add packages for the web server I have used simple HTTP server I'm going to make it available on port 35,000 and there we compose the oyster using that manifest file find VM I'll create just rebase the oyster and make the atomic host using the new custom oyster so these are lots of manual steps that you have to do when in order if you want to I mean build your own atomic host or create custom oyster so so I thought to just simplify more in my work that is why I choose Ansible for that so why Ansible because Ansible is agentless so it says client server architecture based you do not need and also you do not need to install any third party packages on server or client in order to use Ansible you just need to install Ansible and Python 2.7 right now we are going to have Python 3.5 support as well later on few releases and it's simple it just uses YAML that is yet another markup language which is totally like English so you do not need to spend time learning a new programming language and it's powerful even it is simple it's powerful because if you know Fedora infrastructure applications they use Ansible to maintain their applications and deploy their applications so even though it's very simple but it's very much powerful and it's secured it uses SSH to establish connection use Ansible for this composing oyster I'm going to show you the steps to website and just make sure that you have Ansible installed on your host and after you've blown the repository and install Ansible you just need to run this playbook setup.yaml on your system so what it is going to does it is going to install the requirements on your system that I was talking about like RPM oyster packages and packages required to create VM from the computer images everything it is also going to start the web server that you will require to solve the oyster and some basic initializing oyster and few steps so after it is done you will need to find the IP address of the HTTP server you just solved so you can use IPADDR command for that and after you have it just put the IP address in the variable file atomic.yaml so there is another variable file where you can add the packages you want to have that is so I wanted to have VM enhanced and get so I added those packages in buildrepo.yaml and there are a few other informations you need to add that is to create VM from QCao image so the client that I'm going to create that you need to provide I mean how many CPUs you want to give it how much of memory and the name of the image that is whatever image you downloaded so after you are done with putting all this data in the variable file just run the main.yaml file which is going to compose the oyster and start the atomic host client and make atomic host consume the new oyster so that's all you are done with having your own custom oyster just reboot the just SSH to the atomic host and if you do our PM query you will have all the new packages added there so I have I have used cloud in it for adding the user ID and password for the VM I have created since right now it is atomic user and password atomic so if you just go on building the repo you will need to use the same username password but you can edit this as well just go to the template file in the repo there is a file where you can edit the user data so on this network it took me a while to compose the oyster that is why I have recorded the demo so I just ran the setup.yaml file now it is running the yaml file which is installing the packages and it started the HTTP server and to make the oyster available then I checked the IP address of the HTTP server and I just put on the IP address in the file that is this one for me and I'm not just going to run the main playbook right now so it is just creating the VM from the kukato image right now you see all the steps here it's build script that is on peggior then it added the packages in the json file that is the manifest file in on the repo and now it is composing the oyster so on this network it took me more than one hour to compose the oyster so I'm going to skip it still composing compose is done and now it is going to retrieve the IP of the VM the atomic VM we have created so this is the current IP now you just SSH to it now it is rebasing the oyster if you if you want to check your atomic host VMs IP you can use the verse command as well now I just SSH to it and checking the RPM oyster status so if you look at the RPM oyster status it's not visible there so there is a bullet sign beside Fedora atomic here which means the atomic host is still using this oyster but we right now have the new oyster composed so in order to make the atomic host use this I'm in custom oyster we need to reboot of reboot so I tried to automate this with Ansible as well but somehow it was successful but it showed me an error message like time out so that's a bad thing in Ansible we are still working on that thing so after reboot just you need to manually SSH again now I'm checking the status again it's not visible on the screen so now right now the bullet sign is on my customer history and I'm checking the packages that I have added in the manifest file and you have all the packages on the atomic host so get the script lines and these are the requirements I have used in the playbook and these were required in order to achieve this this is the setup ml file so here we are installing the requirements and we are doing some basic setup for in order to serve that oyster repository and then this is a file for creating VM so in last defcon Jonathan wrote a script on this it was a shell script so I literally converted that into a playbook so and it became more readable to me so this is the createvm script it is included in the main playbook so you do not have to care about like you have to run that playbook or so these is the main playbook where we have included the createvm and the compose oyster playbook now we are just doing some basic steps for the SSH after the compose and createvm create it's done we are just retrieving the IP of the atomic host vm that we have created and just copying the SSH key so that we can just go ahead and SSH to it and then reducing the oyster and that's it so in just four variables you in order to use this you need to keep in mind that you add these variables that is this is the HTTP server IP address and this is the workspace I have used in this this is the place where you are going to you want to have your atomic build script that is so I have flown the back yours at federal atomic script that this that is this one the repository sorry so it when you add this directory here it is going to clone that repository in this place and then just add it add the packages you want to have in your new atomic host and these are the details for the vm that's it here is the source and slide and blocks blog post if you want to go ahead and read it any questions on this one there was a proposal on it he wanted to do for bare metal provisioning support for this would you like would you can I mean repeat the question I didn't get it we can so in answer I think it would be best to I mean separate keep it abstract abstract via variable not I mean literally everything in playbook so you can use the you do not have to rewrite playbook all the time you have when you have different requirements you just add your variables and just run the playbooks I'm not talking into this issue actually right now but I will maybe some weekend yeah so actually what I did before I wrote an ansible module for the rpm oyster but then we thought that maybe it would be useful to write ansible module for client side use so so actually we need to decide I mean maybe from the atomic community like what kind of client side module you want to add so you mean the atomic host client side or rpm oyster commands like adding packages or checking status this kind of okay okay yeah that would be great so for that maybe we can create an issue create issues on ansible as RFE that would be useful to keep track anything else


Hello, everyone. Today, Michelle and I will give an introduction and update to Kubernetes 6-Storage. My name is Xin Yang. I work at VMware in the cloud storage team. I'm also a co-chair in 6-Storage along with Sadali from Google. Hi, my name is Michelle. I work at Google and me along with Yan from Red Hat are tech leads for 6-Storage. Our session today will include two parts. In the first half, we will give an introduction. In the second half, we will give an update and a deep dive. In the introduction part, I'm going to talk about some basic concepts in Kubernetes storage and how to get involved. Kubernetes storage provides a way for containers in the pods to consume block or file storage. Persistent storage is one type of storage that live beyond a pod's lifecycle. The technologies we heard the most in 6-Storage are probably PVC, PV, and storage class. PVC, persistent one-and-claim, is a user space object. It is a request by a user for storage. A PV persistent volume is a cluster scope object. It represents a physical volume on the storage system. A PVC and a PV have a one-to-one mapping. Storage class is in the cluster scope. It's a way for admin to describe the classes of storage. Different classes might map to different quality of service levels or other admin-defined policies. In dynamic provisioning, storage class is used to find out which provisioner should be used and what parameters should be passed to the provisioner when creating a volume. A pod is a group of one or more containers with a shared storage and network resources and the specification for how to run the containers. A pod is a user space object. A PVC is used by a pod. In static provisioning, a cluster admin creates a number of PVs, which carry the details of the real storage. The control plan can bind PVCs to PVs in a cluster. However, if you want a PVC to bind to a specific PV, you will need to pre-bind them. When none of the static PVs, the admin, the administrator created, match a user's PVC, the cluster may try to dynamically provision a volume specifically for that PVC. The provisioning is based on storage classes. The PVC must request a storage class and the admin must have created and configured that class for dynamic provisioning to occur. So here are examples of a pod, a PVC, and a storage class. The pod is using the PVC. PVC has capacity, access modes. It's a read-write once here and a storage class name specified. And in the storage class, there is a provisioner that determines what a warning plugin is used for provisioning PVs. The reclaim policy is retained here. That means the PV will remain along with physical volume on the storage system when the user deletes the PVC. If the reclaim policy is deleted, the PV along with the physical volume will be deleted when the user deletes the PVC. And allow warning expansion is true here, indicating that warning expansion can be requested by the user. And the warning binding mode can be either immediate or wait for first consumer. It is immediate in this example, indicating that a warning binding and dynamic provisioning occurs once the PVC is created. But this may result in unschedulable pods. So a class set man can address this issue by specifying the wait for first consumer mode, which will delay the binding and provisioning or PV until a pod using the PVC is created. Storage class also has parameters that are storage provider-specific and opaque to the Kubernetes. And next, I'm going to talk about FMR storage. FMR storage becomes available when the pod is started and goes away when the pod goes down. So here we include local FMR storage that includes EmptyDir, Secrets, ConfigMaps, and Donmar APIs. And we also have CSI FMR volumes here. And then there's also generic FMR volumes. So I will go over them. EmptyDir volume is first created when a pod is assigned to a node and exists as long as the pod is running on that node. As the name suggests, the EmptyDir volume is initially empty. All containers in a pod can read and write the same files in EmptyDir volume, though that volume can be mounted at the same or different paths in each container. And when a pod is removed from a node for any reason, the data in EmptyDir is deleted permanently. And EmptyDir volume can be used as a scratch space. A secret volume is used to pass sensitive information such as passwords to pods. You can store secrets in the Secrets Kubernetes API and mount them as files for use by pods. Secret volumes are backed by tempfs, so they are never returned to non-volatile storage. ConfigMap provides a way to inject non-convidencial configuration data into pods. When referencing a ConfigMap, you provide the name of the ConfigMap in volume. In this example, the ConfigMap is mounted as a volume, and all the contents are mounted into the pod at the path derived from the mount path and the key in the ConfigMap. And then next, I will talk about the downward APIs. So here we can see an example of the downward API volume. It makes downward API data available to the applications. It mounts a directory and writes the requested data in plain text file. And in this YAML file, you can see that the pod has a downward API volume, and the container mounts the volume at this specified location. And each element on items is a downward API volume file. So the first element specifies that the value of the pod's metadata labels field should be stored in a file named labels. So next, I want to talk about the CSI inline fmoral volume. So we set warning types to CSI in the pod spec and specify the driver name and the warning attributes. And for a CSI driver to support the CSI fmoral volume, it must be modified or implemented specifically for this purpose. A CSI driver is suitable for CSI fmoral inline volume if it serves a special purpose and it needs custom providing parameters like drivers that provide secrets to a pod. So the secret store CSI driver is an example. A CSI driver is not suitable for CSI fmoral inline volumes when provisioning is not local to the node or when fmoral volume creation requires warning attributes that should be restricted to an admin. For example, parameters in a storage class. Next, I'm going to talk about generic fmoral volume. The feature allows any existing storage drivers that supports dynamic provisioning to be used as an fmoral volume with the volume's lifecycle bound to a pod. It can be used to provide scratch storage that is different from the root disk. For example, persistent memory or a separate local disk on that node. All storage class parameters for warning provisioning are supported. All features supported with PVCs are supported such as storage capacity tracking, snapshotting, cloning, warning resizing. This feature is beta since 1.21 release and it is targeting GA in 1.23. So that's all for the introduction of fmoral volumes. So now I will talk about warning plugins. Kubernetes warning plugins include entry plugins, auto-tree flex volume, and CSI drivers. Some entry plugins, such as those fmoral ones that I mentioned earlier, will stay entry. But most of other entry plugins are either deprecated or migrating to CSI drivers. Michelle will talk more about that in detail later. Flex volume is deprecated. CSI driver is the recommended way to write plugins. The Kubernetes implementation of the container storage interface CSI has been GA since the 1.13 release. CSI is designed to be vendor-neutral, interoperable, and has a focus specification. It defines a set of storage interfaces so that a storage vendor can write just one plugin and have it work across a range of container orchestration systems. In the CSS spec, we have RPCs for volume lifecycle management. This includes provisioning support, such as create and delete volumes. And RPCs that make sure volumes are available for part to use, such as attach and detach volume and mount and unmount volumes. And it also has other functions, such as expand volume, snapshotting, cloning, holding house, and so on. So here is an example of a CSI deployment that shows various Kubernetes components. The CSI driver and the storage system that is used to persist the data. Here we have the CUBE controller manager on the master node. CSI driver controller plugin is deployed together with Kubernetes CSI external provisioner, external attacher, external resizer, and external snapshotter setcars. Note that CSI driver controller pod does not have to run on the same node as the Kubernetes master, but it's recommended to run on dedicated control plane nodes. The Kubernetes CSI cars are watching Kubernetes API objects, such as persistent volume claims, persistent volumes, volume attachments, volume snapshots, to detect create volume, attach volume, volume expansion, volume snapshots, requests. The setcars, called the CSI driver, and the CSI driver communicates with the storage system to complete those loading operations. On Kubernetes worker nodes, we have Kubelet and the CSI driver node plugin deployed together with the node driver registrar setcar container. Node driver registrar, such as driver information using node.getinfo from a CSI endpoint and registers the CSI driver with Kubelet on that node. Kubelet directly issues CSI node.getinfo node stage, volume, and the node publisher volume calls against CSI drivers to get info and mount volumes. So that's all for the basic Kubernetes story concepts. Next, I will talk about how to get involved. So here is the Sieg storage community page. It has lots of information to get you started. We have bi-weekly meetings on Thursdays, where we go through features we are tracking for each Kubernetes release and discuss any design issues or other issues added to the agenda. This is a good place for a new contributor to get started, join the meeting, and see how the Sieg works, what you're interested in, and get assigned to work on some tasks. The communication within Sieg is through the main list or the Slack channel. I included some resources for your reference. Here are docs that explain what are the Kubernetes story concepts and what is CSI. The last reference is an example to deploy the sample CSI host pass driver. For a new contributor who wants to contribute code, it's good to follow this example and learn how CSI works. So that's all for the introduction. Now I will hand it over to Michelle for the Sieg storage update. Thank you, Xing. So I'm going to give an update on some of the major projects and initiatives that the Sieg has recently completed and is also actively working on. I'll start with a deep dive on two major projects that we've been doing and then give an update on 122 and what we're targeting for the upcoming 123 release and then go into some longer-term projects that we're sort of investigating and designing. So let's deep dive into the first project, which is CSI migration. Next slide, please. This is a subarea of the overall cloud provider extraction effort, where the built-in cloud providers are going to be removed from Kubernetes starting in 1.24. The overall effort has been ongoing for a couple of years now, and we're finally reaching a point of maturity where we're feeling confident in switching over to the external cloud provider model. Now, persistent volumes has some unique challenges compared to the rest of the cloud provider components. The main challenge is that many of the original volume types are built directly into the core Kubernetes API. That imposes a very strict backwards compatibility requirement with this entry API, so we can't just remove the built-in volume plugins directly. So what we've done instead is CSI migration. The CSI migration is a feature that lets you continue using the existing API for your existing persistent volumes and storage classes. But underneath the covers, it actually makes calls to a corresponding CSI driver instead of going through the built-in drivers. So right now, all of the major cloud provider plugins have beta support for CSI migration, including AWS EBS, Azure Disk and File, OpenStack Cinder, GCEPD, and vSphere volumes. And all of these plugins are expected to GA their CSI migration implementations starting in 1.24. Next slide. So what do you need to do to turn on CSI migration? The answer depends on how you are provisioning your Kubernetes clusters. If you're using a managed Kubernetes distribution, then in most cases, you should expect that the distribution will take care of everything for you transparently. But it is best to double check with your provider to make sure that is indeed the case. If you are managing your own Kubernetes clusters, then you will have to install the corresponding CSI driver for your cloud and then enable the CSI migration feature gates in a very specific order, which is detailed in the link here. There's also a few caveats to watch out for. First, despite using CSI underneath the covers, turning on CSI migration does not enable CSI only features for your volumes that are using the older APIs. CSI migration is meant to be able to provide backwards compatibility with the entry API, but it's not designed for future compatibility of future CSI features. So if you need to use the newer features like CSI snapshots, then the best option is to actually re-import your persistent volumes directly using the CSI API. Another caveat is that some of the drivers have some rare corner cases and behaviors that might not be supported by the CSI driver. We expect these behaviors to be rare and not commonly used, so most people should not be impacted. Nevertheless, we have explicitly deprecated all of these behaviors that we're aware of. So please double check the Kubernetes release notes going all the way back to Kubernetes 117 and look for these deprecation notices for your cloud. Also because of this, it's important that you start trying out CSI migration in your non-production clusters so that we can catch any other behavioral differences that we might have missed as early as possible and then we have more time to address those issues. So in summary, CSI migration is coming up really fast within the next half year or so. If you're using one of these internal cloud providers, please help us out and start testing out your workload and workflow compatibility with this feature. We have the CSI Migration Slack channel and we'll be there to help answer any questions that you might have and be able to help you out with any issues. All right, so that is CSI Migration. Moving on to the next feature deep dive is CSI Windows. So we GA CSI Windows in the most recent 122 release. What this feature is, is a way to be able to run CSI drivers on Windows Notes. One of the biggest challenges that we faced was dealing with the lack of Windows privilege container support, which is something required by most CSI drivers to be able to do privileged operations on the file system and the mount points. So to deal with this, we created a binary called CSI proxy. CSI proxy runs as a Windows service and it exposes a GRPC endpoint to CSI drivers for doing privileged operations. The current operations that CSI proxy currently supports include disk and volume operations, operations on the NTFS file system, and Samba or Sips. ISCSI support is also available in the Alpha phase. There's a number of CSI drivers that have implemented CSI support on Windows, including AWS, CBS, Azure Disk and File, GCE, PD and a generic Samba driver. So if you're running on any of these platforms, please check this out. One thing to note, in 1.22, there is also a new Alpha feature put out by SIG Windows that adds privileged container support. So once that graduates in maturity, then potentially we can remove the need for CSI proxy and having a separate service to do the privileged operations. However, to retain backwards compatibility with existing CSI drivers, what we're planning on doing is to turn the CSI proxy GRPC client library into a normal library with using the same APIs. That way we can minimize the transition between the CSI proxy model to making direct library calls and that should mitigate any major changes that you might have to make in your CSI driver to transition between the two models. So if you're interested in the Windows work or if you have any further questions, we have a CSI Windows Slack channel where you can come in and ask any questions or get help on any issues you may have encountered. So now I'm going to give a brief overview on some other projects in the SIG. There's a lot of efforts going on right now, so I won't be able to cover all of them today, but I'll highlight a few. First, in 1.22, we GA two major features. The CSI Windows, which we just talked about, and CSI service account token. This is an important security feature that enables CSI drivers to authenticate using the pod service account token. This allows CSI drivers to be able to support per pod authentication instead of broader shared credentials that many systems may require today. One example of a CSI driver using this feature is the secret store CSI driver, which lets you mount secrets from an external secret manager like Volt. If you're a CSI driver author, please take a look at this feature and see if it can help improve your security posture. So moving on to 1.23, there's a few features that we're gonna target GA. A next slide please, thank you. The first two are related to FS Group and volume ownership. The first enhancement improves scalability when mounting very large volumes that contain a lot of files. This feature adds a new option to the pod spec that tells Kubelet to skip the process that updates volume ownership. Before this change, we've seen instances where mounting a very large volume could take more than 30 minutes, but after this change, mounting of a volume is back to a couple of seconds. So please take a look at this feature if you're experiencing slow mounts and you have very large volumes. The second FS Group feature is for CSI driver authors. It lets CSI drivers explicitly opt in to support FS Group. Instead of using a heuristic that was not accurate in a lot of cases. So if you're a CSI driver author, please take a look at this FS Group policy feature. The last feature that we're targeting for GA is generic ephemeral volumes. Qing talked about this earlier, so I won't go into a lot of details, but to summarize, this feature allows a pod to specify a volume template which will dynamically provision and manage volumes with the same lifetime as your pod. Any existing volume type that supports PVCs today will be able to work with this feature. So go ahead and check this out. All right, and so moving on, now I will highlight a couple efforts that we're actively prototyping and designing right now and where we could actually use a lot of help from the community. First up is non-graceful node shutdown. This effort is exploring ways that we can safely fence and detach volumes when a node is shut down and still ensure that we don't get into a split brain scenario. This is in a design and prototyping phase. So if you have any ideas or you have work in this area that you think will be useful, please join us in these discussions and we would appreciate any ideas or help that you can provide. Another interesting initiative that we're starting is to explore is volume snapshot namespace transfer. This project is looking at ways to allow easily moving volume snapshot objects to another namespace. And potentially depending on the ideas explored here, we could expand that to also look into ways to actually move volumes across namespaces as well. Another exciting area that we're starting. Next slide, please. We're also collaborating with a couple other SIGs on a variety of other projects. I'll highlight the container notifier effort. This is a feature that allows for sending custom signals to a pod and being able to define custom actions for that pod to take when processing that signal. This is the mechanism that we're exploring to be able to quiesce an application before taking a snapshot. So this is a key driver for being able to enable application consistent snapshots. Another interesting proposal that we're exploring with API machinery is leans. This is offering the ability to essentially lock an object and prevent it from being accidentally deleted. Now you may think, how is this different from finalizers? The main difference is that finalizers, their main goal is to ensure that all of the deletion handlers for an object are able to finish before the object is finally deleted from the API server. With finalizers, the big difference is once the deletion process starts, you can't reverse it and you can't undo a delete. With leans, the biggest difference is that leans will prevent the deletion process from starting in the first place. So this will help with the scenarios when you really want to protect some objects from being accidentally deleted. And so that's another exciting effort that we are working with API machinery on. And we're hoping to be able to use this to add some extra protection to secrets, especially in terms of protecting secrets that are used for volumes. All right, so as you can see, there are a lot of different projects that we are working on in sick storage. And we could use help all over in all of these efforts. So we always welcome new contributors. If any of these projects sound interesting to you, please reach out to us through our Slack channel or join one of our SIG meetings that we hold every two weeks. We welcome new ideas and new contributors, and we'll definitely try to help you get involved into the SIG. So that concludes our presentation today. Thank you very much for watching and we look forward to seeing you in the SIG. Thank you.