Good evening, everyone, from a beautiful fall day on the Carnegie Mellon campus. I'm delighted to welcome you to tonight's event, heading to the point, The Power of Representation. I'm Keith Webster, Helen and Henry Posner, Junior Dean of University Libraries, and it's my great pleasure to be part of what is going to be a remarkable event. Here at the University Libraries, we are committed to recognising the importance of representation and diversity. We have a number of initiatives centred around recognising and growing diversity throughout the campus community, several of which you'll hear learn more about tonight. Many thanks to my colleagues who have been working hard to prioritise this important work. Tonight, I'm particularly grateful to Sonya Wellington, our Events Manager, who looked after all of the logistics for this event, as well as our entire External Relations team, and the team at the Office of the Vice Provost for Diversity, Equity and Inclusion. It's my great pleasure to introduce you to this evening's host, Dr Wanda, heading ground. Wanda is Carnegie Mellon's inaugural Vice Provost for Diversity, Equity and Inclusion and Chief Diversity Officer. She also holds a faculty appointment as a distinguished service professor in the Hines College of Information Systems and Public Policy. Throughout Wanda's 30-year career in higher education, she has established programmes, policies and practices fundamental to the advancement of inclusive excellence and DEI. Her wealth of professional experience and volunteer involvement on the boards of non-profit organisations and civil rights committees have earned her a reputation as a cultural architect able to build and sustain real and lasting change. Joining Wanda this evening in conversation is award-winning novelist, college professor and CMU alumna, Dr Joel Parker Rhodes. Joel has a rather special story about Hunt Library where I'm sitting just now. I hope she'll share this story with us at this event, but let me just say that an instance of encountering representation in the Hunt Library stacks kick-started Joel's literary passion. Since that moment, she has had an incredibly successful and impactful career. In 2021, she was awarded an honorary doctorate of Humane Letters by CMU, joining her CMU bachelor's, master's and doctoral degrees, and so I describe her as a four-time CMU alum. She's the author of seven books for children, including the New York Times bestseller's Black Brother, Black Brother and Ghost Boys, which has won over 30 honours and awards. She has also written adult novels and memoirs and writing guides. And for those of you on campus, we have a small display of Joel's work in honour of this evening's event on the first floor of Hunt Library. Joel has won the American Book Award, the Black Caucus of the American Library Association Award for Literary Excellence, at Coretta's King, Scott King Honor Award, and the Jane Addams Peace Association Book Award. Joel is currently the founding artistic director of the Virginia G Piper Center for Creative Writing, Narrative Studies Professor and Virginia G Piper Endowed Chair at Arizona State University. I'm also delighted that Joel is a member of the Carnegie Mellon University Libraries Dean's Advocacy Council, and she is an increasingly important ambassador and advocate for our work. Towards the end of this event, you will also have a chance to hear from my colleague, Crystal Johnson, the community collections archivist in the libraries. Crystal's primary role is to increase representation in the university archives, both by making collections featuring underrepresented groups available to students and researchers, but also by building connections across campus with student groups and organisations so that we can accurately record their history. Crystal will share some highlights of her work with us and how it enriches our community's understanding of Carnegie Mellon's diverse history. Thank you again for joining us this evening, and without further ado, I'll turn things over to Wanda and to Joel. Thank you both. Thank you, Keith. Wow. I don't know how I'm going to show up with this wonderful woman that I keep calling a queen. And I love when she smiles because it is absolutely the truth. Hello, Joel. Hello, Wanda. How are you today? I'm doing well. And I was just so warned by Dean Webster's earlier comments because I just think it's such a wonderful thing that we're at this time of our culture where we're finally super committed to DEI and we're all pooling in the same wonderful direction. That's right. That's right. Well, you know, I'm not sure who's all out there, but I welcome you. Okay, I'm in my office. I'm sure Joel's in her home in her office and maybe, but we're welcoming you to our living room right now, you know, our family room. And let me tell you, I was excited about the opportunity to have a conversation with Joel, but I did not know how she was going to just knock my socks off when I first had a conversation with her. It was supposed to be 30 minutes. I think it was maybe an hour 15 minutes later that we got off the phone and we laughed at my jaws hurt and, and I was like I can't wait for this. So, I'm inviting everyone into our space today to hear part of the conversation here, here, here are our conversation. One of the things I wanted to say about what really just sort of stood out to me and and we got to this. The title around representation is because so much of what I understood and knew about this author that's sitting across from me really just resonated with me. For those of you who don't know I'm just going to share a few little tidbits here about myself. I went to a predominantly white institution. I went to that institution site unseen. I arrived here, not knowing exactly what I was going to get into. I am a first generation college student. I chose the school that I chose because I didn't have any role models or anyone to tell me anything different or to look for certain things. I also was not able to go and visit the school I didn't even know anything about the, you know, how do you say admitted student visit days or those kinds of things I went when we could afford to go which was when schools started. There were times when I learned for seeing someone and being with people that looked like me that somewhat sounded like me something and it didn't always exist. But here's the other great part is the part that's great though, that I was welcomed, and I was embraced. And there were things that I knew that I did not have and to my total preparation for being where I was at. And, and folks helped me to not just survive but to also thrive. And then the third thing that happened. One, made a place that I, maybe not the only one but it felt like that to not quite sure how to navigate being in college but I found my way and I thought there were people and three, it was the beginning of more of a formal journey for me around justice and and equity and inclusivity and understanding it I always did the work, given who I am but it really became more formal. So, when I think about that. And I think about what I was initially told about Joel, I was like, we got to talk. And then we did. And boy, I was like, I'm going to say a few things in the beginning. But I want people to hear all about your story CMU. If they're not I know they are though, very proud of having you as one of its alumni and alumni and I am so glad that we're having this conversation so with that in mind. What I really want to do is really invite you to tell me about and tell all of us about somewhat about your upbringing your background how did you end up at CMU. And, and, you know, maybe at some point we're going to get to when you show up with your I'm like, I can't wait to hear that again. So, so in that regard. So, you know, I invite you just to share with us a little bit about how you got to see him you. I'm happy to do that.  And for that, I'm going to always always be grateful.  We're all a mixed blood stew. And that idea of mixed blood stew of how you know, we're part of the same genetic gene pool still informs my work today. Literally skin tone is but another superficial difference, and that we want to break down the barriers to keep us from seeing one another and communicating with one another. Now, when I was a little girl though I was pretty lonely, I was pretty shy, pretty sad, because my mother had left the family when I was eight months old. My sister and I were raised with my cousins by my grandmother and aunt and father and uncle in this big extended house on the north side of Pittsburgh. And even then I was a reader teachers librarian started giving me books, and I'd read and read and read and most of those stories are about what not most of those stories all those stories about white kids right and they all seem to live on a farm. And so I did. And also about dogs identified greatly with the dogs that were hurt unfairly, and then of course, were transformed with human kindness, you know, so basically be trying to give myself what I needed, but I wrote stories, all the time. My first story was in the third grade it was called the last screen at Homewood Elementary School, and my teacher let me walk last screen last green so you know how it ended Wanda. And I illustrated it to, and actually I still have that book, but I remember that connection when I was telling my story, and students were like, Oh, yeah. Afterwards, shortly after that my mother who I hadn't seen in eight years decided to come back and raise the family, and she took us all to California. And that was just heartbreaking for me. And there I really found the power of the arts that that was the way in which I could express myself. Also, since I didn't see myself in books, I could see black people doing sports, I could see black people entertaining, you know, on TV, so it's sort of like, Oh, I can do that. Yes, very much wanted to be out of where I was. I wanted to be in a far, far happier place. And actually, my mother when I was 15 years old, she decided to divorce my father, and ended up kicking me out of the house. And I remember saving money and being at the Ontario airport, getting in one of those little prop planes, La x, and my father was standing on the on the tarmac because you could do that then. And they were playing is it mama's and Papa's I forget the song. On a jet plane. Don't know. And I went back home to Pittsburgh and grandma, and I had the knowledge. My drama teacher in California said, Oh, you should go to Carnegie Mellon. I think it's in Chicago. Oh, no, it's here. I was 15 years old. I was young. I was really young. And grandma put me in Allegheny Community College because she was worried that I would become, you know, someone who didn't go to college, you know, which as first generation, many of my family didn't. And I was also afraid that I would end up in jail as many folks in my family, unfortunately did, or else that I would be a team mother, as many folks in my family, including my mother were. She sent me off to community college while I waited for my addition for drama school at Carnegie Mellon. Now to be clear, my educational background was very poor of the discrepancy between what children of color are taught in public schools. Because, you know, kids who are white, say more affluent suburbia is just horrific, and it's still going on today.  The school that accepted me was Carnegie Mellon, because I could sing, dance and act, and they were starting an inaugural theater dance program. You know what, there were about seven of us, seven of us, and four of us were people of color. One was Asian, and when I thought back on those days, you know, we had a black writer in residence, we had a black director in residence, I got to do a main state show that was about anti apartheid that there were ways in which the arts and Carnegie Mellon drama did give me representation, and they filled me up and make me feel as though I was growing into a person. I want it more. I want it more. You were finding yourself. Yes, exactly.  And one day, and I don't know why I was going to the library because sorry, Dean Webster, we didn't go into the library a lot. That's his drama. Yes, but I but I went into the library, and on the news fiction shelf. I Gail Jones called Carigador, and I read that book and it was about the slave trade. And that was the first time that I had ever ever seen myself in a ball. I was 19 years old. And the next day, I was switching my major to English. And I ran a Dean stay on standard because I was poor I had no money we could talk about that later. But I was worried I couldn't afford to go to school longer than four years. I was barely making it, you know, through my junior year. And I remember Dean Steinberg saying, Oh, I've got the degree for you. He opened up, you know, his file folder, and he pulled out this degree in drama criticism. He says nobody's ever taken it before because you have to do three years in theater, be accepted there and then two years in English. But if you do this, you'll graduate. In English, though, where I was the only person of color, and it was different because when I started writing, I wanted to write about what I knew how I had grown up, you know, things that were true to my community and show the humanity of that.  And I'd go, Well, how come you don't tell me your characters are white. And I learned the real tragedy go on. I had learned to read white, that I have been, you know, a culturated, waiting for someone to say, Hey, it's, you know, a person of color. And the tragedy is that through all of my English courses, you know, many, many years, I never once read anyone who was a person of color. It was a problem, I think, for the time. But in terms of the professors, in terms of loving and supporting me, I couldn't have asked for anything more. And in fact, if my writing teacher David Walton it said, you know, you know, sorry, you can't write these stories or you can't do this. I would have tucked tail, and I would have run because I was so insecure in my own agency, my own voice. And with time, Carnegie Mellon gave me the educational background and I remember running to the library Dean Webster running to the library. What people were talking about, before I could figure out what my homework was, is making up and having you know teachers help me with my writing, not just English teachers, but philosophy teachers. And one thing that I had, that I would always say to myself, even though I was ill prepared for college, was I still smart, you know, could I learn it. And that sense of yes, I could have been echoed in theater, have been echoed in English. And so I did work and in fact I got graduated with 4.0 as an undergraduate and graduated with distinguished letters for my doctoral degree. And that's when my eyes went so bad I had to start wearing glasses. You say something very interesting, because you know that all of us don't get that in terms of being told that we're smart or the sense of that we can do it. So I mean, I remember being struck by that when we were talking before and, and sometimes that is missing so much that that played a significant part in you really being able to move forward. Absolutely. And to have someone to see, to recognize my, my talent, and particularly in a time and I hope it's a lot less now. I also remember when I did my GREs that some people said, Oh, they weren't very good. Oh, you can't go to graduate school, but I remember twice that scores from SAT and GRE sort of stopped, would have stopped things for me. You know, who knows just just just because I could sing and dance. I was able to break through. And one story I shared with you is that I remember in California being in the fifth grade with a student in Hispanic girl, and we were the two smartest girls in the class. What smartest kids ever.  So we were being invited to the fireman's picnic.  I actually didn't care because I was going with a friend I was wasn't going to be alone. But I can see how in terms of representation that she thought it was important for me to know, and maybe for her to know that I had done really, really well. And literally years later, I'm becoming an associate professor. I've been, you know, invited to campus at Cal State Northridge. I'm getting my badge and you know my ID card, and the person taking the picture was in fact my fifth grade who had not gone to college, who had not moved beyond environments for whatever reason.  There are millions, millions, exactly who are gifted and might need support and love. And yet if the world doesn't have representation. If you don't see if you don't know that these other avenues exists and I had learned that also through just imagination fiction, and then through my wonderful teachers, then you can just, you know, not even take that risk to say, you know, I'd be interested in trying this, but there is no reason that at that point we should have had such a gap, equity and achievement. If you were saying based upon on talent. I know I had more, perhaps who knows who knows the stories, but I had, I know I had more opportunity. And the other thing. Oh, and I love Carnegie. The other thing is that they never ever wanted anything than excellence. Always. And so standards were never changed. It was like, okay, you have to meet them. And I remember the first time in graduate school I had to analyze a poem for the class. And he said it was an Edward Arlington poem, and and he said, No, that's not it. And then very politely and kindly, you know, went back to the whole program and explained all the things that I should have been like explaining or include. I managed to say that no you're wrong, but to teach me with such a loving embrace that I kept working and kept working. And three years later I wrote a paper 26 paper page paper on Dylan Thomas is a refusal to mourn 13 lines of a poem. And she wrote at the bottom of my paper. Yes. I have every single paper every single comment from every single person at Carnegie Mellon, and I setting the bar high which is another thing it helps with representation, because you do feel and I don't know if you felt this wound up. You do feel that sense of that that you're fake, or you're not authentic. What's the. The imposter you're talking with imposter syndrome. And that haunted me for years. And in particular, it haunted me because, you know, there wasn't much attention given to African American literature. I now recognize that I was part of the generation by the time I graduated. They were saying, can you teach African American literature, can you teach women studies that I was part of the people who became that way that got into the Academy women studies, that, you know, we were, we were in fact, the vanguard, and how we've made a difference, you know, that way, which I think that sense of agency and belief in your voice, I still got from Carnegie Mellon, very often still my family or my community was sort of like, I became much more like a stranger in a strange land that it was like, you know, I remember my father even saying, I don't, I don't understand. I don't, I don't get why you like the things that you do. And now we have much more of a sense that rather than liking things that society lets you see or have access to that for everyone regardless of color regardless of gender regardless of gender, religion, you know, ethnic background, et cetera, should have access to everything, everything. Yes, there, we didn't know or this is the great violinist, and we didn't know. So we just need to keep pushing. Well, you make, there's a few things that have come to mind. The first thing I sort of want to say is that, you know, to who much is given much is expected and required, and I think as educators and, and, and those of us who are very specifically in certain kinds of professions that we cannot afford to be leaving millions of talent on the table. And part of us that happening has a lot to do with not nurturing and fostering and caring and the kindness that we should be providing to all children to all people. And I think that there's an aspect in here that I hear from you but I know for myself, if not being provided that level of access and opportunity, along with my own talent and ability that I could have been somewhere and my talent on the table, and we would not be having this conversation. And so one, I call to to the attention of folks who are K through 12 and post secondary to really, you know, I plot the work that they do and also the understanding of how the important role that they play. The second thing for me even as you talked about the imposter syndrome or just be bringing it up. I also put out there with some of the scholars are talking about in terms of, you know, some of that. I hear that folks believe they have it, but I also have started to lean myself towards it, you know, we have to question some of the inequities and biases that are in the system. That really is more of the problem rather than me feeling like an imposter or that I don't have it. It is what's happening there. But then the third thing I want to say that brings me to where I want to go is, you write to a specific audience in terms of around really thinking about our young people, eighth graders and middle schoolers and the topics that you're writing about. Oh my God, around privilege and race and justice and actually your books are saying so much and leading. You said something to me that stood out, which was the first and the first conversation in some ways. And I absolutely agree after finished reading ghost boys is how your books were way ahead of where we were at as a society and thinking about some of the stuff that we think about. So I'm even like even before ghost boys but just even thinking about that piece.  I mean, it's, it's fascinating. And thank you. You know, I do think that I went through that period when I was younger that I needed my, I needed Carnegie in the environment to sort of set me on my two feet and give me the resources to assert myself. And I want to say to the K through 12 and college students all the enrichment programs that you do, they matter, you know, don't judge children by their, their, their numbers but just to see beyond. So, you have to keep doing that. I also want to add in that my biggest problem was money. And I don't know whether Carnegie Mellon has this but I remember times not having enough enough to eat and crying, and my grandmother having dressed to dress up and bring me cans of food, because even though I got financial aid and I can tell you a wonderful story about that. But the, my parents kept claiming me as an independent claiming me on their tax returns but they didn't give me any money. So I even went to work at Carnegie Mellon working in Scebo and I wasn't supposed to, as a dramatic program is all day long. So I think there are also ways in which that it really also was poverty that would have pushed me out that I needed just a little bit more help. I was getting but there's that is. But then I think once I found out that black people wrote books. I immediately started with voodoo dreams. My grandmother I believe was a hoodoo woman she taught me about spirituality and science and holistic healing. And I remember sitting in a history of science class at Carnegie and the professor saying that the only people that had done anything worth knowing about or caring about in science was the Western community. And I knew that couldn't be true but I didn't have the substance to back it up. So I went into the search of, you know, other stories and went in search of, why does voodoo have a stereotypical response in America, and what is the true African diaspora spiritual tradition, and writing that book, it grew me up because actually you can actually look at the Daily Pick a Union and find a newspaper with a newspaper says it's true. Maria vote walked on water. And by the time I finished that book. I went from being Joel Parker drew well to Joe Parker, you know, I am. I am. I was able to grow myself up that why couldn't a black woman have power. Why couldn't a black woman walk on water. Why couldn't a woman, you know, make this kind of information. And then from that I got interested from a parade magazine article in the Tulsa race massacre. So I'm the first person to write a fictional account about the Tulsa race massacre in 1998. And the book was planned would send me hate mail, and I'd see them in Oklahoma. I'd get a great review for the Tulsa world in Oklahoma, but the book editor would tell me at a conference. Oh no we can't publish that. I believe me it's not personal, and that literally the book disappeared because people want to believe that massacre didn't happen. And now the book has been reissued because 100 years later, you know the narrative is much more clear. People are now beginning to accept that it happened. So I think I was trying to gather a cultural framework for me to live in.  You know, I love people. But now I think I'm in my writing for you. I always knew that I wanted to write for you. But I didn't think I was good enough. And I thought kids deserve my very best. I put all those adult novels practice and trying to get ready. And I'm so glad I did because I take prejudice, privilege, bias, racism, I take real world problems. I take 911. I take colorism, and I find a way to tell it in a story that I can relate and begin to discuss that I'm not trying to patronize them. And this week I, they're well they're making a film of ghost boys, they're making a film of black brother black brother, Sesame Street, Apple TV. This is a series called Ghost Rider, and they're new season out. And the last two episodes are based on my book by you magic, and it shows this beautiful black African mermaid. Wow. And someone said I think this is the first time I've ever seen you know, an African mermaid, you know, in Hollywood, but the tail was that you know African mermaids, followed the ships of the enslaved, and wherever slaves went they remained in the water. Hey, your community is still here. So I think now, I'm not, yes, I'm not an imposter, though, I can tell you just last week, somebody thought I was a service girl, and little kids ask me all the time, Joe, do you do you have racism every darn week. And I've discovered that I can be dressed in several thousand dollars worth of clothes which I used to do when I was younger you know Joe Parker Rosie. Oh yeah. Can a white person would turn to me and say can you get us some more hot water for our tea, or I could be with a date at the Princeton club staying at the Princeton club, and we're getting off the elevator going to our room, and a white couple might say, say to me well can you clean my room Missing the social cues are being at the Arizona built more with my family, my son and my daughter, and having you know, a white couple passed me by, and the gentleman wanting to know when I was going to sing, which actually I don't know what I'm saying. But, but those things still present my day, though, is that they can still hurt. They can still make me cry. But that's why in terms of external appearance. I don't put much talk in that, because there's a phrase that no matter where you go, you still be black. I must tell you, no matter where I go.  I mean, it's not that I don't know these things but you know, I told you and I told people who are here how I felt about how I see you. And just, and, and it does sort of hurt to to know that that you've had that experience and even the last one.  It's like people stop and we got a lot of work to do. One of the things I want to turn to because is that we're having a conversation and I'm watching some questions come in. And I want to begin to invite other folks to ask some questions because some of the things that you've said. If you have people it looks like they're they're saying, tell us more about being, you know, growing up in Pittsburgh someone is asking about wanting to hear a little bit more about your upbringing here and what it made me think about you had told a story about even just your familiarity and not familiarity thinking about even, you know, August Wilson and whatever's coming to mind for you. A little bit more about that about in terms of your experience. I think I was meant to be a playwright. And if I was back in school now I'd probably study screenwriting, because dialogue is my thing. And then hadn't quite made his, his move, and I wasn't aware of it, although I wasn't Pittsburgh downtown at hotel at a writer's conference, and I was sitting with August Wilson, john Edgar Weidman, his cousin. Oh, I'm blanking out on his name but his cousin wrote a very well received first novel. And there's actually a photograph of his cousin who was a photographer for the Pittsburgh post gazette that they took of me. When I was in my bare lady all dressed up and I've got that picture. Oh, you have to show it to me. These, I will. I'll three three artistic gentlemen from Pittsburgh, from the Hill District, from Homewood, and feeling as though I had died and gone, gone to heaven. I don't know that they existed my folks for steel mill folks you know, and my grandmother, you know was the caretaker of all the children in her household. And when I left my grandmother, you know, my grandmother of course was devastated as was I, but my dashiki story was when I came back from California.  I had my sore. So think of the black panther Wakanda. That's what I look like. I had my love beads. It told me it's cold in Pittsburgh came back it was January. So I brought a sweater and grandma took me down to Mellon Bank right in central downtown, and she was dressed up like she was going to church she had and she had her and my grandma was also a part time Reverend. Okay, Reverend. And in those days the black men did the most risk life taking jobs you know really dangerous jobs in the steel mill, and they drove me to the bank and there was this middle age white man, who my grandmother saying she wanted to borrow $500 so she could buy me some winter clothes, and you have me go to Allegheny Community College while we were waiting to see if I got it to see a new sky kept looking at her and looking at me and looking at her. And he kept saying what's your collateral, what's your collateral. Well grandma didn't have any collateral. And my grandmother pointed to me, and she said, she's my collateral. And this white man middle age white man went, and I think the president can help me find him one day, said a smart girl. That's very good collateral. And I swore I jumped up and down and I thought to myself, this white man reads books, both teach empathy, and he said yes to us and we went right next door to learners and got me my winter coat and some pants and all that kind of stuff. And reading literature that in that sense of empathy became part of my life, but not only from what my grandmother told me, but from a demonstrable affection and support that Carnegie Mellon gave me, and I went through my trials trying to give her permission to speak my voice, trying to say, how come I don't know this history about black people. How come I wasn't in not just black people native people, Hispanic people. So now I write stories that include a multi ethnic community, Hispanic kids, Vietnamese kids, you know, other Chinese kids black kids white kids. So now I write stories where the affirmation is the allyship of all these groups of people coming together, like the people came together for Martin Luther King's marches that that is how we change the world. Gosh. There's there's two questions that's coming on one. There is a question. I think you sort of went there but there is a little bit I think those on want to hear a little bit more about. Maybe if you could talk about in terms of CMU. How familiar you were with CMU and did you feel in terms of when you came and you got here and you went to CMU. What did you know about CMU. Nothing, absolutely nothing.  I did not know that white people existed until I was five years old, and grandma took me on the trolley downtown. You know, Carnegie Mellon, that was like it could have been the moon, but because I had teachers in school would say, Oh, Joe should do this, Joe should do that. They hooked me up with the University of Pittsburgh. I do not remember Carnegie Mellon program at the time, but I went to the University of Pittsburgh for summer, and that was the first time that I was in among school elementary where we were all, you know, of different ethnicities, you know, and that it was like a completely new world and I met these kids who belong to different class groups. And then I knew that there were other children who knew other things who did other things than I did. So I would also argue, you know that, you know, Carnegie Mellon probably has a problem now, but coming to the campus it was so beautiful. It was so glorious. I was lucky that I had some black classmates and at least drama, but I do remember feeling very, very, very lonely. And one of the things that can also have happened is, you should not. There's also interracial prejudice, as well as intraracial prejudice. That's sometimes we don't give enough attention to intraracial prejudice, you know, and the combination of how they can echo off one another. And for some reasons, I was at times ostracized by some of the student student body, and some of that did have to do with the color of my skin, that I was a little bit too light to maybe be authentically black. And when I wrote you do dreams and went to another university, there was this one student who wanted to hit me with a baseball bat, because as a light woman there was no way that I could have written such a black text. And I think there were not an homogenous group. That's right. In which racism has taught the people who are oppressed. And I wrote about this in Season Moon Hurricane, that there actually is a vampire myth and African tradition that a lot of Africa was colonized by the British, the Dutch, the Portuguese. And so there were African vampire myths where the vampires were actually bureaucrats just as dressed as bureaucrats, and a lot of times look like firemen and had buckets of blood.   Racism does harm to the person experiencing it. That's right. To the person, you know, who, who is racist. And so that sense of in Black Brother, Black Brother, I write about all heritage is lit. And that actually came from personal experience. I'll share this. I married a white man at a time when it was just now legal in Maryland for there to be an interracial marriage. And my son Brad is six four Norwegian descent long blonde hair and now because he's old he looks like Gandalf. And he, he, he I married him because he was a literature major and a computer science major. And we bonded over stories, and he was my true love he was my tea cake. If any of you know that reference to Zora Neil Hurston, but when we married I cried, I would not let there be a celebration we went to the courthouse, because I was scared to death with the world was going to do to us as an interracial couple. And I had agreed to marry him but not have any kids. And he convinced me that I shouldn't patronize my kids. Okay, I won't patronize our kids. Well sure enough, we had a daughter who looks like her dad, white appearing, a son who looks like me, Brown appearing, and their life is totally different. And one of the things that Evan has experienced, including, you know, assaults by police would never in a day hurt her Kelly, and sometimes to I wonder and all of you can think about this, had I been a boy. Had I been very dark, would CMU and other places that I've been, would they have been as gracious to me then.  And I always think I'm taller than I am. When I saw pictures, I'm such a shrimp. You know, wow, wow. So, so now you've just opened the door for, for us to do this again, open the door for us to get to know each other. I even more I know you're going to be visiting here. We have about just about 90 seconds before I'm supposed to turn it over. I do, but I do. I do like for your last words like just given what you just ended with, you know, there's one, there's a question here and I think it's important to end with, you know, how do we tell healing stories about the kindness of people that helped with overcoming and justices without negating the real effects of consensus built prejudices. So basically, I know it's a big question, but in a short period of time, could you just sort of leave us with any thoughts or ideas that you might have about, you know, the, the, the, the, the power of storytelling and how it helps to heal. The storytelling is wonderful. I have always wanted to affirm that, you know, all people are connected by their, by their common humanity. And I'll give you a concrete example. I wrote ghost boys and it nearly killed me and I do urge all of you to read ghost boys. Very often, a black person will be murdered. Emmett Till will be murdered. And the rest of the stories about society getting itself right.  I found a way to give them agency that all spirits are alive. Yes, spirits, all of our spirits are too great not to continue to have a lasting impact. For me, the dead are elders, spiritual resources, and to feel that there is agency in telling stories about how the past can inform the present or a person to make them better. And then that way we see that all of life is as interconnectedness from before now and moving into the future. And those are the kinds of stories that I tell to heal. Thank you so much, Joel Parker Rhodes. You got to you have my bedroom ready because I'm coming to live with you for a few days. I'm coming to live with you but you're welcome to come and see me. Well, this is I'm so appreciative of your honesty, your candor, being very vulnerable, spending some time with me as an inaugural vice provost here at right. I'm loving my time here I'm loving the people here. And I'm so grateful to have met you and I can't wait to get to know you even more and CMU is definitely blessed that you have made this time your time in your life. That's, you know, been a part of your journey here so thank you again. And I just like to again thank everyone who made this possible and at this particular time, I am going to turn this over to Crystal Johnson Community Collection Processing Archivist who will close us out and share more about the library's work uplifting historically marginalized stories and voices who have shaped CMU. Thank you. Welcome, Crystal. Thanks, Wanda. As Keith mentioned in his opening statement. In the community collections processing archivist my primary role is to increase representation in the university archives. In 2020 the university libraries curated a virtual exhibit titled what we don't have confronting the absence of diversity and the university archives. And this exhibit my fellow archivist explored the gaps in our archive, and acknowledged that our collections are not a true reflection of CMU diverse community. The past year, the university archives has been building a framework towards a better archive. The conversation we heard tonight and Jules story of finding a sense of belonging by visiting the library illustrates why it's so important in life in like potentially life changing. It can be for students to be able to see themselves reflected on campus. We want any student to be able to walk into our reading room and be able to find themselves in the university's history. Roar, which stands for repairing our archival record is a community archives initiative dedicated to preserving the history of under engaged and underrepresented communities at CMU. It includes the voices of alumni students student organizations current and former staff faculty as well as programs and committees which have served these communities. This initiative is heavily influenced by community archives movement, which emphasizes a participatory community based approach that retains and amplifies the community's voice. We want to fill in the gaps in our collections, but we need to do so in a responsible way. We don't want anyone to feel as if their community history is being tokenized or viewed as a collectible. We're focusing first on having conversations and building relationships both on and off campus. We want you to know that we exist.  It's an integral part of telling CMU's history. I'd also like to mention a few of our current and upcoming projects. First, we're working to make collections from our backlog, ready for public use to collections which will be made accessible in the upcoming year are the papers of Dr. Gloria Hill and alumni Sanford Freeman. Edmund was a writer who graduated from the College of Fine Arts in 1949. He's best known for his 1965 coming of age novel totem pole totem pole is believed to have the first representation of a gay Jewish protagonist and American fiction. The late Dr. Gloria Hill was a beloved figure on campus. He was the former director of the Carnegie Carnegie Mellon action project also known as CMAP, as well as the assistant dean of Dietrich College of Humanities and Social Sciences. Secondly, we are also embarking on a few community based projects, one of which will include Dr. Hill and CMAP's legacy and build on the CMAP collection that we're home to. For those of you who are unfamiliar was founded in 1968 with the aim to increase the recruitment and retention of African American students at CMU. In 2006 it was renamed Carnegie Mellon Advising Resource Center or CMARC. The program expanded to serve Hispanic, Latinx and Native American students. These programs help students of color find community at CMU and provided resources that helped them succeed in their academic pursuits. I've been talking with former CMARC staff members about how we can build a fuller history of these programs which has left such a lasting impact on so many. We're in the very early stages, and we'll have more details in the new year about how you can get involved in future projects. We're also collaborating with our friends at the Center for Student Diversity and Inclusion on a project related to the history of LGBTQ plus life at CMU. For decades, members of the LGBTQ AI plus student organizations have been collecting and storing what are now historic materials in the cabinets of their community space SOHO. We had a pop up event in October to celebrate both LGBTQ plus history month and announce this important collection. We hope that this will be the first of many opportunities for community members and current students to engage with these materials and participate in community archiving workshops. In addition to all of those, we've increased our outreach. We were present during orientation, student activities fair, Scotty Saturday, and hosted an archives open house during Tartan Community Day. And we will continue doing more outreach in the next semester, so look for us around campus. If you'd like to learn more, discuss your community's history or get involved, please send me an email or visit our initiative website. The link will be posted in the chat. You can also drop in during our open hours on Thursdays between 10am and 4pm. Thank you. As Jules Professor said, yes. It's been a wonderful evening. Thank you Wanda and Jules for entertaining us, for educating us, for challenging us. This is an event that will stay with all of us for a long time and we're truly grateful for your observations and for sharing your deeply personal experiences. I'm grateful also to Crystal for sharing with us your important work in the University Archives. If anyone would like to learn more about Crystal's work, I think we've dropped links in the chat. But she also is available on Thursdays between 10am and 4pm if you'd like to drop into the University Archives in Hunt Library. At the University Libraries, we value innovation, accessibility, collaboration and service, and we strongly believe that diversity, equity and inclusion enhance these values. If you'd like to support our work, I encourage you to consider giving to our friends of the library fund. Your support strengthens our vital role as a convener of interdisciplinary activities and is a central, neutral, safe gathering place for the CMU community to have campus-wide discussions, such as the one we've had this evening. There are, of course, other ways to stay engaged with this important topic at the libraries. There are frequent book displays that explore various topics around diversity and inclusion. Our current display featuring some of Jule's work has been curated by Wanda's office, focusing on the power of representation. And if you can't make it into Hunt Library, our exhibitions are also made available on our website. We're also launching a book club kit that will be available from the Hunt Library circulation desk. You and your group of friends or colleagues can borrow those kits, which include multiple copies of a book and some discussion questions to help you dig a little deeper. One of our first kits features Jule's novel Ghost Boys, which was an instant bestseller and won more than 30 awards. So thank you for attending this evening. I'm so pleased you could join us. I hope that you will come to our future events. And the next one that we will be hosting is Fine and Rear Part 2, during which Curator of Special Collections, Dr. Sam Lemley, will share more about the library's plans to celebrate the 400th anniversary of Shakespeare's first folio in 2023. Without spoiling Sam's presentation too much, I'd like to ensure that you're all aware that we will be partnering with the Frick Museum in Pittsburgh to have two exhibitions, one here in Hunt Library, but a very special one at the Frick Museum running between April and October next year. Because we are holders of one of only 230 surviving copies of the first folio. And it's a tremendous opportunity for us to celebrate such a landmark work. And for us all to learn more about its history. So with that, again, thank you for being with us this evening. Thanks again to our speakers, and have a great evening.


guys this is gonna be a quick two minute video on using chat GPT so here I'm using it to create a think script very simple one but it can be very helpful so let's go ahead and create a think script code to calculate the let's say the moving app or the average volume alright so we're gonna say it's gonna create a chart label that displays the 50-day moving average for the volume alright so let's go ahead and see what GPT comes up with and here it is and that's pretty much it there is the length there is the average volume function and there's the label just like we saw in the previous video now you can also do this over in the mobile version and here it is this time we're creating the relative volume script but in this case in this case we told it to create the script and it actually plotted it see see at the bottom we have where it plots we haven't gone to that yet so let's go ahead and tell it this time to create the relative volume script but instead of you know letting it decide what to do with it we'll go ahead and tell it that I actually want it as a column the way we created it in that previous think script video so see here displaying it as a column and here is what it comes up with there it is pretty much what we got alright so I hope you enjoyed the short video and I look forward to seeing you guys in the next few videos and remember if you have any idea let me know in the comments below have a good one


Okay, I'm gonna get started. Hello, my name is Brad Giesemann. Welcome to Hacking and Hardening Kubernetes by Example. If you want to grab the slide link here, it's the first one. And then the GitHub repo that all the demos are being run out of, it's in a separate window for me. You can also run those two in front of you if you're far back and not able to see the text if it's going by a little bit too quickly. I have to apologize that I have to go through it so quickly because I have so much to show and I want to show it to you. So this is more of an index, so to speak. You can go back and then dive in deep as you're at your leisure. A little bit about me. Formerly a penetration tester consultant last five or six years using the cloud almost exclusively designing ethical hacking simulations or capture the flag exercises. And in the past year, we've been running, my former company, we've been running capture the flag exercises on top of Kubernetes inside AWS. That sounds crazy. It is a little bit, but we worked very hard to make sure that was a success. In the past few months, I spent a lot of time looking at as many clusters as I could researching Kubernetes security and policy. And so that body of research that work is what I want to share with you today. So for the past five months, I've installed a few clusters. I've dreamt that I was installing a cluster while I was asleep. It was very surreal. By show of hands, who has a cluster that's listed here or an installer or uses an installer that's listed here with one of those versions are similar to that. Okay, fair number of you. Welcome. How many of you run your own distro? You rolled your own. Brave souls. Awesome.  The biggest takeaway from a security perspective for me is looking at all of these installation mechanisms that the thing that stood out was a malicious user with a shell. By default, I'm saying default on purpose, can very possibly and almost very likely exaltrate source code keys, tokens, credentials, elevate their privilege inside the cluster from a non-privileged state to a privileged state inside of Kubernetes, which often then leads to root access on the underlying nodes. And I think bullet point number four is probably the most interesting or something that hasn't been talked about as much, really expand the blast radius to your entire cloud account in some situations. So I hope to be able to get to that quick enough to be able to cover that in its entirety. The goals of this talk, I want to raise awareness of those high-risk attacks in as many installers or distributions as possible so that everyone has that knowledge. Demonstrate the attacks live. I'm not brave enough to type live, and I don't type quickly enough live, so these are recorded typing sessions, which then that offers you the ability to have at home and look at and examine. Finally, I want to provide some hardening methods for those specific attacks and then additional guidance that goes a few more steps beyond that. So like Morpheus, I'm beginning to believe that high system complexity means for users who are new to a project, that getting it to work from an operator's perspective, getting it to work is hard enough. It's such a wide range of new terminology tools and mechanisms that most people use the defaults the first time through. Look, they probably know better than me, I'm just going to accept the defaults, let's go, see how it works. But defaults tend to have inertia, so defaults in use early tend to stay in use, and systems hardened late tend to break. And that's kind of, as I was going through all the clusters, that was what I was doing, so I was running into it left and right. So my belief is that having default values be secure early on in terms of a project or how you're distributing your project in source code has positive downstream effects to the community. And when something like Kubernetes literally blows up, has widespread adoption, that inertia is big and it's real. And what that kind of leads us into is a, I call it a security capability gap, I struggle with the name for what this is, but basically the community at large is somewhat behind the major dot releases as they're coming out, so maybe you're between 1.5 and 1.7. Most mortals can't literally deploy overnight a brand new Kubernetes release. But most installers and containers of service offerings are keeping up, right? But the trick is that security capabilities and features are coming in newer releases. So if you're still on 1.5 and 1.6, RBAC is really rough for you. But if you're in 1.7 and 1.8, it's been baked in, it's been battle tested and things like that. So it's tough because you have to keep up with those ever fast moving releases. And so it's up to you to add additional security hardening. If you're on 1.6 and 1.7, don't despair, it just needs a lot of elbow grease. And the things I want to talk about today are not extreme in depth esoteric attacks, kernel level exploits and things. I'm talking about low hanging fruit. I believe I found enough of it to share with you and that's enough for a start, right? Want to raise the bar, just doing the basic image safety, RBAC, network isolation, just doing those things and enforcing those basic controls that are already there, already existing inside clusters. So when you go to harden some clusters, what are some of the challenges? Well, a lot of folks like to use Dissistig or CIS benchmarks as a, what do you say, what's the security posture of my cluster? Well, at the operating system level, those specific benchmarks don't take into account the workload that's running on them. They say, you know, is that the password and that's the group, are those properly set with permissions but it doesn't know anything about Kubernetes. And conversely, if you're doing a CIS Kubernetes benchmark, it's not taking into consideration the OS but it's not also taking into consideration how the installer places things and where it puts it and where it grabs it from from the cloud provider. So basically, properly harding your Kubernetes cluster is highly dependent on your environment, your add-ons, your plugins, and the defaults are very often not enough. There's a lot of knobs you have to tweak and we're going to go through some of those. Something I like to call attack driven harding. This is just how I think. It's been built into me as a pen tester. Every time I look at a system, I think this way and I try to reason about a system in this way in terms of its security posture. And the best way I can summarize it is and how I think, I think in progressive steps. I say, from where I am, what can I see do or access next? I pick one of the most plausible methods and I say, all right, assume that happened. All right, now what does it look like? What can I see do or access next? And I repeat until it's game over, until the worst data's got and extracted. And then I work backwards and I harden as I go further away. It's basically quick and dirty attack modeling. So everybody here today can take that persona of the external attacker. If you're looking at a cluster, typically these are the methods you're thinking of right off the bat. Are you going to be able to get SSH access to the nodes? Maybe, not likely. Go through the API server, maybe, not likely. You don't have credentials for either of those. But what about getting a shell on a container inside the cluster? That's where it gets interesting.  That's hit or miss. Not all apps are extremely vulnerable with a remote code execution. Tricking an admin into running a compromised container. That's interesting. Or compromising a project developer. Compromise their GitHub keys or their Docker registry keys and modifying the project images and binaries. Throughout this research, I did find somebody's credentials in a Git commit by accident. I was just looking at code and I found it and they were, after I reported it to them, they did say it was indeed their company's ability to push to quay. So that is a real deal. So protect your keys. So which is easier? I'm going to pick on number two today, teaching an admin. I've written a couple blog posts, but I've read thousands. And I found something that's kind of a pattern. If you say, here's something really complicated. Use my custom images. Hey, here's my Docker file. Everything's on the up and up. In those instructions is what? A kube control create from that URL. Just slam all these pods and services in and then figure it out and see what happens. I like to think kube control create from URL is the new curl into bash. Because it really is. And it's often worse because now it's distributed across thousands of nodes. I said this is about hacking and hardening. Let's make with hacking. For the rest of the attack structure, this is my 3D diagram of a sacrificial cluster. And the lower left, you have the master node. And the upper right, you have two workers. Very straightforward, very simplistic. We've got a couple of pods running, not all are represented here, but just the ones we care about in this case. And we have the metadata API represented as that yellow block up there. So, my handy-dandy little attacker icon here, if he's able to exploit the vulnerable app in the default namespace, if they get a shell, can they? Install custom tools. And by doing so, prove internet access, which is something that penetration testers always love to have try to pull down your tool sets. Can I install curl, netcat, and that? Can I pull down the kube control binary and put it into a place and run it? That's always interesting. Another look at things. It's not common anymore, but in <IP_ADDRESS>, if you're running <IP_ADDRESS>, a lot of the installers back then, or if you rolled your own, you might still have the insecure bind address on your API server. That's a big no-no. Because there's no authentication or authorization on this. This is a direct pass the cluster admin. So notice that little red triangle. That means a bad day whenever you see a red triangle. Whenever you're doing a penetration test and you break into that first system, the first thing you do is say, what does the world look like? I have no idea. Where am I going? I'm running scanning tools. I'm just throwing packets everywhere. Well, in a distributed system where everything's based on APIs, that enumeration is just a couple curl commands now. If I hit C advisor, heapster, kubelet, Prometheus's node exporter, kube state metrics, any of those, and it's just like, tell me about yourself. Well, here's everything about myself and what they're named and where they're running and what their pod hashes are. Everything is right there. So that leads me to my first demo. Because we have kube control, because we have that access, we can list the nodes and we can see the IP address of one of the nodes. And C advisor runs on <IP_ADDRESS>. And you hit the metrics endpoint. C advisor will happily tell you everything about what's running on this system, including pod names, which are always randomized, the namespace they're in and the container names and the versions, the shahashes, basically everything that I'm running. There's my Redis. We'll get to that guy later. This one I think is fairly well known, but it's incredibly important. The default service count token, if that's located in this directory, it's auto-mounted in a lot of clusters specifically before RBAC. This is a really big deal. If you have RBAC enabled, we'll get to some of that. But if you can run kube CTL, kube control, sorry. I was corrected this morning in the keynote. Kube control, you can get pods, get secrets, and your cluster admin. Again, Red Triangle, bad day. So we can install some tools, download the kube control binary, validate, we can hit the API. Yes, we have the service account token mounted. We can get pods, list all the secrets, look for the good ones, and dump those contents. So four, five curl commands, and we've escalated. Next thing we want to look at, the Kubernetes dashboard. Raise your hands if you run the Kubernetes dashboard. Awesome. Are you running one seven or higher version of the dashboard? Okay. All right. So as you know, there's no authentication on it. It needs protection. All right. So if you're in this vulnerable app pod here, most often you can just hit it by its service name. You don't even need to know what the IP address is, right? Well, that's kind of tough. How do I hit that? It's a curl command. It's a big dashboard. We can forward a port over SSH. That's really two commands away. So yes, we're inside Kubernetes. Let me get the service. Yep. The dashboard's there. Let me get the IP address by pinging it. That's a cheap way to do it without having DIG installed. And then we're going to SSH out to my bad IP. That's my attacking system. Say remote port 8000. Funnel it on down into the dashboard.  What about tampering with other services inside the cluster? As you can see, there's a vote and a redis, the Azure vote front and Azure vote back application. It's a very simple Python app with a redis back. You can vote for cats or dogs, right? Hack the vote. We're going to tamper with it. I grew up with cats, so I'm going to pick on cats today. So we're going to get the service. Azure vote back. Get its IP. Yep. Port 6379 is open. Let's install the redis CLI. Can we connect to it? Yep. We can. Dump the keys. I like cats being 1,000. Let's set cats to 1,000, and let's go hit that web front page. I apologize it's in curl, but you'll see it at the very bottom there. Cats is 1,000. Dogs is 6, right? Take that and extrapolate that to any unauthenticated service inside the back end of your cluster, right? Redis I just picked on because it's simple and straightforward to demonstrate. Here we get a little bit more interesting. The kubelet exploit. How many of you have heard of this attack method? The kubelet exploit. Well, it's basically not an exploit. That's why it's kind of an air quotes. The kubelet API allows this. And in clusters, without certain settings on the kubelet, will allow anybody to connect to this endpoint and exec into containers, ask for logs, and do other nefarious things. So what we're going to do is we're going to ask the kubelet to run a command in a given container on it. So by one curl command, we can say, hey, I want you to exec, you know, list this directory inside that pod right there running on that node. So we get the node IPs right here. Port 1-0-2-5-0. That's the read-write kubelet API port. 1-0-2-5-5 is the read-only metrics port, right? When we hit the method running pods, we're going to cat it out into a file so that it's easier to look at. It's a nice JSON object. Again, very much like CAdvisor, it's everything that I'm running. On the kubelet, this is all I know and what I'm running, complete with the hashes, the namespace, the pod name, and the container name, which is important for the next command. You got the azure vote front, that's the one we're going to pick on. So we're going to look at the web directory of the azure vote front app. Run is the action, default is the namespace, azure vote front, numbers, that's the pod name, and then the container name, and you just say, hey, run the command, list this root directory. App looks like an interesting directory. Main.py looks interesting. We've just extracted the source code for this super sensitive application. Okay. Accessing SCD service directly. Most clusters don't expose SCD to the workers, but some install a separate SCD instance to support Calico or network policy back-ending. And in some cases, that's also exposed with no TLS or authentication or authorization. So in this case, you may be able to defeat the system by storing your network policies, saying, you know what, if there are network policies, but you can hit this at CDN point, you can go in there and say, Calico, forget about all your network policies, and Calico will happily remove all the network policies from your nodes in your cluster. This is pretty rare, but I'll get to the frequency of this one. Now, any of those methods that I showed about getting a kubelet or a service account token will let you possibly schedule a pod that mounts the host file system, add your own SSH key, and then SSH into it. Now, we're getting into the multi-step parts here, but what we're going to do is we're going to get the external, sorry, we're going to get the node name as it's represented inside of Kubernetes, the external IP address of that node so we can SSH into it later, create a very simple pod specification. I pick on engine X because it's based on Debian, but we make sure it's privileged as true and we mount the root file path. Here's what it looks like with the node selector in there so they get scheduled on that one single node. We run it, we exec into it, we trute into the slash root file system bit and now we're on the host as root, add our own SSH key, back on out, and then SSH directly in. So if you're root and you're able to run docker containers under the hood that Kubernetes doesn't know about, run backdoors and solve things, it's a pretty bad day. The last classification of tax I want to talk about is accessing the metadata API. Who's heard of the 169.254, 169.254? We know what that is. One of the things that it does is gives instance data about itself, what region it's in, it's bootstrapping information. That often in some of these installer cases has sensitive S3 paths or CUBE ADM join tokens. So right then and there, that's a bad day, but also most of these cluster installations will provide IAM instance roles attached to the workers and the masters with permissions. Also available via that metadata API are those AWS keys. They rotate every few hours, but they're just a curl command away. So let's curl those and get those. From that vulnerable pod that we talked about, we run one command and we get keys that are valid for a couple hours. We export those into our local shell and our attacking system, right? And then we have the permissions that are available to those keys. So describe instances, list me all the instances in your entire account, not just your cluster, everything in that AWS account. And describe the instance attribute called user data on every single instance in your entire cloud account. How many of you have sensitive things in your user data in things that are not Kubernetes maybe, possibly? That's why this blast radius is pretty bad, because you might not compromise your Kubernetes cluster, but that web server there that bootstraps that has a GitHub key or something in it that might be delivered via user data, you can reach over and go grab that. So that's a bad day for the other administrators. When I talk about IAM permissions, the masters and the workers typically have something that looks like this. Describe star for the worker. Masters have EC2 star. ECR ability to pull images from AWS ECR and some S3 capabilities, but we really want that EC2 star, don't we? That means any AWS EC2 command is available to us. So how would we get that? We need to make sure that curl originates from the master. So there's a couple of ways of doing it, compromising existing pod running on the master. It's kind of tough. Or using one of those two issues that we just talked about, if you find a service account token, just ask in the API server, or just ask the cooblet running on the master to run a command for you inside of a pod. So it looks like this, basically wrapping a curl command this way or this way. Notice how close they are. Basically the same thing, just asking somebody different to do it. And the final example of why this is a bad day, if you have EC2 star, you can create a new VPC, create a new security group, create a new SSH key, create a new instance, and snapshot every volume from every single instance in your entire cloud account, and then go ahead and mount it on that instance. So that can be automated, as you can imagine, within five or 10 minutes. It's a pretty bad day. So if you're also on the master, you then might be able to, in some cases, based on the installation by default, list everything in AWS S3. Who stores logs and sends it to backups in S3? It's a bad day. So attacks 9 and 10, I'm switching gears. I'm now talking about GKE and GCE. And GKE specifically, there's an attribute, much like the user data endpoint on the API. There's an attribute called kubnv, and that's what the kublet uses to bootstrap itself. It gets its keys from it. That's often reachable directly. Just clicked. There we go. So here's that listing. Part of the security features, though, you have to pass a header into Google's API to make sure that you're doing it, not through a server-side request forgery. But configureSH looks interesting, kubnv looks interesting, user data looks interesting. So we can go poke at those. This reference is the kubnv. So right there, you can see there's a lot of good stuff. We know what the release is. We know where it's getting things from. We know what the IPs of the master are. But we can see that here's the kublits information on where it gets its keys, cert, and cipm. Right? This wall of text is what I call the one-shot. So if you get a shell on the container inside of GKE, you can become the kublet in this one-shot, awesome bash hunk of junk here. Pull down a kube control, grab the kubnv from the metadata API, strip out the parts, base64 to code them into the kublits authentication tokens, and then run kube control to list all the pods and all the namespaces. So one of the things of note is you want to probably get the secrets, right? Well, the kublet doesn't have the ability to go list all the secrets, but it can pull a secret if it knows its name. Well, the best way to get that is to output all of the get pods in YAML, or a pod that you know of specifically. And I did the dashboard here because I know it's got the cluster admin token. You can say, hey, dump the pod spec in YAML, and it will tell you the mounted secret by its name. So now you know what that is. You can go get that secret directly. In this case, it's the default service count token in the kube system namespace. So what we're going to do with that is the same thing. I'm actually going to skip this part for the sake of speed. Mount host file system at SSH and SSHN. The second method through the GKE and GC metadata API is just like in EC2, assigning permissions to instances. GKE does the same thing. They give you an IAM token, and they give you instance scopes. And that IAM token lets you talk to the Google API, compute API, and run actions on things inside the scope of that project. And one of those things that you can do is enumerate all the instances, of course, but you can also use this really handy-dandy API method called add SSH key. So if you have these privileges and you have this token, you're able to be on worker one, call for the token, go hit the API saying, hey, add my SSH key to worker two. And Google will happily do that if you're authenticated. And then you can SSH into worker two, or anything inside that scope of that project. So if you're running multiple clusters, that means any of those nodes in all the clusters in that same project. So we're going to get the external IP, so we know what to SSH into when we're done. We're going to basically list the instances in the project. Okay. We're going to page through it a little bit, just so you can see how much information is here. A lot of good stuff. The addresses, external NATs, things. The user data, the QBNV, for all the instances inside the project. You're doing an AWS EC2 describe instances, that equivalent inside of Google as well. Okay. So I'm going to go ahead and do that same thing, but describe instance. I'm going to see everything about this one node, so I can get its fingerprint, which is needed for this API column about the form. And forgive me, I use curl and bash to keep it simple. But you don't need to download the extra tools. There's no malware running here. It's all curl and bash and such. So what we do is we make a post body with that fingerprint that we just pulled. Add my SSH key, as you can see, the public key. We're going to post it to that API. I'm going to show you what it looks like rendered. That's what the final post body looks like. Here, Google, go add me to worker 2. Okay. Happily does it. And we're root on that second node. Again, a bad day. Okay. RPS. How prevalent are these issues? This is what compelled me to do this talk. And I want to stress something. This is not the entire security posture of every one of these clusters. This is a narrow band of these items that I've identified here. It doesn't say anything about the rest of them. These specific versions, the ones that I tested, note those versions. I started testing in August and September. We'll get to what the latest releases look like. So it's prevalent, right? You'd admit that it's not uncommon. So don't despair. We can do it. We have the technology. Attack 7 through 10. If you're running an AWS, I recommend what's called a metadata proxy, something that makes sure that when you go to 169, 254, that you're allowed as a pod. Kube to IM or KIM both worked in my testing to make sure that an AWS you're taken care of. Excuse me. The GCE metadata proxy and these steps, and I apologize the word these steps is actually masking Google's GKE hardening blog post that was released very recently. That is an incredibly important link. I apologize that was a late addition. That is really useful for blocking those attacks that I just showed, right? And if you're running network policy on 1.8, that is also a valid method egress blocking. And if you're running older versions of Kubernetes like I was and you're using Calico, you can use Calico CTL under the hood to get that same effectiveness. It's not through the Kubernetes API, but you can do it. Protect the KubeLit. Authorization mode webhook. If you don't see that, that's your KubeLit is probably allowing that KubeLit exploit bit. Isolating workloads. Remember that hack the vote there, a very simple network policy literally stops out in this tracks. You say, every pod that has the label azure vote back, make sure that it only gets ingress from azure vote front. Excuse me. This is almost 99% perfect drop in if you're running the dashboard and you have network policy ingress support. Drop this in and it will protect your dashboard. Excuse me. It's a bit of a trick. We have the pod selector that says the Kubernetes dashboard only, but there's no rules. So by default, that means a default deny. This does not block Kube control proxy. That works through the API server. This is blocking from pods, which have no business talking to the dashboard. Restricting the default service account token node and RBAC and node restriction. And I want to stress something. You have to exact in the pods and verify this. It's very easy to miss this or do this incorrectly if you're messing with RBAC. And monitor all RBAC audit failures. You either have a misconfiguration of your app or somebody's attacking you and they're failing. And I'm happy to say in 1.8 and above supporting egress natively, this policy works in your clusters as a really nice default deny platform. Apply this to every single one of your namespaces, which says ingress and egress, nothing is allowed. Except Kube DNS lookups to start. Put this down as a cluster administrator and then deploy your network policy for your workloads with your workload lifecycle. So when you're deploying Azure Vote front and back, apply the network policy that allows those two to work together at that time. Excuse me. And I'm happy to say that throughout these last five months I've worked with every single one of these projects directly disclosing the issues that I found. A lot of cases they were already in progress, already in flight fixing them. But with newer releases and Kubernetes 1.8 and a little bit of elbow grease, we can look like this. We can literally wipe out this classification of vulnerabilities for good and make the infrastructure nice and boring. Two tools I want to tell you about. Kube ATF. It was a tool that I wrote to help automate the creation and validation and destruction of all these clusters in a sane way. Because I spun them up every day for two hours and threw them away, kept going through all of them. Excuse me. And Heptio Sonobooey. I wrote a plugin, it was basically a proof of concept. There's so much more you can do with this. I currently run a CIS benchmark using Aqua Security's Kube Bench. So by deploying this plugin into Sonobooey, we can continually scan our nodes for posture assessment in a very sane way. So even more security hardening tips. This is where it goes above that line on that apple tree I showed you. This is where it gets a little bit more advanced. Let's assume that you've all done all the things that I just suggested. Here's what you want to look for. You want to verify that all your settings are properly enforced. I can't tell you how many times I thought I hardened something and I go validate it and I didn't do it just correctly. I didn't get that labeled just right, it's important that you validate those. There are other features in every dot release. Audit all the levels that you can, the OS, the runtime and Kubernetes, I like the CIS benchmarks. Log everything outside the cluster. That's important. Practice safe image security. There's all sorts of good talks and blog posts about this and tools that help with that. I already covered the Kubernetes components a bit, but the network security policy bit is incredibly important now that we have Ingress and egress, use that to your advantage. You can mask a lot of attacks by just not having network access. Protect your workloads by default by saying no Ingress and egress and then apply what you're allowed to. It's white listing, not block listing. I added this the other day, considering a service mesh, there's a lot of benefits other than all the things that it does for your application and visibility and mutual TLS, but it makes your workloads more isolated when they talk to each other, just by default and how it works. I think some folks have talked about this before, but namespaces.pretendant is really good when you combine it with that default deny policy set. If you have microservice here, microservice here, microservice here, they're all default denies. By default, microservices can't talk to each other until you allow them. You can be explicit. This is something we learned from the capture the flag exercise. Make sure that CPU and RAM limits and I know disk and network are somewhere down the line to prevent malicious actors from just filling the disk or consuming all sorts of RAM with all sorts of tools. Something that people don't talk about which I think is kind of interesting is on your pod specs, if you're running a pod that has no business talking to the API server, in your pod spec, don't mount the service count token. You don't need it, don't put it there, even if it's got no permissions, right? Defense and death. And use a pod security policy to enforce container restrictions and protect the node. That's something that's going to mature over the next few releases. And shout out to some of the vendors that I talked to. This is kind of an important note, the container where malicious activity and behavioral detection capabilities. That's incredibly important for stopping the initial attack right where it started. It's at an assist call level. Shell cannot happen. A curl cannot be downloaded. It cannot be exact, et cetera. You stop it, it just tracks right down there. Number three in the miscellaneous security bit, separate cloud accounts, projects or resources groups for different workloads or different clusters. I think a one-to-one mapping is safe for now. It's just too... There's too many ways to hop across. And don't run dev and test workloads and clusters at the same time as production or in the same place as production. Again, because of so much opportunity for crossover. And depending on your regulatory requirements, separate node pools for separate workloads using annotations to make sure sensitive stuff happens here. Non-sensitive stuff happens over here. Here's some of the tools that I came across that I found of note that you might want to take advantage of when you're looking at auditing. The CIS benchmark has been updated from 1.8. That's a great resource. CUBE Bench implements it nicely. Very straightforward to run. The CIS, OS and runtime hardening, stuff from DevSec and Ansible hardening from Major Hayden and the other folks from OpenStack are really good at making sure the underlying postures on your systems are great. And then CUBE Audit, which I'm looking forward to. I think that's the next talk. And Sonabui. I think there's a lot of room for growth here in this space. Notable security features in 1.8. Network policy and pod security policy, whitelisting. The egress is huge. Volume mount whitelisting prevents a lot of those node access bits that I was just showing you. So in closing, as a community we're all responsible for that safety and security applications that power our world. Let's make that foundation secure by default and incredibly boring. Thank you. Okay, first I want to say thanks to all those folks that I listed on the left there on the right. Yes, some of those installers implement CUBE ADM, CUBE Admin. It's the join token. You've got to protect that. And then there's a lot of good stuff that happens. Yes, yes. Certificate rotation expiration. Yes, sir. The questions, what's the number one? All of the above. But the first thing is enabling RBAC. A huge classification of those things don't happen in a properly configured RBAC enabled cluster. And then the rest still you have to do. Because notice how all the things that I was doing required no special tools. It was just access that you have. So combining that with network policy just, it shuts off everything to start. You might have a vulnerable cubelet, the cubelet exploit, but if you have an egress policy you can stop that network access assuming you put it under your namespace. So you can mitigate and work around without having to fix the underlying things with some clever policies. I knew that question would happen. Did I look at OpenShift? The answer is yes. I wanted to focus on vanilla Kubernetes because as you know OpenShift is a slightly opinionated distribution of Kubernetes. The only thing that applies to OpenShift from the things that I talked about by default is the metadata API. However, they don't put anything sensitive in user data but I am credentials associated with those workers by default that you would get that. If you go ahead and do that, then that's available and they'll need the metadata proxies but that's the only thing. I hesitated in lumping it in there because it's such a different beast compared to how all these were lined up. It's a little bit of an unfair comparison but I would highly recommend you look at OpenShift in terms of at least a reference point for their security model. I have no horse in the race. I'll be in the hallway if you have any questions. Thanks so much.


Hey, what's up guys? This is John Hammond, and I'm looking at Bandit from Over the Wire, which is weird because we pretty much finished that. But as I found out yesterday while I was doing a live stream for the first time, that was pretty crazy cool. A few people noted and let me know that there are new levels that have just been added to Bandit. So they're all Git-oriented and I want to showcase them because we were able to run through them. Myself, another individual that we opened up a Discord server that I'll give you some more information at later in the video. But I want to showcase these challenges because they're pretty neat. So we've got to review back to where we were at level 26. If I go ahead and connect to it, remember this would just print out a certain banner because there is no real shell for this user Bandit 26, but they give you this thing that we would actually go through because it's being ran through the more command as we saw in another script there. So if I actually just use Terminator to shrink the size of this, we can get eventually that banner to be paginated or buffered so we can enter Vim with the V key. And then how we done before, we can like edit a file, right? So we have wanted to use colon and then each add another file. We could go into etc. Bandit pass and we would want Bandit 27, but it wouldn't let us read that. So as we had done before, we were able to read etc. Bandits pass Bandit 26 and that works just fine for us. But that's the password that we have to this current level, right? Like if I were to check that out, that's what we already know. So what else can we do here? Well, we can get a full on shell. So if I were to run shell, that doesn't do anything for us. It puts us back in whatever Vim editor that we were in because it looks like it's still trying to run the more command because remember that is the shell for this user. So we have to set the shell to something else. And we can do that in Vim, right? We can set shell to equal forward slash bin forward slash bash. Now when we run it shell, cool, we've got a prompt and we are in that bandit user. So if I LS check out what we have here, we have text dot text, which is the big banner that we saw. We also have this other file that looks like a set UID binary. Yep. Bandit 27 hyphen do owned by Bandit 27. So if we just run that, it looks like it will do whatever we want. It can just do ID. Perfect. Let's get the password. Let's cat out, etc. Bandits underscore pass bandit 27. And there we go. Now we've got the password for Bandit 27. Let's take note of that. Bandit 27. Cool. I've got it set up here because I've been trying to run through these already. And now let's break out of the shell. They have to quit a few good times. But now we can switch over to Bandit 27. Once we are connected, we can LS, but there's nothing here. So I want to check out the prompt for this on the website. It says get a shell for this 26 that we did check out 27 get repository at this location, which we can clone. And it's using the exact same password as the bandit 27 user. Okay, so let's make a home for us to work in in temporary directory. I'm gonna have to use a couple of these because I've already been working. So it doesn't matter where you put yourself, but as long as you have a place to work. And let's get clone this guy, paste that in. Yes, that's fine. We will need to know the password. So let's check out that for Bandit 27. I'll throw that in my clipboard, paste it in. Cool. And now we have a new folder here called repo. So let's change directory into repo. LS, there's a read me file, which we can cat out. And okay, it just straight up gives us the password for the next level. Pretty, pretty easy, pretty simple. I guess that one was just trying to explain how we can get clone. So if you haven't used get before, I can do a whole another like video series on this stuff if I particularly if you need me to. But it's awesome. And it is a free and open source control system version control system that will essentially allow you to like keep track of the code that you write, no matter what version you're on, it'll keep track of your history, the changes that you make, et cetera, et cetera. So it's also kind of houses your application puts it in a repository, essentially a special folder that can do lots of cool things. And you can keep track of other different renditions of it between a different branch. There are all the changes can be noted with a specific message like a commit message, et cetera, et cetera. So let's take keep track of Bandit 28. Perfect. We've got that. And once we disconnect, we can go try the next one, Bandit level 28. Looks like it's just a different name here, but that's just fine. We'll switch over to Bandit 8 now. Once we are connected, still no repository. So let's make a new directory for us to work in. Get clone. Yes. Let's cat out. We're on Bandit 28 now. Yep. Throw that into our clipboard. And now we have that new repository that we can check out. And let's see what that file is. Okay, looks like it's trying to hide out the password for the next level. So let's do what we can do in Git and just get a log of the previous changes. Git log. We can see, oh, okay, there was the initial commits. It says admissing data. That initial commit is noted by that Git SHA-1 identifier, just that hash. Same thing with admissing data, fixed info leak. So this must be the most recent message that we are on, right? So if we want to check out the missing data, maybe that has the password. So if we wanted to, we can take that SHA-1 identifier and Git show that one. Okay, cool. So you can see it removed the minus minus in red here. It removed the password to be determined it used to have. And it added this password here. So that must be the password for Bandit 29. Sweet, let's take note of that, Bandit 29. We're just cruising through these because it gets pretty neat, pretty cool, pretty simple. Let's call it quits for now. Well, let's just verify that that is the password for Bandit 29 because I don't want this video to get too, too long. I do still want to give a little shout out to my supporters, the people that love me on Patreon. And I want to point out the Discord server, which is super duper cool. It is slowly growing. We have a decent amount of people that are usually all online and some offline individuals, but live overflow came and joined. So totally a shout out to him. He's super awesome. There's been a lot of really cool conversations, a lot about hack the box over the wire, some other war games and like practice stuff between Pico CTF, et cetera, other ideas for the YouTube channel. So if you want a small community of programmers and hackers, like come hang out, I'll put the link in the description, but it's awesome. Totally just jam with us. That was a product of the live stream that I was doing. And I hope to do more live streams in the future. I might try and do them every weekend or every other weekend. So, but it also will just come at random. If you want to know more or have more conversations with me about what's happening when, like hang out on the Discord. It'll be really, really cool. All right, some love to the people that love me on Patreon. I cannot say thank you enough. And I do this every time at the end of every video because it's the least that I can do. Thank you so, so much for your support and your donations. It's just incredible and surreal to me to see that there are some people willing to, like take out their credit card and, you know, go on this, don't go on this journey with me, do this adventure through the internet, through YouTube, with me. So I'm grateful for that. Hey, $1 a month on Patreon will give you this special shout out at the end of every single video. $5 a month on Patreon will give you early access to all of my videos that I release on YouTube. And we might do some new stuff now that we've got the Discord server and see what other, other stuff we can do for other cool incentives. But hey, if you did like this video, please do press that like button. Maybe leave me a comment if you're willing. Maybe subscribe. I would appreciate it. If you wanna check me out on Discord, certainly can, link in the description. My Patreon account, please support me for a little bit. And I've got a new website, www.johnhamman.org, that hopefully it'll fill with content soon. But it's kind of in the works right now. So, sweet, see you soon, next video. We'll finish up Bandit on these new levels.