to setting up. There went recording in progress now, a beginner's guide to setting up a basic website content planning. So just so quick thing. Hi, I'm Sarah Snow. I'm a former teacher that could come out today. This is kind of a writing exercise today to help you get started with confidence on your, you know, either your first website or to help other beginners with whom you may work if you have clients. So yeah, you will possibly hear some barking in the background or whistling. We have dogs and birds here. I love spicy food learning languages, studying American sign language, Spanish and French, and making lots of websites. I'm a training team contributor slash man scientists sponsored by Automatic today is one of my mad science experiments. It's basically to work on content of a website. And we're going to talk about why that isn't just a quick second, but just some quick expectations. If you are brand new to the space, of course, stay courteous. The golden rule here is respect yourself and everyone in this room. We are brainstorming together today and a little bit separately. And this only really works with kind deliberate participation. So definitely try and find your chat box today. And then oftentimes there are no wrong ideas throughout, you know, whatever comes to mind so long as it is, of course, courteous and respectful. So this is an interactive experiment. We're going to see how it goes together. And this is a recorded space you are welcome to view this later. Oh, goodness. Let's copy this from one of mine that already had the animations on it. So first off, what kind of website are you hoping to create? I'm really curious about this always just because it helps us to kind of generate ideas with you and things like that. And this could be if you're brand new beginner, like, oh, I want to make a website for my business, like, I don't know, one of my friends is starting up like a dance business. Or if you work with clients, if there is a website that you're like, okay, like I think that they're going to be wanting this thing, what type of website are you, are you making today? What are we planning for today? Someone is considering taking over a site that already exists and improving it. Oh, that's a great idea. Overhauls are really important. And this is definitely something that can help you to overhaul the content of your website. Let's see a WordPress site for an NGO, a community website, oh, like a social network. Oh, how cool is that? Oh, fascinating. Okay, that one's really complex. That was more about other people writing content onto the platform, I suppose, although I guess there are use cases. Anyway, that's that's a really fun idea. An online bookstore, oh, an educational business for a client. Excellent, excellent.  For me today, I'm going to I built a lot of new websites lately. So I think I'm going to like, imagine that I'm building a website. And I think it's going to be for like a pet sitting business, it's not going to be real, but you're going to kind of see my thought process as I go through the creation of this business website. So with this in mind, I'm curious, like how confident you are that you could write like a high quality website or how confident your clients are. And this workshop today came out talking to several people at a local word camp that I attend, or not just like not a word camp, but like a meetup. And some of my friends who have websites and when I tell them, oh, I help people build websites, their first question when I've seen it is, is it bad? Oh my gosh, is my website bad? Like, wow, what a question, right? Like I don't, I realized that a lot of the people with whom I was talking lacked confidence in the content of their website, which is how this came from, like where this came about. So I'm curious where you ball and this can be for you or if you're taking over somebody else's website for a client, things like that, like, how confident are you that like the content on your website, they're writing the pictures, all that the fonts, the colors, that it would be high quality. I'm just, I'm just curious, there are no wrong answers here. I'd say that when I build websites for myself, I'm usually probably around a three. And if I were to spend more time brainstorming like this, I feel like it would be higher. But I definitely know that when I've asked other people, like for example, I did a website for a bird rescue several years ago, they had no idea what to write on the website. So how confident are you that you could write a high quality website now? Just show your hands. And when said therefore, great, okay. A two, all right, all right. Somewhat confident. Yeah, all right, great. I hope that we learned things today. And if you are super confident, I hope that you throw out some really wonderful suggestions because we are all learning together today. A three usually two of it is a complex site. Okay, okay. Cool. That's really good to hear too, but confident. All right, all right. Let's keep going. So we're going to use a thought organizer today. And it's going to help us organize our thoughts. And the idea behind this organizer is to really start with the end in mind, like what do you actually want to create with this website, not just a website, but the full, robust, what you plan to do with it. It's kind of like the Alice in Wonderland thing, right, where the Cheshire cat in a tree is like, well, which, which road do I take? She asks, and the cat asks, well, where do you want to go? And Alice replies, well, I don't know. And then Alice answered, well, then, or not, then Alice, I'm sorry, the cat answered, my brain's not working very well. I have not had my coffee yet. It doesn't matter where you go. So it's really easy if you're just mapping around to create a website, it doesn't have to be pretty. But like, starting with the end in mind really, really helps. I know someone in the chat said, Hey, the client supplies the content. I love that. I've recently worked with a dentist website. And the content is actually what they struggled with a lot. So today's thought organizer that I'm going to share with you can also be shared with clients so that if you're, you know, asking clients for, Hey, what do you want on this page, that page, whatever else, if it's kind of like pulling teeth and they're not being very forthcoming, this may be helpful for them. So this thought organizer looks like this. I'm going to be writing in it and with you today. And if you are, if you have a Google Drive, you are welcome to make sure that this is shareable. Yes, anyone at the link can view this. You're going to want to make a copy. So you're going to head to file. And what does it make a copy? So you can either make a copy to your own Google Drive. If you have something on your computer, you can download it here as any one of these, even just a plain TXT file. The formatting will get kind of weird. But if you've got like Microsoft Word or things like that, you can definitely download it that way. Or if you are on a mobile, as I know, a mobile phone, as I know one person is, you are welcome to have a handy dandy napkin piece of paper and just reference this as you write by hand. So you've got three different ways to access this content. So I am going to do the same thing. I am going to make a copy because I like to always write examples as we go. And I am going to do the same thing. Make a copy. My website brainstorm. I'll remember what it is. Shop example. Here I go. I'm making that copy. And once it is yours, you are welcome to write on that or write on your piece of paper, either one. All right. Two seconds or something I was going to say here about this. So the first thing we're going to look at specifically, I want to present a view. There we go. I actually have notes this time. I'm not just winging it, which apparently slows me down. That's fine. So we want to start with the goal of your website. So a well-structured and concise message that conveys your business purpose and services that helps visitors understand what it is that you offer quickly. So this can be what you offer, what is your mission, what are your goals. Any one of those or all three of them can be answered in this first one. So I'm going to write with you. I'm going to set a timer. I'm just going to set this one for let's do a minute and 30 seconds and see if that is enough time over here. I will let you know when we stop. And I'm going to write with you today. So first off, what is the goal of your website? We're just going to write a quick sentence, maybe two. Say, hey, this is the goal of my website. So for me, I'm thinking about, let's see, what kind of website am I hoping to create? I am creating a bird, that's it. This is not a real one, just a fake one, just for today's example. But I'm going to write. So I'm going to think about what that goal of the website is. I'm just going to answer these questions here. So I'm going to go ahead and start that timer. What is the end goal of your website? And go. You are just entering. We are currently working on a graphic organizer. Here's a link to the instructions in order to access this in the chat box. Finish the sentence you are on. Was that enough time? Too much time or too little time? Just so I've got a sense of how we're writing today. We know in the chat box. Oh, good.  Jean, I love that candle. You're starting to ramble. That's totally fine. Good. Rambling is good, but this is just about getting your ideas on paper. You're going to shape those a little bit later. I find that editing a blank piece of paper is a lot harder than editing something that is right here. So eventually, you'll want to write a well-structured and concise message. But for right now, we're just going to kind of like brain onto the page. Okay, welcome. We have another new person. If you are joining us today, I'm dropping a link in the chat. And we're working on this graphic organizer. That link will get you set up with everything that you need today. Okay, cool. So the next question that I have is on content. And I just want to reiterate that you are the source of incredible ideas. So with this in mind, when you're like, okay, I'm going to write a home page, an about page, a product page, just like a services page, a blog page, the biggest thing to remember is always your audience first. So we're going to be writing today specifically about your customers like pain points, right? It might not be a customer, like you might not be selling something, but you are offering ideas. You do want, as you said, people to do something with your website up here. So we're going to take another two minutes. And I want you to ask yourself, hey, what are your viewers and customers needs? What do they know that they need that you can offer? And again, this doesn't have to be for a business website. Like if you were teaching people to crochet, for example, they might be like looking for patterns that maybe they would buy off of you, maybe you would get, I don't know, generation, things like that. But also what don't your customers and or readers know that they need that you can provide? So we're going to take two minutes this time because these are three questions here. We're going to take two minutes and we're just going to write what are your viewers and what are their needs? I'm going to answer these two questions here. So I'm going to write with you and start. You haven't moved on to what your customers need? Move on to that. Oh, I'm so glad I didn't share computer sound with you. That was very loud. Finish the sentence you are on. I feel like I should have given this example first. So for this one, like I wrote one here, and I kind of surprised myself here when I was thinking about like what customers didn't know that they needed with like their parents sitting, like how cool would it be if you had a bird and you came back home from vacation and your bird could like do a trick, right? Like that's something that customers didn't know that they need that I might be able to offer, right? Does anyone have, does anyone discover something that surprised them about something that they didn't realize customers either needed or didn't know that they needed? Just curious or readers. Just curious in the chat box. Keep your secrets. Totally fine. So this is basically how we generate content, right? Where we think about what they need, what they don't need. And this is where things I find start to get tricky, right? A lot of people know these first two steps pretty well. Like, well, I know what my goal is. I know what my content is. But then these next few ones are a little bit more complex. So we're going to talk about calls to action. First off, what are calls to action? We'll come back to this in just a second. Margaret said, what hit me was that I need to train clients on the need for website maintenance. Oh my gosh, what a great realization. They don't know that they need that. I feel like that's a constant struggle. All right, so let's come back to this. What are calls to action? This was honestly a new topic to me when I came back to start teaching WordPress, which is not the greatest thing. But Jean hit the nail around the head. They are, you're getting someone to do something on your website. And I realized as I started researching these that like these are everywhere all over the internet, people and companies and social media and whatever else are using these all the time is, oh, click here to read more is an example of one or it is by now or read my product or leave a review. You see these calls to action all the time, and they do tend to be pretty effective. So what are some of the calls to action that you've seen around? I'm just curious. Have you seen anything recently? One thing that I think of is just the read more call to action is one of them. Jean says, oh, sign up for a newsletter. Yes, collecting people's personal information. That sounds awful to me anyway. But like getting their email, things like that are ways that you can build an audience and build a mailing list and reach out to them through marketing. Like if you are starting a business by now, yes, oh my gosh, if you want this by now for a limited time, it says commercial things. And they're not all quite that in your face, right? Like when you think about like, I don't know, one of those really late night car commercials, if you're in the States, they're just like, find out, you've got to do this thing, come down, come down to the car dealership like all the time, like that you can do them, and they don't have to be quite as quite as out there. So there are several other ones that we can do. I'm just going to write some of these down in the chat. I meant to make a slide for this, but I think that that didn't happen. By the way, we just have somebody new jumping in. So Samantha, I just dropped a link in the chat, Samantha, so that you can get the graphic nicer that we're doing. So there's a lot of things like, so you'll oftentimes see buttons that say things like, by now, you learn more, read more. That's a really great call to action. There are other things that are things like, hey, fill out this form, right, like specifically for contact forms. So you can do that, or people could fill in the forms with some details and receive a free download access to content. There's also something known as product or service discovery. So it's a preview of a product that includes a button or link to a detailed page. And those links read things like discover more or explore or similar props. And again, shop now is another one. Shop now, buy now is another great one. I don't see anybody who's using anything with events. Let's see.  So one thing that you want to really consider as your writing is what do you want people to do with your website? So when you think about when people come to your home page, or when they come to your about page or service page, and what I'm doing is I'm just listing out different things. We want to think about what we actually want people to do on each these pages, right? So I've listed out like the four general pages that are going to be on my website. So on my home page, when I'm thinking about what I want them to do is I want them to, I'm thinking for like my calls to action here, I think I want them to learn more about me. So I want some sort of link, right? So you learn more about why Sarah is right for you. For your bird, there we go. I want them to learn more about me. I want to maybe put some sneak peeks of my services. So I would definitely want buttons to my services page as well. I might think about what this looks like as I go. So for each of the pages, I would start by listing out the pages that you plan to have on this website or that already exist and think about what you want people to do on each of these pages, right? So like on my contact page, obviously I want someone probably to like give me a phone call. That's the fastest way to secure a bird sitting appointment probably. Or I could have them fill out a contact form. So list out in the next two minutes, list out all the pages on your website and think about what you want people to actually do because we don't just want them possibly reading. We want something to inspire them to do something with our content. And it can be as simple as, you know, leave a comment on my blog, share this on social media, anything like that. So two minutes, list out your pages and jot down some calls to action. What do you want them to do on each page? And go. All right, that is time. Finish the sentence you are on. Man, something just kind of tickled my brain here. I'm starting to think about other things up here. Like in my websites content that I haven't thought about before. So like, for example, I realize that someone doesn't know that they need someone who like really knows birds, right? So for example, in my local community, there are a lot of people who pets it.  And I would probably rather have somebody who's been like a lifelong bird owner, watch my birds than a standard vet tech, because they just most standard veterinarians don't work with birds. So I'm finding the urge to go back and add things. But if you're starting to realize, oh, hey, there's more that I could put up here. That's totally fine too. Did anyone have like an interesting call to action?  I should have them do X thing. Because one thing that I thought about with services is, hey, I could definitely do like a form, right? Like, what's one question that I have about like my services? Or tell me about your parent? What do they like or dislike? This is a way for me to discover, you know, potential clients, what they like, what they dislike, if their birds have special needs, like things like that, that just never would have popped into my head and that I have never seen on a website nearby. And that's part of why I'm doing this is just because locally, there are a lot of websites around that I've been looking because I've been looking for a pair, etc. And they're just they're missing these things. And man, I haven't called any of them. But if they'd had this, I want to call them. Does anyone have one that they want to share a call to action that might be might be useful? Just interesting. Okay, so wording, Jean says that they should not have volunteering for an event, not just register to participate. Word choice there really matters. If you're looking at some of these, and you realize that that you could you could adjust your wording so that it's more inspirational, more altruistic will resonate more with your audience. That's a great one. I love it. But if you think of one or if you've become emboldened by all means drop in the chat box. Okay, and I need to make these appear faster apparently because if I just had that on the screen that would have been a little bit clearer for myself even coffee. All right, the next one is going to be about building your credibility. So I want to ask you in the chat what makes you trust a website? And what makes you distrust a website because like that's a really important distinction to have in the back of your mind as you're generating content. So for me, a lack of an SSL certificate, you want to take my payment details without one? I don't know about that. But what builds credibility for you? Or loses it? Okay, Margaret says that the site loses points if the buttons disappear off the screen when I enlarge the text a little bit. Okay, so that responsiveness is really important. Experience is absolutely something that works. So like for example, I've been I've had birds for what like 10 years now of my life. That's that's quite a lot of experience. So we're going to start with the next bit over here and building our credibility. We really want to figure out, hey, what is your expertise or your client's expertise? What are your credentials? What sets you apart from your competition? Do you have any industry memberships? Do you have accreditation awards, things like that? Gosh, I wish I had some of those for my pet city business, but I don't. And then do you have customer testimonials or any existing reviews that you could utilize here? So let's take two minutes. Just think what is your expertise? Put it here because we're going to want to weave it into our website later. Two minutes, go. Finish the sentence you are on. So knowing other people who have used the service, yes, testimonials absolutely can build you up here for sure. Just the other day, one of my friends helped her build her first website. He was asking me for a testimonial for her website the other day. It was really, it was nice. And it did help. She actually got some thoughts from that. Okay, so we're speeding right along since we've got such a small group here today. So the next step in our brainstorm is doing some research. So we're going to start looking at some similar websites. And it's a good idea to look at like three other ones, right? So something that is similar to yours. So you may already have some of these in mind. And if so, you can write down the links to them there. And you're going to want to look at like colors, fonts, and other noticing. So I'm going to pick something that doesn't live near me to demonstrate this just because I want to be fair to the people around whom I live, especially since this is just kind of a fake one. So I'm just going to start googling something. Denver, Parrot, or Denver, because it's not just Parrots, it's like pets. It's hard to find specifically. And you'll start to see like the big companies like Rover.com trusted house sitters. But let's see what we can find just around. And so we're going to find this. And we're going to put a link here. And we're just going to see what we notice, right? So you might notice something about the colors initially. You might notice something about the fonts. You might notice, oh, hey, they're accreditation things. And you're going to want to look at some of these different things. So I'm going to pull up a couple of these just as an example of Denver pet sitters. Again, I'm just going to compare these because I want to see like what are other people doing that's working or that I'd like to do differently. So we've got this next one, Denver Metro pet sitters. And the other reason to pick, especially like these top three is because these are the, I mean, these are the ones that when you will put in your search term, this is what Google and other searches have said, hey, this is the best of the best. They're not advertisements, but they are, they have really good quality content, which is why Google has listed them as number one below like, you know, the sponsored care.coms and things like that. So I'm going to pause here. And we're just, we're just going to notice what like, like similar colors, similar fonts. And then just anything else that you notice, you know, what is it that they're doing well that you're not going to copy, but that maybe you could emulate? And what are some things that you might do differently about them? And then someone asked, hey, if I'm redesigning a website, could I look at the current site and compare and contrast that to others? Absolutely. Absolutely, you can do that. What we're doing here is we're looking for exemplars, like the best of the very best, which as told to us by our Google search, because that does give us inspiration. And again, we don't want to copy, but we do want to find inspiration. What are they doing well that I could also do well in a similar, more, yeah, in a similar way. So I'm going to give you a little bit more time for this one. Let's do four minutes, just jot down some things that you notice. And go. You haven't moved on to a second website. Now is the time. You haven't switched to a third. Now is the time, I've extended the time because this takes a little bit, but let's move on. Finish the sentence that you are on. So as I was doing this, I realized that a lot of these websites had some things in common. I think I might, for future ones of these, add a section down here, things that these websites had in common. If you want to add this to pan, you don't have to. But I mean, I just noticed that like on almost every single pet sitting website I saw, there was a blue color, like blue emphasis, which I thought was kind of an interesting choice, right? Like I just wonder why that is. I wonder what associations come with that. They definitely had a lot of white background, things like that. So basically going through and you can find things that you like on these websites that you can emulate, write them down, but also look at the things that they have in common because what they have in common, you may want to consider using on their own website. So if your website or a book, what genre would it be? Like what should, if you were to picture a library, a bookstore, and your website were a book, like what shelf would it go on? You would definitely see certain things. So taking the time to sit and think, okay, what does this need to look like, right? Because like if I'm writing like, I don't know, a technical manual about how to use word press, right? You don't want like Fabio on the cover, like embracing a woman, right? Like that is not a feature that would go on the cover of that book. So these noticeings are really, really powerful and impactful. So yeah, did anyone notice anything interesting as you were looking at some of your similar things, things that you liked, things that you're like, oh, wow, I didn't even think to do that. My ahas was that they had a login on one of these websites for like your pets that are like that kind of boosted the credibility, like, oh, there's a login, I can just maybe schedule something online, right? Like that would be, that would be the dream. Anyone else see anything like that, something that just kind of caught their eye that they'd inspired something in them? He says that they noticed a soft call to action button, Grizzly Peaks Centering Register now. They might be the same event. So it might be interesting if you, if you're, for example, if you're making a website about events, I'm thinking back to like your volunteer stuff, Jean, too, that you were talking about, can you put different links to the same event, something that might resonate with one audience versus another? It's pretty cool. All right, now our next one is going to be to do some keyword research here. So to start with the keyword research is it's loosely related. And again, you could spend hours and hours and hours looking at other people's websites. We're sticking to three because I don't want to paralyze you with that. But when you think about your business, when you think about what you want to do, I want you to write down what phrases do you think people will search for. So for here, there are no wrong answers at all. We are just going to list out, hey, if someone were looking for your website, what are they going to look for? So take two minutes, jot down anything you can think of that might go in here, and they can be as similar or as different as you would like. But we're just, we're just brainstorming here. We're going to refine it in a minute. Here we go. Two minutes. All right, finish this until you are on, up a little bit into the chat for some asking. I think they were asking how to get this. So let me know if that wasn't correct. So what I discovered as I did this is I didn't really know as many keywords as I thought I would. So now what we're going to do is we're going to use your favorite search engine. So I'm going to use Google for mine. And if you're interested in seeing what I'm doing, which you're welcome to use, you can see mine as well here. So if you would like to see my version and just kind of watch me write, that's fine too. So it's empty for you because I'm hoping that you will write in it as well. Like this is supposed to be a data generation for you as well. So we're going to go to our favorite search engine. And we're going to actually start putting in some of these phrases. And we're going to pay attention to what the auto fill is here. And you're going to notice like what comes up for a second or third. So I'm going to take my keyword list. I'm just going to start at the top. Bird babysitter. Like is anybody even looking for that? Bird baby. Aha, they are. So bird babysitter, bird babysitter near me. Woo, bird babysitting. Oh, I guess that's the name of a company. That might be someone I could look up. How to become a bird sitter. Oh, interesting, that might be a topic for a blog. So you might start to notice things that generate other ideas. But it looks like bird sitter near me is lower. Bird babysitter is another one. So there's definitely that bird babysitter, bird sitter near me. And the things that are closest to the top as you start to do this are parrots. Right now I'm looking for parrot in home. I'm not seeing anything pop up with this. Like a lot of people apparently aren't looking for parrot in home care, right? Like there's bird home care. But as I look at this, I'm looking for like parrot home cage, parrot home called what? Crossword clue. Parrot in home, good or bad. So looking at this, it doesn't look like anyone is actually looking for that. So with that in mind, I wouldn't use a phrase like parrot in home care. Let me split that. I guess I transpose it in home parrot. Let me see if I can find that in home parrot grooming in home bird grooming, bird nest, bird care. Okay. So in home bird care is one. But what I'm learning here is, Hey, if I do offer grooming, because that's one of the things that I mentioned on my website is, Hey, I can cut your bird's wings. I clip them so they can't fly. That might be something that I want to embed on my website. So as you're doing this, a lot of these things are that these are the keywords that you want to embed in your content. So anything that you see that's relevant, you want to add to this top list. So what was that one again in home parrot bird or in home bird grooming? That's the number one in home bird grooming. Use this phrase in my website because that means that someone who maybe is looking prepared grooming might find it based on that keyword. I know people are looking for it because it appears up here. So let's take two minutes and you're just going to start putting in your keywords and seeing a if they're even there at all when you're looking on B what other phrases appear there. And anything that you're like, Oh, I need to make sure that I include this keyword. Those keywords are what search engines are going to look for to index your website. So you're definitely going to want to list these out here and you're going to use them later as a reference. So I am going to do that as well with you starting the timer and go actually use the search engine on my search bar. So let it out of complete. I'm not typing out the entire thing. You'll notice Disney World. Okay. So there's market near me. I can see where I'd be very successful. Looks like Los Angeles is first. The UK is next. I'm Kent in London. I'm putting Florida. Okay, Florida keys. Leap ones. Or maybe like I like them in a red to remind myself not to use them. So why am I not going up with anything? One last one. So it looks like a parrot pet care bird pet. So it looks like bird pets that are is looked up way more often. Okay. Today I learned. Okay. So one thing that I learned is that the things that I thought people were searching for people are not. Did y'all learn anything interesting about what you were searching? Not sure if I just look up weird things or what, but it's hard to think kind of how my audience would think if I were to actually create this website. So now let's just go ahead and move on. So this whole thing was slotted to take an hour, hour and a half if we were like really, really chatty. But Jean says that they had a term come up that you hadn't already thought of. Yes, that is exactly the point. Definitely want to spend more time doing it than just like four minutes for your actual website. But the idea is to come up with those terms that you hadn't thought of. So as you look at this now, I think we're going to wrap up a little bit early today. We've got all of these things now that you can continue to work on, continue to do, continue to generate. But what is it that you want to do with all of these? Your next steps with this would be to probably write out your different pages, right? So all the different pages you want to have for me. Well, I'm not going to do it right now, but you would list all the different pages you want to have. And you're going to go back through and look at what the things that you did. And you want to have a little bit of all of this on every single page. Because one of the things that I've realized is that I tend to have a lot of pages on websites that I build that people don't actually do anything with. They just kind of read like a book. But that's very passive. If there is a goal that I have in mind, if I want people to do things and engage with my content and reach out and become active members of my website or at least active readers, I need to make sure that I have the goal in mind for every single page, right? I need to make sure that on every page, within reason, of course, if you don't want every single page to have like all of this content, but pick and choose, right? On every page on my home page, how do I tease what my viewers' needs are? How do I tease what they don't know that they don't know? I know I want that on my home page. Obviously, a lot of this will go on your about page and on on like a services page. But you want to weave these things throughout so that every single page brings value. Calls to action. This is an important one. You want to weave those onto the different pages. Every page should also have some type of credibility throughout it. And then this just ensures that you are putting your website on the right bookshelf. And the last thing, the last little bit that's really important is, hey, whatever phrases that a search engine is suggesting, I highly recommend doing this a lot more.  Because those are the phrases that people are looking for. These are the phrases that Google is looking for as well. And the more of these that you have on there, thoughtfully, you don't just want to pack them in with no sense. That is how search engines and people will find your website. So let's finish up today. I'm just going to ask a question. How confident are you now? But I think the real question that I'm curious about is what was the most valuable part of today? And what's something that you would like to have seen different? So I'll write this in here. What is the most valuable thing you learned today? What is something that you've done differently in the future? Answer one or both of those questions. And then I will thank you for your time. And I will, if you want, see you later and we'll put some of this information on an actual website. All groups today. It's a one person appreciated comparing sites and picking out what you liked and did not like about them. What you would do different or better. Yeah. I was like the phrase differently because I think what resonates with one person resonates differently with somebody else. But I like that. Okay. Anything else? Next time I'll start with that. The Google Key Search has explained, I think you're saying that was the most valuable thing that you learned today. Or if you'd like me to go over that a little bit more in depth, that would be good to know as well. And thank you all very much for your time today. I hope that I see some of you a little later where we actually build up a website of our own and you can use this information to do that. Maybe something I could do differently next time is to have people find three sites before the workshop starts. That's a great idea. Okay. I'm going to do that next time. Great suggestion. All right, we'll talk to you later. Thank you so much. It's been good to see you.


We're not going to let that go yet, so you can waste your time holding your parents up. First of all, I want to say thank you for your pictures, for your communication with each other today. DAB is something special, 100 years old, doing amazing programs. This is just one of them programs. And if it wasn't for DX, sorry, MX, MX. We couldn't do this. And Taylor, today your speech was something that blew me away as a businessman. And you guys have the resources of an amazing company and 350 plus employees to help you on your journey. When I started my business, I didn't have that. When I transitioned from the military, whereas a cook from special operations, you're out in the military, that's it. You have the resources of DAB and each other. And I hope as I leave tomorrow, I hope they use each other as resources, use the DAB as resources. If I can help any of you, which I told today, and we've done some amazing things today, making phone calls immediately to companies to help you. So, with that, I just say thank you to Dave and I from the Robert Irvine Foundation. I say thank you for having us. DAB again, something special. We are going to do an awful lot, partying with them in the next coming years, as we've done in the last couple of years. Taylor, the legacy that you and Brandon are part of, and you continue to support this amazing company, is something special. You want to say a few words about that? Yes, so just as the founder of Patriot Boot Camp, it's been an amazing journey over the last 12 years. And IMAX has played a critical role in us getting, not only literally to today, but the evolution in for Patriot Boot Camp becoming part of DAB. And if there is one central character to that, it's this amazing human. It was Brandon Dwick. He's a founder of IMAX. He was someone who cared about helping other humans and making society better. And he found passion among many things, but one of them was showing up and helping the founders here at Patriot Boot Camp. So, we just want to say on behalf of DAB Patriot Boot Camp, a big thank you for the partnership over the years with IMAX. And Brandon was someone who also connected deeply with nature. So, as we let these birds go free, we do it in the memory of Brandon and his legacy that we all can take for it at Entrepreneurs. So, thank you. So, to sum it up, this is Patty. Patty is from Great Basin Wildlife Rescue. Obviously, we're here and do this amazing thing. I want you to stand in the moment, because this happens very rarely and the two people are going to let these birds go. This is history for you. You'll never do this again. We'll never watch this again unless we come back here. It's really special. Why is it special? Because what Patty does is what DAB does. We rehabilitate, we give the resources, and we give you new training to go on to do something better. Patty rehabilitates not only these birds, but bears and a few other things we don't talk about, right? But it's amazing the parallels that Patty does, that DAB does, for veterans. It's unbelievable. And I'm just so proud of what you do. And I got to spend the time with two amazing people last night, Sonny and Paul over there, and told me all about, I will be donating to your charity. Thank you, James. I will, because I think you do amazing work. But it's so, when you watch this and you talk to Patty and understand what she does and how she does it, it's no different to DAB. It's parallel to give the resources that you need to get better at what you do or to move on to a different business. So think about that as we hear when we let these birds go because it's really important. Our job, and I said this this morning very briefly, is to build veteran-owned companies. America, and yes, I'm an American citizen for 16 years. America was built on small business, and that is every one of you that's here today. As a cook that now runs 11 companies with 5,000 employees, if I can do it, you're all way smarter than me. Have fun doing it. But use the skills that the military taught you. Use the mentoring skills that the mentors here have taught you at the boot camp because DAB boot camp is something special. I've got to tell you, I didn't have to stay all day today. I could have done my piece and walked away this morning. I was enthralled, enthralled by each one of you as you stood up with your passion about your companies. That's what America is. So thank you for doing that. Thank you. Patty. Thank you. You said everything I wanted to say. You took it all away. There is a process to do in rehab, and I'm really excited. I can't do any of this without my volunteers, and they've been out in the valley over here. And so I'm grateful for them. In doing the rehab on these two particular eagles, they came out of Cottonwood Canyon, and it was during last year's heat and lack of water, and they were so emaciated and dehydrated that that was it. They were on death's door. And so what the process is, is getting them back up physically, but then getting them flying again and getting the right nutrition, but just to get them ready for release right now, because I don't like to take something like this and release them during the winter. It's hard enough trying to find food, so I wait till spring. But it's making sure that it's not too fat, and it's not too thin, and he's ready, and he's rabbit-killed and ready to go out. And like Chef said, be in the moment. And I tell that to my volunteers. I don't know how long I'm going to be doing this. I have no clue. But enjoy it while you can. Take what you can from this. And if I leave here and nobody's crying, I haven't done my job. Aww. The Marines will all be crying. There you go. All right, yes. That's fine. That's fine. See, we're going to come back to the side. You're going to make sure you've got it. Because I'm worth a lot of money. Yes. I'm not. I can imagine. Yeah. Okay. So if you want to take, you have photography, you can go ahead and come in front. Take those two knotted parts. And they are a little bit hard to do. So just pull it to the side. Pull it to the side. And then take your hand and go like this. And then step out of the way. Okay. You got a handle on him? I do. There you go. Take that. There you go. Perfect. Why don't you turn around so I can see you. Wow. Are you ready? No. Hang on. Hang on. Hang on. Hang on. Hang on. Okay. So we're going to count off to three. Okay. Are we ready? Camera's ready? Yep. Okay. One. One. Two. Three. Oh. And he's going right in there. So what they'll do is they'll actually wait probably until the morning to kind of take off again. But they'll kind of hunker down for the night because they don't know where they are. Hang on. I'm crying. Okay. And that is yours. That is so awesome. They fit really good in this little plexiglass baseball. Oh, yeah? Yeah. I know. I'm going to have it mounted. Don't worry about that. I'm going to do that. Hey, where's my mount? Where's my mount? I'm going to shadowbox me. I'm going to shadowbox me for this. Yeah, you do. Especially right there. Yeah. Very cool. Let me take a picture of you. Yeah. What's your number tag number? Hold it up. Pretty awesome. Patty. Oh, yeah. Two. So you know each one is federally protected, right? So they have a number and it's ringed. This is a number of that eagle. Which is pretty cool. Very cool. So they can follow it. You just signed that for a minute. I did. I think it was fine. I didn't have to sell it out. I thought he knew that. I 100% knew. He sold me out earlier. So. Robert. Hey, back to me. I know. That's why I'm still watching. Just in case you come back. I'll send it from my warden in the house. Grabbing the cat. That's awesome. Something beautiful about that, you know. One of those cat in the world. I'm going into this. It's just a fight. Or is this a. Okay. Who's the second? Awesome. Look at him. Look at him. Just don't go in front of her. Just don't go up in front of her. I do want to point out. This is. This is. With a certain number. So what happens is. If they get hurt in the wild.  Somebody finds this and they trace it back. It'll trace it. Actually. To where it was born, you know, hatched. And so we have a history. Wow. Just don't go in front of her. Awesome. You're going to take these two. This one is much larger than the other one. You can tell the age also. She's a monster by the. So she's relatively. Yeah, she's. Yeah, she's. So she's relatively young. She's probably maybe going into her third year. The other one I think was going into her second year. Here we go. There we go. We're going to count all. You know where to focus. Are we ready? Yep. One, two, three.  To the hedge. All right. She's going to come back. She's going to be watching us. She's hiding. Run of applause!


Hello and welcome everyone to this session of the Open Source Summit 2021. As you might have guessed, this is not an in-person session. Unfortunately, the current situation did not allow travel to Seattle. But nonetheless, I think this is a great event, even virtually, and I hope you enjoy this session. My name is Christian Brunner. I work as a software engineer at Canonical, which is the company behind Ubuntu. And I'm part of the team that is responsible for developing Lexdi. Lexdi is a container and virtual machine manager. Consequently, it allows you to run containers and virtual machines. It's a bit different from application container managers such as Docker, Pogman, and RunC, in that it focuses on running full system containers and it can be managed and treated just like virtual machines. And so Lexdi uses Lexdi to run containers underneath. And I'm also the maintainer and developer for Lexdi, which is a shared library which provides a simple API to run containers. And we also develop and maintain LexiFS, which is a tiny fuse file system, which provides a virtualized view of various system resources. And in addition to that, and this is mostly relevant for the talk today, I spend a lot of time and maintain and develop various stuff in the upstream kernel. I do development in a lot of different areas that are nearly all areas that kind of touch containers, but I also have a focus on some aspects of process management on file system abstractions. And today we're going to look at a feature that together has just recently landed in the upstream kernel, which is actually developed and which greatly expands what one can do with file systems from various use cases. So we're going to take a look at what I call IDMAP mounts. So for a quick outline of what we're roughly going to be talking about today, first we're going to look at how file ownership can be changed or by what it is affected by and how file ownership is usually expressed on a standard Linux system. And we're also going to look at various limitations of this ownership model and take a look at various use cases that can't be dealt with nicely in the current model and with the current tool set that we have. And finally, we're going to introduce IDMAP mounts and explain how they make file ownership more flexible and we used to solve use cases that we will mention earlier in the talk. And last, if time permits, we're also going to do a simple demo. This is not so much about looking into the actual implementation. I will be giving talks at other conferences about this probably, but this is sort of more understanding what is the motivation and what can this actually help you with. So file ownership. Well, on a standard Linux system, file ownership is expressed through UIDs and GIDs, obviously. And most people will be familiar with this, right? So note that UIDs and GIDs are not universal. They're neither universal across operating systems nor across file systems within a single operating system. So for example, Windows file systems usually don't implement UIDs and GIDs. They might still provide a form of ownership, but it is usually different from what we understand as ownership on Linux. And similarly, some file systems might not support UIDs and GIDs at all or support them in a very limited form only. So for example, the VFED file system implementation on Linux only provides a very rudimentary UID and GID support and all files are owned by the same UID and GID and ownership can really be altered in a meaningful way apart from remounting the file systems. And also some networking file systems won't really or don't necessarily need to implement a full UID GID concept like we are used to with standard file systems such as X4, XFS or ButterFS. So if you look around your file system in a terminal, you will see that all files will have ownership information associated with them, right? So this ownership information is visible in the output of LS, which usually shows it in the form of user names and group names. So if we go into my home directory, you can see there's brown or brown, which is my local user. But you can also display the raw UID and GID values with the LS tool. And then you can see that my UID and GID on this system is 1000. And this association between UIDs and GIDs and group names is arbitrary for the most part, although quite a lot of tools will pick standard names for their UIDs and GIDs. So there is no necessity, there is no really necessary coupling between UIDs, GIDs and so the raw numbers and names. One thing to notice is that this ownership information is persistent. This is something that we're going to be touching on later in the talk. And what I mean by this is that if I turn off my computer and I restart it again, or if I unplug an external disk and plug it in again or a USB stick, the ownership information will still be the same. There are a few things that you can do to all that is actually, but overall the ownership information can't be easily altered. We will see how it can be altered. The association between the IDs and the names, for example, might have changed, but of course the raw UID and GID values are usually the same. And so file systems such as XFS, X4 and ButterFS and loads of others will store ownership information on the device itself, the underlying device itself. So an SSD disk, your hard disk, a USB stick, whatever it may be, a phones disk. So if I create a new file system, if I create a new file in the file system, it will record the ownership information associated with this new file by recording or storing a UID and GID onto the device. This is information that is on disk, metadata about the file or about the data, for sure. And this allows the kernel to retrieve the ownership information when the device shows back up later, which we're going to be talking about now actually. And this brings us to an important step when dealing with file systems on Linux. When a new disk device, be it a disk in your laptop or desktop or your phone or an external disk or USB stick shows up, the kernel needs to be told to make the files and directories available somewhere in the file system hierarchy. The kernel won't just randomly mount any device or make the files available and we will briefly touch on why it probably shouldn't do this. So in order to expose a file system in the file system hierarchy, the kernel needs to create a new super block, which is done when the file system is mounted for the first time. And the super block is a kernel. It's nothing but a kernel internal data structure that exists for as long as the file system is mounted and records various information and state about the mounted file systems, file system type and so on. I know size, whatever it may be. And once the super block has been created and the file system has been exposed in the file system hierarchy, the files and directories can be accessed by users on the system wherever it may have been mounted. So in order to facilitate very fast access times, the kernel maintains various caches, right, including the so-called decache or dentry cache and the eye cache or eye note cache. We're not going to be concerned with the decache and we will only be concerned in a very high level sense with the eye cache here. Suffice it to say that it would be very costly if every time the kernel needed to do permission checking, it would need to call into the actual file system and query the file system for information about the state of a given file and would slow things down very much and you would notice this probably pretty quickly. So the eye cache is a cache for eye notes and the kernel maintains an eye note in this eye cache for each file and directory on disk. So a file is skipping over a few details. A file or directory is uniquely identified by an eye note. And there's obviously a bit more to it. When you mount the file system, the kernel obviously is not going to go through all of the file system and create cache entries for all of the eye notes. It will obviously lazily create eye notes in the cache. And it's also important to note that it is different from file system-specific eye note structures. What we see here on the slide is the VFS eye note structure, which is a generic abstraction, so to speak, which is used by the VF eye notes cache to represent file sort directories. And when a file is not found, then in the eye cache, new VFS eye note is allocated and the file system will fill in various fields with the information of the file that is stored on disk. And as you might have guessed, the VFS eye note structure also records ownership information, which you can see right here in the IUID and IGID fields, so here to be precise. And once that ownership information has been filled in, it will be used to determine if and how a caller can access or alter a given file or directory. And so this information is relatively stable. The VFS can generally expect that ownership isn't changed constantly for a given file or that directory. There are, of course, a lot of cases where ownership of a file needs to be changed or updated. And in order to change ownership, the caller can use one of the shown system calls, which we see on the slide right here. There are a bunch of them, as you can see, they all have slightly different semantics. And the caller can, but for all of them, the caller can specify a UID and GID and for the file and directory, and it is supposed to be changed to and then ownership will be changed. Of course, this is restricted to a sufficiently privileged caller. An unprivileged caller can just change the UID and GID associated with a file to any other UID and GID. There are various restrictions in place that ensure the changing ownership only happens when it is safe to do so. Otherwise, an unprivileged user could, for example, change EtsyShadow to be owned by any user, including themselves, which obviously would be very bad. So one of the drawbacks of Chilin is that it is a relatively costly operation. In order to change ownership, not just do you need to take various locks in the kernel and ultimately the VFS iNode structure needs to be updated and at some point written back to disk as well. It's also the context switch that calling into the kernel will incur. And for various use cases, we must even change ownership for a whole root file system or for a really, really large directory or a complete file system. And if the file system contains a lot of files and directories, this becomes very costly and then quite a few circumstances prohibitively costly such that it needs to be avoided at all costs, let alone the possibility that a recursive churning operation for a file system or a large directory might also fail for whatever reason. And if you don't handle this correctly, or if you didn't expect this, then you end up with a half choned directory or a half choned file system, which is inconsistent. And to recover this is usually very, very difficult. It's not great. So another way to change file ownership is by making it possible to mount a file system inside of a username space in order to understand how this works. We will briefly take a look at what our username space is, but we won't go into it into any depth simply because we would be too much time. So in short, username spaces are an important building block of safe containers. And they isolate UIDs and GIDs as well as capabilities and other privileged concepts on Linux. So they achieve this by establishing mappings between ranges of UIDs and GIDs. For example, I can create a username space where UID 1000 is mapped to UID 0. And this means that a process running with UID 1000 inside of a username space will appear to be running with UID and GID 0. But if we look at this process from the outside, then we can see that it is actually running with UID and GID 1000. So I've put this on the slide just with a slightly different example. Let's assume we have a user ID of 100,000 outside of a username space, which is mapped to UID 0 inside of this username space. And if I look at a file from outside this username space, I can see that it is owned by UID and GID 100,000. And if I look at it from inside of such a username space with such a mapping holes, then I can see that this file appears to be owned by root. The nice consequence of such mappings is that when a process breaks out of a container that runs with UID 1000 or is located inside of the username space with such a mapping and it breaks out of a container, it can only do as much damage as privilege are assigned to the specific UID and GID and usually UID 100,000 will not have assigned any privileges. Yeah, and these mappings are known as ID mappings as you can see. And they will play a crucial role when we look at ID med mounts in a little bit. So one other way of affecting ownership of files stored on disk is by making them mountable, making the file system mountable inside of a username space. And they will take the ID mapping information associated with that username space into account at the point in time when they are mounted. What this means is that when the VFS allocates a new I-Node and the file system fills in the ownership information, it will map the ownership information stored on disk according to the ID mapping associated with the username space. So for example, going back to the example from before when we looked at the I-Node cache and the VFS specific I-Node structure. If the username space has an ID mapping associated with it that maps UID 0 to UID 1000, then any file that is stored as being owned by UID 0 on disk will be recorded as being owned by UID 1000 in the VFS I-Node cache. So the actual raw device ID that is stored on disk is mapped to something else at the time of mounting the file system. So this is a one-time read the I-Node create an I-Node cache entry for this given file but apply the ID mapping before storing the UID and GID into the VFS I-Node structure. So there are various limitations with both approaches. So first let's look at Chone. The Chone approach has various limitations. First of all, the obvious drawback is that file ownership is always altered globally and permanently. That means if you change ownership of a file, it will be changed for everyone on the system and everywhere that file system is exposed. And it is also persistent permanent as the ownership change will be written back to disk usually for most file systems. So if the file system is unmounted and mounted again, the change in ownership will still be in effect, which is usually what you want with Chone obviously, and not criticizing the implementation itself, but it can be an unwanted consequence. And the other drawback was already mentioned is that choning whole file systems is very costly. And it's also not easy to actually write a coherent code to do it. So what about mounting file systems in username spaces? This is similar drawbacks and some other ones. The file ownership is changed for everyone on the system and everywhere the file system is exposed. That's the same as for Chone. One advantage is that file ownership is somewhat changed temporarily in that if I unmount that file system and mount it outside of any username space, where there will be no ID mapping in play, then the on disk ownership will correspond to the iCache ownership. And of course, it's one way to allow the cost of a recursive Chone. But in addition to the aforementioned problems, most file systems cannot be mounted inside of such username spaces, as the creation of a superblock is a privileged operation for nearly all file systems. And there is a good reason for this, namely to protect against malicious and corrupt file system images. An unprivileged user should not be able to mount a random device. This can be used to attack the kernel. And only some file systems can be mounted inside username spaces. And usually these include so-called pseudo file systems. So file systems that don't really deal with real devices. So for example, ProcFS or SysFS, DevPTS, or in newer kernels, OverlayFS. And most of these file systems aren't really that interesting, apart maybe from OverlaySS, because they don't include file systems we really care about, such as Explore, XFS, or ButterFS. Which is where usually we are most interested in changing ownership information in an easy way, and want to expose it to multiple users. So with these limitations in mind, or with the current tool set in mind, so we can either shown, recursively shown a set of files, or we can mount a file system inside of a username space. Let's look at some use cases that have emerged over the years, and how they can be handled with our current tool set. The first thing I want to mention is that new SSD versions introduce a concept called portable home directories. And the idea is roughly to provide a way to take your home directory from one computer to another without much hassle. And while that sounds trivial at first, it really isn't because there is no guarantee that you will be assigned the same ID on different computers. So think about a shared university workstation. You might have UID-1000 on your local Linux machine, but it is highly unlikely that you will have UID-1000 on that uni workstation, where there are a lot more users maybe already registered. Instead you might have something, might have been assigned something like 1125 or 1001. And when you bring your home directory on a USB stick or an external SSD disk to that computer, you usually have a problem, because all files on disk will be owned by UID and GID-1000, but your login UID and GID will be 1125 for example. And that has the consequence now you can interact with any of the files on disk, annoying. And you can change ownership because you can't use your SSH keys, you might not be able to access directories or even create files. So one alternative is obviously to recursively churn the whole home directory each time you change computers. So when you go to the university, a recursive churn operation is supplied, for example by system D, and then you can interact with your home directory.  Sounds costly also sounds dangerous, especially considering that a lot of home directories nowadays contain huge amounts of data, files and directories. So you also might face the problem that the recursive churn fails, right? I've briefly mentioned this before. So you might end up with a partially changed file system, which also sounds rather nasty. Okay, so but maybe we can just mount it inside of a username space. Well, that could work. But as we've seen, it's unlikely that the file system you're interested in or that you're using in using on your home directory will actually support that. So it also means that you will lose a lot of privileges, as you will also need a new mountain name space and you will probably need to be located inside of the username space to even be able to mount this directory. The file system and to be able to interact with this. So all of the regular behavior that you would expect is now gone. And this doesn't really work either. It's not a very pleasant, pleasant solution. So another huge set of use cases comes actually from from containers. So root file system ownership, for example, is the first the first example we can look at. What I mean by this is when you have a container that is an unprivileged container so a container that makes use of username spaces and ID mappings, you will usually not usually you will need to change the directory the root file system that the container uses to match with the ID mapping of the container. So the files on disk need to be owned by the UIDs and GIDs delegated to the container in the ID mapping of the username space. Other ways you face similar problems as in the portable home directory case that there are random UIDs and GIDs on disk that don't mean anything in your username space. And you can't interact with the root file system. Okay, the container won't boot. It's not a workable solution. Again, here we have the option to recursively churn the root file system, which nowadays is what most implementations do, or to mount it inside of the username space of the container, which again is problematic because of the limited support for the root file systems, and also, and for other reasons we mentioned before. So another use case is data sharing between the host and a container. Often you will have a scenario where you want to share your home directory, for example with the container with an unprivileged container or some data that you want to container to have access to. And once again, if the container needs to be able to write files and lacks permissions aren't really an option and the solution is to recursively churn the the directory as you're going to share with obviously is annoying, because then you will alter the ownership permanently and for the host as well as especially with your own directories might not be what you wanted you also might not want to punch a hole in the containers ID map such that you had 1000 outside and inside of the container correspond to because that would mean it get access to technically access to all files that get added to the container with you ID and GID 1000 and we can't really mount the file system inside of the username space of the container because it is already mounted on the host. And also because because we wouldn't be able to share it with the host if we did mounted in the container not easily at least so again we run into quite a few limitations. And then we have the have another case where we want to share data between different unprivileged containers, which have different ID mappings which you might want to use to increase the isolation between different containers. And in this case, you're really in a pickle because your container will likely always end up with files, they can interact with because churning to the ID mapping of one container will render it inaccessible to the other one. And the same logic the same thinking also applies to mounting the shared data inside one of those containers. So it seems we have reached an impasse right now. Our current toolset has too many limitations to handle these use cases elegantly. And however, the idea of ID map mounts allows to solve these use cases and hopefully I can show you how. So in order to solve these use cases we looked at above, we need to we obviously seem to need a way to expose the same set of files at multiple locations, but with different ownership. This is sort of the goal this is this is this is what we want to address this problem. If we look closely, we don't really need to reinvent a lot of things. Right. So, because if you break the statement down, you see there are multiple problems in this series like, first of all, you want to expose the same set of files multiple locations. And Linux already provides a way to expose the same set of files at multiple locations in the form of mounts or more precisely bind mounts. And there are various definitions that you can give for bind mounts but essentially amount of bind mount is amount of a directory or a file of an already mounted file system to another or the same location within the file system. So, for example, I could bind mount my slash temp folder to slash mount. And this would cause the same set of files to be exposed that are exposed at slash slash temp to be exposed at slash mount. And in addition, the kernel already allows to specify different properties for different mounts. So that's great as well. So mounts have additional properties we need to solve this problem. So first of all, the changes associated with a given mount are restricted to the specific mount. So they are local changes, not global changes as we've seen with both the mount a file system instead of the username space and the recursive shown solution. So for example, I can expose the same set of files as read write in one location and another location as read only and the two mount points don't affect each other. The properties of the two mount points don't affect each other. Another important property of mounts is that the changes associated with them, or with a specific mount are tied to the lifetime of the mount. So they are not reflected on disk or remembered in any form and thus they, they aren't persistent meaning they are temporary only exist as long as the mount exists. And what is missing obviously is the ability to change ownership on a paramount basis, similar to how we would change the read only property on a paramount basis. And this is this is not something that the kernel knows how to do or knew how to do I should I should say. So this is where the idea of ID map mounts comes into play. As we have seen early in the presentation, the kernel already has a concept that allows to change ownership on a file system wide basis by making a file system mountable inside of a username space. But we have seen it is not just limited to few file systems of which most aren't interesting to us, but it is also a file system wide change. So this really won't help us. And ID map mounts start with the idea that file ownership really should be expressed on a paramount basis instead of on a file system wide basis. So the motivating use case for this exists. But even if the motivating use cases we mentioned before exist, but even if they didn't, it seems like something that would be generally useful, especially in the world where you were the coupling between names and specific UIDs and DITs is rather arbitrary and where different operating systems and the respective file systems have to or want to interact with each other. So at the core ID map mounts make it possible to alter ownership on a paramount basis. So they allow locally and temporarily restricted exposure of a set of files under different ownership than the file system mount thus. Obviously you can also ID map a whole file file system. So let's go back to our use cases. So portable home directories and containers. The idea behind portable home directories was to make it possible just to recap this quickly to take your home directory from one computer to the next and transparently use it on both. Specifically all file system interaction should work equally well on both computers independent of the assigned login UID and GID which might be different. So shown and mounting inside of a username space didn't allow for a clean solution. But with an ID map mount we can specify an ID mapping that transparently maps the UID and GID on disk to another UID and GID. So this means I can map the UID and GID on disk to for example the assigned login UID and GID on that computer and expose the home directory with this ID mapping. Another neat consequence of this is that it allows us to assign random login UIDs and GIDs in the future which is desirable as well. So we don't really need to take care, need to care about the on disk ownership anymore as much on kernels and file systems where ID map mounts are actually supported. So another set of use cases we had seen came from unprivileged containers, so containers making use of username spaces. And for the containers root of S we always need to ensure that the on disk ownership needs to correspond to the ID mapping of the containers username space as we've seen in order to for the container to interact with the file system. And this again can either be achieved by joining or by mounting the file system inside of the username space but again we've seen that this is problematic. So with ID map mounts we can simply attach the ID mapping of the containers username space to the root of S mount of the container and be done. And the other use case we mentioned is sharing data between the host and the container or between the container with different ID mappings. And in both scenarios churning and mounting inside of the username space really don't work with ID map mounts. We can simply attach the ID mapping again to the username space of the respective containers to the mounts of the shared directory and files and expose them to the containers. So all of the use cases that we've seen above are actually nicely handled by ID map mounts. So let's briefly look at a demo. We don't have a lot of time to do an extensive demo of this but I can sort of hopefully get the idea across. First of all I want to show you how ID map mounts can be used to map one ID to another ID. So the first thing that we're going to do is to create an ID map mount of my home directory and expose it at a different location. So as you've seen my home directory there is a bunch of files in there as we have seen. We have seen that there is a bunch of files owned by UID and GID 1000. And let's say for example I want to establish an ID mapping where UID 1000 is mapped to UID 1001 so you can mount ID mapped. So this is a little tool that I've written. It's available on my github. There's a sub command called map mount and then the B stands for both so map both UIDs and GIDs and map UID 1000 to UID 1001 and only this one ID. And apply this to my home directory and expose it at the mount point slash mount. Okay let's do this. Again if you look into look here you can see all files are owned by UID and GID 1000. Now let's go into slash mount where I mounted this file system ID mapped. By the way you can also you can do grab ID mapped and then you can see okay cool it's exposed ID mapped. So same thing but now all files here appear as owned by 1000 1000 apart from this dot we get hosts file that was owned by UID 0 for which I didn't establish any sort of mapping. My user right here my user I'm running with ID 1000 so I can't really create any files in here. I can't do anything. It gives me E overflow which is a kernel speak for this ID is not mapped but if I map in as my test user I should learn how to write. My test user has ID 1001 1000 1001 and now I can create files in that directory. Cool and if I look at them with the ID mapping established in here I can see they are owned by my UID and GID 1000. So PVV is here is there but cool thing is if I look at this within my from within my home directory. I can see that they are owned by UID and GID 1000. So they are written back with the correct UID and GID UID 1000 which is how I would want them to appear on disk. There's also avoids issues such as for example you go to a different workstation where you have a different ID. And then you end up with a mixture of ID 1000 and ID I don't know 1235 on there and you end up with a mixture of files you can't really do anything with. Yes. This is the first example this is for example the home home portable home directory example you can see how you can create ID map mounts of your local home directory. Now let's let's look at a use case for containers. Let's launch a new container images. H1 I think that's the current release creating a new container unpacking takes a bit of time. And that's it. It's rather quick as you can see and the reason why it is that quick is because we don't need to recursively shown the root file system you will see this in a bit. We can do H1 and everything will look fine. The root file system will look good apart from proc and sys. But that's okay because they're always owned by nobody, no group, no group. And then we can take a look at the root file system of the container from outside. Val lib lexty we'll take a shortcut root of s and and we can see on disk it's owned by UID and GID zero. So we haven't recursively changed ownership at all. We just applied the ID mapping which is exactly which is exactly what we want. In contrast, if we look at other containers, I hope I have still containers that have an ID mapping applied. I do. So for another container which doesn't use ID map mounts, we needed to show in all of the files to according to the ID mapping of the container in order to be able to interact with them. So this problem is solved with ID map mounts. We don't need to do this. So now let's look at the case where we expose a directory that we want to share between the container and the host. You can lexty allows to do this lexty config device add. We tell it which container we give this device a name. In this case my share, we tell it that this is a disk device. It has a source path, which in our case will be home brawner, which is my home directory. And we want to attach it to my share and we want to say apply an ID map mount to it. Well, actually for fun, let's not do this right away. Let's just add this to my container. Let's go into this container. And then access my share. And you can see like all of the files are owned by nobody, no group. I can't really do anything right here. Permission is denied because I don't have permissions to create any files. And it's really not very useful for me. As you can see, I can't open files. I can't write to files. Not great. So let's remove this and add the device. But this time we tell lexty, please inject the share as an ID map mount. And so we do this. We go into the container. We see there is a mount for my share. And then we go into my share and we can see there is an ID map mount applied. All of the UIDs and GIDs appear correctly owned. I can write files. I can, I'm not using X. So whatever. And I can create files. If I look at them from the outside, we will see that they are correctly owned. Looks nice because in the container we created files as the root user. Don't be fooled. If I were to log into the container as Ubuntu H1 and then go into my share and do touch GID. And then look at it from the outside. You'd see that it's correctly owned by UID and GID 1000. So exactly what we want. And the really beautiful thing about this is that if we haven't created a whole in the containers ID mapping, we didn't need to do this. So if I add the same my home directory at a different location, my share to, and I don't apply it. I should rename it obviously. And I don't apply an ID mapping to it. I will have exposed it right here once with the ID mapping applied and once without the ID mapping applied. And so I can very finely control what access the container will actually have to a set of files. So I hope this sort of gave you an impression of how powerful this mechanism actually is. So right, let's summarize it. I think we're slowly coming towards the end. So as we can see, I didn't bounce really greatly expand the usability of file systems on linux. That's at least the hope. And they provide a localized temporary way of exposing the same set of files at multiple locations with different ownership. And the lifetime of these ownership changes is bound to the lifetime of the mount. And the changes are restricted to the mount. And the change in ownership, something which I haven't stressed so far is instantaneous because it's pretty obvious, I think. So there is no cost even remotely close to what a recursive shown would cost you. And it's a single system call that allows you to change ownership. IDMIT mounts can be created for a whole mount tree. So they can be created recursively. And that's the API was designed so that you can change the ownership of a set of mounts in one go and a bunch and other mount properties as well for what it's worth. And IDMIT mounts aren't a container feature or exclusively a container feature. They are useful for containers, but they are not just designed for containers. It's important to notice they have major use cases outside and completely independent of containers, such as a portable home directory use case just being a very obvious example. And they allow us to solve a major set of use cases. So IDMIT mounts are currently supported starting with kernel 5.12. They are supported by X4, XFS and WeFat. Currently, file systems need to implement support for IDMIT mounts. It's not a very complicated change usually apart from only if you have a special file system. And with the release of kernel 5.15, ButterFS is supported as well. We merged this in a 5.15 merge window. And in the future, we will see support for even more file systems. So if you want to know how IDMIT mounts are created, including a sample program, I would ask you to check out the Linux man page project on the website. Your distro usually doesn't have an updated man page version or a new enough man page version, or you might not even be running a new enough kernel. And the man page is quite extensive and it aims to explain the API in great detail. And with that, I hope you enjoyed this talk. And I wish you a great rest of the conference, whether it is virtual or in person. And if you're in person, please have a beer for me as well. And thanks. See you.


So, good afternoon everyone and thank you so much for joining us here. I hope you have been enjoying OSSNA 2018 so far and weather at Vancouver is also very nice. So today my talk is focused on opportunities from AI and machine learning and how that changes our world with the evolution of cloud native applications. And I really hope by the end of this topic you will really learn something out of it. So quickly moving on to agenda. First of all I will walk you through what it really means to a telecom service provider by what we say cloud native applications. Then I will introduce you to the concept of closed loop which will be a static closed loop and then we will see how to write dynamic policies and dynamic closed loop which is basically assisted with AI and machine learning. The very next thing I am going to do is to walk you through some AI and machine learning practical use cases and show you an architecture that is going to drive the orchestration analytics and AI and machine learning all together. And at the end we will see what are the open source projects and orchestration space and the AI and machine learning space that will drive and help the community move forward. So let's dive into it. The picture here on the on the screens that you are seeing I will start with a brief history of our telecommunication applications. Some of you might be coming from a web scale application experience and maybe some of you have experience with telecom applications and like Jim mentioned before he introduced me here that the biggest challenge at telecommunications industry right now is that the applications that we work with are really complex and they are really hard to automate and really hard to make that to the agile world of orchestration where we are portable scaling in and scaling out. So I just wanted to introduce you some of our challenges. So the physical network functions that you're seeing on this side of the picture here we used to have a number of sub functions represented each in a big box. And at that time when the telecommunications were pitching in their services to the users the criteria for developing that physical network functions was that you need to build a joint big box each with internal sub functions tightly coupled and that will run on a proprietary hardware. And at that time the vendors would say that this particular box can support 50 million subscribers. So basically the code the way it was written was not written with keeping the cloud concepts in mind to make it cloud native or to make it something that's going to run in today's world. So it was the biggest challenge for us to right move into the world of NFE. Then that changed the way of NFE started coming into the picture. So all those vendors what they did is they helped us move that function to alleviate the need of running on a dedicated hardware. And now that function can actually run on a cloud. But the internal component and internal composition of that code was not possible to change overnight. So we were still in that phase. Today we are trying to automate what is called NFE 1.0 and the amount of challenges that we are facing are of course enormous. To represent application like that you need to write a very complex Tosca template and the VNFD packages are very complex and you have to have proprietary interfaces to be able to scale them in, scale them out and to be able to perform configurations. The solution to these problems is our evolution to the cloud native applications. I was really happy to attend a seminar hosted by Linux Foundation a few days ago. The cloud native computing foundation is helping us drive that vision across the industry with a number of vendors. So that we can really deliver that vision where all these internal functions can be represented as microservices. And we can have generic APIs to be able to scale them in and scale them out and be able to develop that stuff more in an open manner. Plus they'll be thin, lightweight, microservices based. We are looking at a world of automation with much better agility and ease. So once that happens for the telecom service providers, the landscape of service delivery will drastically change. With the evolution of 5G and our support of network slicing from our orchestration layer where we can represent the whole network slice with a Tosca template or we can look at the APIs that would lifecycle manage a network slice like that. We're looking at a software defined agile service delivery model in which each offer or product or service can be represented by just one slice. And these teeny tiny micro VNFs in that are represented would be agile. Faster to respond to your automation templates and orchestration workflows. So really looking forward to the evolution of cloud native and the realization of a vision and dream like that. The next important thing that has to happen before we are even able to build the foundation towards AI or machine learning or orchestration or automation is our ability to bring our APIs on a common API gateway. So for example, we have different tools across the organization. We might have testing tools, we might have APIs coming from our cloud infrastructure. We might have APIs from orchestration or analytics. And if we are working with the AI machine learning, we are developing models. And you can even call those models through APIs, mobile edge computing, security, all of that different domains can actually work together much better. If you have a gateway for APIs where you can cross communicate that. So the availability of APIs and management of those APIs on a common API gateway is going to help us set the foundation towards AI and machine learning. So now I'm going to walk you through a simple example of a static closed loop. This is a closed loop orchestration 101. What this example really shows you is that we have a situation where a virtual firewall is serving a production traffic. This virtual firewall, we have associated a policy here, which says that when it hits 80%, we need to perform a scale out action. So the policy has two constructs, a condition and action. Right now, we are looking at a static policy and let's see how this whole thing works. So we are at the stage where the analytics are running very smooth for this virtual firewall. Gradually, there is a production hit and we reach the 80% mark. So we have a static policy executed which is going to perform a scale out operation. We are back to our healthy happy state. This is called a reactive response where static policy has received a trigger and has executed on a scale out operation in almost reactive manner. Now we are going towards a stage where we will build AI application and why do we really want to build an AI application for that? Because there will be too many triggers from complex analytics stream that will hit us in the future. Plus the problem that we tried to solve here was almost real time in nature. But sometimes we want to solve problems like I want to do my cloud optimization by looking at my data or the trends in that data for a longer term. Or I want to optimize my network, or optimize the quality of service delivery of my network on my SDN control plane. Or I want to improve the quality of service of my users based on the user's demographics. Those kind of problems cannot be solved by writing static policies or by reacting on your analytics real time inputs. What we need to do is, we need to build AI machine learning applications to be able to learn from that data to be able to take proactive and predictive inputs in that case. So let's review how do we really build an AI application? The very first step of building your AI application is, think about the problem you want to solve and identify that what problem can be really solved with AI machine learning. That's the critical step. The next step is gathering of real sizeable data around that problem. So say for example, I picked a problem that I want to optimize my cloud for capacity or energy. What I need to do is to get the real time production data of the cloud usage for over few months or years, maybe to take a look at the pattern and trends and how that is evolving. Once that is done, the next important step in the world of a data scientist is the preparation of data. Cleansing the data feature extraction that helps you prepare you for the training stage. And the most compute intensive stage is the training stage where you run your algorithms through a large data set which has the answers pre-populated in the data set so the algorithm would learn an acquired behavior. Once the algorithm has learned, that's called a predictor or a model. And how accurate your predictor or model is, you run your test data through it. Once that's done, if you feel that it has reached a certain level of accuracy, you're good. But if you haven't reached that phase, you have to go back and rework on your model. And when you have your model and it's ready with you, either you feel that all the complex work that you have done so far, you have accomplished your goal. Or you feel that you will have to rework some of that stuff because generally getting to a right model and getting to a right level of accuracy is not a one-time iteration.  Yesterday I was going through my slides and my daughter saw this picture and all she got out of that was there is a lake and the elephant doesn't want to drink water from that lake. So that's not the message that I wanted to convey out of the slide. So I think I have to spend some time on really explaining what this really means to me. This diagram is helping bring the world of orchestration, analytics, and AI and machine learning training all together. So on the orchestration side you can see that this is the orchestration layer which is capable of working on your workflows, your templates, your services definition. So based on that, this particular orchestration layer can look at the static policies or dynamic policies and it can take actions. And those actions can be mapped to either services, create services, deploy services, optimize services, or your cloud. Do something with your cloud. Optimize your cloud resources, shift your workloads from here to there, or turn off some of the clusters to save some energy for you. Or then maybe look at the network and take some actions on your network. All the actions are going to come from cloud which are listening to a written workflow or a template. And it has ability to run either static policies or dynamic policies. Now let's look at the analytics side. There are a number of analytics sources across your organization. Some are services level analytics, sometimes cloud level analytics. And you really need to bring all those analytics into a monitoring engine where you can have a consolidated view of how these things work together. So that you can create a holistic result for a policy engine to take action on. And if you want to do static policies, you will be taking the reactive input and taking that path. But if you have a use case or a problem that needs to actually sit in your data lake where long term data is stored, that's the stage where you will run your AI machine learning algorithms to gain something out of your patterns or your predictions. So either you can do the training on site, which means that the GPUs or the TPUs have to be built on your local cloud. Or I have been learning that there is a possibility you can take your data off site and some of the public clouds can basically provide you the facility of training your algorithm. Either way, what you really need to care about is a model. Once you get that model, you can onboard that model and send it to the orchestration layer. And based on that model, your dynamic policies will be able to take actions on the predictive or proactive inputs coming from your machine learning algorithms based on subject to the test data. Did that make sense? I hope it did more than elephant in the water. All right then. Another important thing that I wanted to get across in today's our message is that AI is becoming easier as we are heading towards the future. A very long time ago when we would start writing our AI machine learning applications, we were really looking into the nitty-gritty details of mathematics of developing your neural nets or UBASian algorithms. Gradually over time, the libraries became more mature and you had ability to have libraries that you could use some functions from them and you could use some AI machine learning. Still, it was not that easy. Then more advanced projects like TensorFlow came into picture. The evolution of AI machine learning started becoming more and more common. And now what I'm recently learning from the Linux Foundation that there are projects like, for example, Ecumos where community can share data, where community can share use cases, and you can really get models just by invoking an API call. The world of using AI machine learning is drastically becoming simplified for people who are not data scientists. So it's a good thing, a good shift in the industry right now. That's why people are excited about the technology to learn more about AI machine learning. The other good things that are happening right now is the wide availability of analytics. We have more sophisticated platforms today. We have the ability to look at the data at a more granular level than past. We can save the data as much as we want to because the storage has become cheaper. So good accessible data. And the other side is the compute power. So look at the story of mainframes. If you try to train your algorithm on a mainframe, it would take months and years. Now look at the compute power of your handheld devices. We have very low cost, high power GPUs and GPUs widely available now. So we can really make machine learning training stages more practical. With that said, I want to comment on the landscape of open source. And Linux Foundation has always been playing a significant role in the world of open source. And I'm a huge fan. So in the space of orchestration, if you look at a few of the orchestration open source projects, we have been really involved with them. And we have been doing a number of POCs. ONEP being my favorite one, have done some POCs this year, looking forward to do more POCs next year. And I feel that there is a huge promise in that project to deliver the orchestration solution that we are looking for, a 5G in our future. Similarly, there are a lot of open source projects for AI and machine learning that you can contribute and you can participate. The notable one here is ECUMOS that is leveraging a lot of open source projects. And once you start doing a contribution in community and start learning some things out of it, you really feel that open source is going to drive this roadmap with all the community members' participation forward. Taking one last stab at summarizing what we have really learned today. And this slide is the last slide before you are able to get your coffee break as well today. So what we learned are three things. Number one is the cloud native applications. We really need the cloud native applications to make automation faster, automation possible. And the next thing is that we cannot live with just writing static policies. We will need to write dynamic policies where we have to do the proactive and predictive solutions towards our use cases. That's where we will need to create AI and machine learning algorithms and work on them. And the last and the most important thing is that open source adoption and participation will help us get there. Thank you so much for joining us this evening. I still have four more minutes. If there are questions, I'll be happy to take them.