So, again, thank you, William, and I want to thank Ellis and TAP for hosting these great topics and webinars for the community. We all hope for the ones that host and also participate, we hope that these are helpful. These are short presentations, a lot of information at times to take in at once. So the hope is generally to get sort of the thought thinking. Some people might already have good ideas and trying to fill in the gaps. Some might be new to the game and trying to just get some information on, okay, so I know there's a need, we need to do this, but I have no clue where to start. So these presentations that Ellis and TAP does, you know, I think it's great for both cases if you're just starting and trying to just see where to go or if you have something implemented already and trying to sort of figure out the gaps. So next slide, which we won't spend on too much time on. So I'm Michael Hernandez, I'm the Director of Client Services at Just Tech and the majority of the staff at Just Tech, we're born in the legal aid community basically. We've had positions, IT and non-IT directly working with nonprofits and that's how Just Tech started. We saw the sort of need to try to help the community and so that's how Just Tech started and as I introduced William before, William I'll let you give your own spiel, but again William's been a part of this community for such a long time and valuable and information that he's provided and willing to help wherever he can. But William if you want to say a few words. Sure. William Guyton, I'm the Director of Information and Technology at the statewide LSE funded program in Oklahoma. Great. Okay, next slide. See if I can do this. So I put this slide up because I just want to preference that our focus is really on help desk support and there is a difference between help desk and network support. So help desk generally is kind of the one off single user type of issue. In this example, this guy is having a word formatting issue. So that's a part of help desk support. You hope engineering time isn't being spent on that type of issue and network support is more things that affect a group of users or to an entire office to an entire organization that might have multiple sites. So that's kind of the examples we gave here because our focus is more on the help desk side versus the network support. So I just wanted to distinguish between the two before we get started. Next slide, please, William. So when you talk about help desk, you have to first understand the model that you're using and there's a lot of options for models. So one is where you have basically a person internally that is wearing an IT hat. So they're not necessarily a actual technical person, but they're savvy enough to wear the technology hat for the organization. You also have where you actually have an internal person as the technician. So you have a dedicated person, whether it's part-time, full-time, but an employee of the organization who is playing that help desk role. And then you have your outsource solution where you don't have an actual employee playing the role of help desk, but you have it outsourced with a company that provides it. And then, of course, there's sort of the hybrid where some places have internal IT people and an outsource solution for certain things. So that basically covers kind of the four areas, the four common areas of it. So I think first you want to identify, know what your support strategy is and who's going to be doing it. So that's preference of this slide. Next slide, please. So one very important thing, regardless of the model, and William, I'm sure you'll have some input here. So whether it's a person playing an IT role, so it's not really their full-time job in the organization, but they wear the hat of an IT person. If you have an internal staff person working IT or outsource or hybrid, you want to make support a central place. So that could be done a number of ways, simply setting up an email address, support at your organization.org. As an example, you want a central place where all of your support issues go because there's valuable sort of data there. And also it makes it a little bit easier if you're sort of sharing that responsibility internally. So to give you an example, if you've got support email addresses, emails going to a specific person, look, people get sick, they take vacation, they're busy sort of doing other things. So if you're emailing support issues directly to someone who's either not available or not in, chances are your issue is not going to get addressed. And then you have all this data that's going in there, and the data could be just looking at common issues. As an example, what are the typical issues that the organization is having? You have all this data sitting in someone's personal company, but personal mailbox. And if you want to share that out with other people, it makes it more challenging. If that person leaves the organization, then you have all that email sitting. So regardless of the strategy that you have, a central point contact where all that data is going is really critical. And this is whether you have just going to an email or that email is associated with a ticketing system that then allows you to really dig into that data. So centralize your contact similar where if you have a support number where you want people to call, again, you want to make it easy one number for them to remember if you have a phone system that's going to allow you to route the calls to different people. So again, depending upon the strategy that you have, if you have more than one person doing that, you can set up a single number to ring on multiple people's phones and to provide assistance. Again, if you're in the same situation with email, if you are calling someone's phone directly and that person's out, otherwise tied up, it just delays the response to support. So centralizing how support is contacted, I think is key. William, do you have anything to add to that? Because I know you always have great information. Oh, yeah. Well, we didn't do it from a data perspective. We had to do it in Oklahoma from an accountability perspective because there wasn't a help desk when I got here. There wasn't a process and there was not a process by which you could ask for help. It was, you know, well, I asked so and so and they were supposed to tell you and it came down to accountability and that works its way both up and down an organization from the staff level up to the executive director, right? I don't know that I should be held accountable for a problem that I was not notified of, right? So you have that conversation with your boss and say, please provide me the opportunity to at least address the problem. So from our perspective, from Oklahoma's perspective, it was really more about accountability versus the data. Once you have a process in place and or a contact process in place, then the data becomes, you start, you can dig into the data at that point. But what initially drove us to implement a help desk was the fact that there was no accountability. You couldn't hold the staff accountable. Did you open a ticket? Did you make the right people aware that you had an issue? And then once you get past that challenge, then you start dealing with, well, how, you know, who do you contact and when and by what method based on the type of incidents you're trying to open a ticket for, right? So we prioritized things, obviously, because there's only three of us and 200 staff across the state. So there are things that get a higher priority. Internet's down in one office that that's a priority over, say, you can't print for the next hour. But from our perspective, from my perspective, initially, when you're coming into a brand new organization and trying to fix things, accountability is a huge part of that and gaining the trust of staff such that they can be honest with you about their technical support issues, because a good amount of time, it's not a technical issue, right? It's a people issue or a training issue or a lack of communication issue. So the help desk is more than just tech support. It is a help desk in that sense, because a lot of what we do is not necessarily tech support, but it is support. Great, great points, William. I absolutely agree with everything you said. Definitely some valuable information there. Okay, let's go to the next slide. So when you have a help desk, when you're thinking about your help desk model and you have something implemented, there are definitely things that you could do to both help your help desk as small or as large as it is, and also help the organization with it, being conscious of time and trying to address potentially some issues, or as William said, sometimes it's actually not an issue. It's a training issue. It's some information that you don't have. So here I'm just going to run through what's on this slide. So create a frequently asked questions. And so again, here is where having that data in one place is very important to help you create a frequently asked question. So as an example, if you're getting a lot of tickets to say, you know, how do I create an email signature? How do I use remote access? How do I change my voicemail greeting? How do I change my printer, my default printer? So you might see a trend of frequently submitted issues. And if you create an FAQ section where staff could go to, you could push users to go there first, depending upon the issue. Now, if their computer is not starting, not connecting to the internet, things like that, you don't want to push users to an FAQ section there. But for those common step based kind of issues, that would be a great place to have users go. And I think when you're thinking about FAQs, you also want to think about those questions for your onboarding process for new users, because a lot of times this is questions that people are going to have from day one. And if you build this into your onboarding process, you're going to, you're going to basically going to reduce the number of tickets, the number of requests that you have from users, because if they know it from day one, or it's part of their packet, if you're giving something like that out, they might later remember, oh, this was covered during orientation. Let me pull that information up, because it actually told us how to create an email signature, because someone might do a name change, someone might want to change their email signature. So having that information again, if you have the information accessible and easily, and the people know where to go, they're not going to submit a support request, and whatever the turnaround time is, they'll go, they'll get the information themselves, they're able to accomplish what they needed, and again, not submit that support request. So it reduces it. Similar with having a good knowledge base. And so here, for me, this is more for your help desk. You know, part of the challenges a lot of organizations has, especially with the small IT department, is you do have people come and go. And when you have a small IT department, especially let's just say a one man band, that person's leaving with a lot of valuable information. So creating a good knowledge base for your help desk team is going to allow that knowledge transfer to happen. If someone leaves, you're going to allow your help desk team to have a resource, because there are some issues that come up, not often, and between if it happens six months or once a year, but it's important information, having a resource to go back to and look like, oh, this happened a year ago. Let me see what I did to address it then versus trying to start from scratch, again, or some examples of why having a good knowledge base for your help desk is important. William, do you have anything to add on those two topics before moving over to the ticketing system?  I mean, we've tried FAQs, we've tried knowledge bases. And users generally, in my experience, don't really look at either, in most of my experience, the folks that are going to look at an FAQ, I've had better results training them how to search on Google versus looking at an internal FAQ or a knowledge base. And in fact, most of my answers when I'm on the help desk are actually just Google queries. It's like, how do I set my greeting in Teams? And I'll actually Google it, find the two minute video that Microsoft put out and send them the Google URL. Because what I want to do is reinforce to the staff that the vast majority of what you're asking us to do for you, you could serve yourself and just do a good Google search. And good searching and sorting is one of those key information technology skills that you need, even as a staff person. And they do that in the legal realm. They just need to get comfortable doing it in the non-lingual realm. So part of what I try to do is say, I don't know the answer, but this is the first thing I would do to try to find an answer. And this is all I'm going to do is if you open a ticket that I don't know the answer to anyway. Right. So that's a great point.  Let me let me email William. He'll get me the answer right away. So I think in your case and look, I think probably the case with other organizations as well, they're not going to spend the time to look up FAQs. But for those organizations that it takes forever for support to get back to them, or if they know, oh, you know, our one man band, he's out today, they're the places that are that users will use the FAQ section, you know, because, you know, so at JustSec, when, you know, even if we're doing an assessment where we're not doing sort of the help desk role for an organization, when we come in and talk to folks, you know, that's even been sort of a common concern where it's like, you know, we have one person doing, you know, wearing an IT hat, they're not really IT, you know, they have a caseload, and I can't always get their attention. Sometimes they're at court.   So it's an interesting one, but a lot of times when people asked us questions, we would, if we couldn't find a good Google-able resource, then we would create a short video and people wouldn't necessarily find, well, strangers would find the video online on YouTube, but we would get those same questions again and again, and then we would just send them a link to the video, which saved us having to explain it again and again. And we've done, we've done that. Yeah, that's great. Sorry. And yeah, I think, sorry, that's also a good point. You know, FAQ does not have to be all text, it does not have to be all kind of screenshots, you know, on the steps. If you have a good video that you could share, absolutely.  Yeah, that's a good point. Yeah, no, absolutely short with videos is key. I think, you know, part of the, one of the responses we've seen over the years is someone clicks on a video and they see it's more than a few minutes, they're like, Oh, forget this, I'll do, I'll figure out some other way, or they end up playing it in the background, not really paying attention, then they have to end up watching it a couple of times to really get the information that they need. So sure when possible is really good. For the next point, ticketing system, which I touched on earlier. So there are some free options, there are paid options, but I would seriously consider using a ticketing system. Again, you want you're trying to centralize where all this this data goes. And having just an email address for all support issues, that's a good way to centralize that. But then you're basically manually going through all those those emails to pull out the data that you need. And with with a ticketing system, that's going to allow you to really to get the data out and not not do it in sort of not not do it in the manual way, but be able to do some reporting on the types of issues that that come in. And as I think as as William mentioned before, all issues are not actual issues. Sometimes it's a training issue. So with that data, you might see, yeah, this type of issue or this type of request keeps coming in to support, we need to we need to do a training on this, or we need to send a tech tip to everyone on this to to to cut off, you know, additional support requests on on that topic. So centralizing that in the ticketing system and and and being able to get that data out is is very key. Depending upon the type of ticketing system that you go with me, there are tons of options and features that that you could use, including building in an FAQ section. In that so, you know, as an example, you might have the ticketing system auto generate a response. And in your response, you could have some some information, including a link to an FA your FAQ section. So that that could be used as another way to drive users to an FAQ section. For that example, you could you can also do that with with just a if you're just using an inbox where you have an order reply, you know, sent back with with some information and including, you know, here's the FAQ section, you know, here's here are some examples of, you know, the information you could get by looking in the FAQ. So as William pointed out before, if users, you know, won't themselves go to the FAQ section and still email support, they might end up going to FAQ. If it's taking too long to get back, they're like, Oh, that that bounce, you know, that order reply received had an FAQ section. Let me take a look and see what what's in there. So just a few examples of how you might use a ticketing system. But depending upon the system that you get, it could be very robust, could could do your inventory, could do your, your your your patching of machines, you know, the sort of endless possibilities with with the different ticketing systems that that are out there. And there are some free options. And and then, you know, there's, there's paid options as well. William, anything you'd like to add on the push for a ticketing system? Not really. I mean, the from our perspective, it was about accountability. And now, you know, we've had the same ticketing system going on four years now. So we're looking at how can I make it easier on the help desk by putting all the tools they need to do support on in one pane of glass. So we're, we're, we're looking at what we like and dislike about our existing ticketing system, and looking forward a year or two and trying to figure out what integrates with our cloud first 365 Azure take on things and trying to figure out how to collapse those what are now separate tools in the in the one pane of glass. But that's that's an internal kind of institutional thing that we're trying to figure out because I want to make it as easy as I can on the help desk folks to be able to give high quality help to our staff. Right. So to add to William's point, what to what he's talking about is basically I was mentioning, there's a lot of tools available. So one tool would be basically remote access. So even if you have one site, it's, it's still useful. But if you have multiple sites and you have one person doing it, ticket comes in, and there's an option to remotely access that user's computer. So if they know, okay, you know, this is something I need to actually see myself, or I need to do on the machine, you know, tools like that are going to make help desk a lot more, more efficient and productive if there's a way that they have to to easily remote into your computer, to walk you through what the fix is to see, you know, it might just be a matter of they need to actually have you recreate the issue, you know, so they can then look into to the issue. So that's, you know, one of the tools that you could possibly get within within the ticketing system. So thanks. Thanks for that point one. The last two points you might not be thinking about for when you're thinking about best practices for for help desk. But my I think my point with this is a part of help desk a lot is hardware related, you know, my my computer's not turning on my computer's running slow. So those are, you know, some common examples. If you have if you don't standardize your equipment, then you're sort of you're forcing your your your help desk person, whether it's an internal person, or outsource to to have knowledge of different computers.  And while we're on sort of the topic of equipment, you know, having equipment like cycle policy, kind of goes hand in hand with that because you might be creating more issues for for help desk and for the users by having very old equipment in place.  But I think, you know, having a lifecycle policy in place at least gives you an idea when when equipment should be replaced, when to budget for it. Because look, the older your equipment is, the more help desk issues that you're going to see.  And you could prevent some of those types of requests coming in by, you know, having a policy in place on swapping out older equipment. So you don't, you know, you don't get to that that point. William, anything to add on on that? We do both. We have a standardized equipment and a standardized software loadout. So we know what that baseline is for both hardware and software. And we also have the equipment lifecycle lifecycle policy. And, and that's both a budgetary requirement and a fairly semi educated wag at how long we've seen the equipment that we have in place now be, be good for. So we do both of those things. One of the interesting things that we've figured out, and then it's not hard to figure out, but was going to go into 365 Azure allows a BYO D implementation, which is interesting. And Microsoft has done a fairly decent job of developing and deploying all of their apps that are exposed in 365 to a bunch of different platforms. So we're at a point now where standardized equipment is not quite as important as it used to be. Because we've got a generation of attorneys coming into the firm now that are bringing their own, they're bringing their own Mac or they're bringing the they're bringing their own iPad or they're bringing their own whatever it may be. And the cloud vendors have been pretty good at making sure that their applications are cross platform. So we're the standardized equipment kind of has an asterisk to the side of it a little bit depending on how you're deploying your software resources. Good. Yeah, good. Good point. Good point. So but glad you guys have both of those in place. It definitely makes it makes it a little bit easier to do all the other things that that come in when you when you have those policies in place. So we could let's go to let's skip a couple slides. Since I've touched on the frequently asked questions that I'll space let's pull up the ticketing system slide. Actually, yeah, perfect. So so here are some you know, free or low cost options out there. You know, I think if if you have an internal person that's wearing an IT hat, I think a little bit harder to go for those more robust systems. And it becomes more of a you know, you just want a central place to try to collect that data and have, you know, multiple people have have access to it. So it's it's not sitting in the single person's mailbox. But these, you know, here are some options that you can look into. But they're they're definitely a ton more available. And I think, William, you've used your fair share over the years. And if you know, people had questions about that, you know, feel feel free to, you know, to email me. I'm sure William would offer any any help as well, even just, you know, post it to the forum on LSNTAP. You know, if you have specific questions about any of the ticketing systems listed there, or what other people are using, including, you know, pros and cons of we've got a talk here from somebody we're using Spiceworks web version for help desks, but find it kind of annoying for various reasons. We'd also like to integrate with Microsoft Teams to get a kind of a single pane glass there, but doesn't seem like it's great for a free solution. Have you tested anything that seems really promising? William, do you know of any that that integrate with with teams? Yeah, you know, teams is more popular over the last couple years. But yeah, we're we're testing a couple of different platforms, and they all have Microsoft has opened up the API is beginning to open up the API to teams a little bit more, such that we're starting to test a couple of different providers via the Microsoft Store. So most of the big players are in that space. So the the go to assists and the and the Spiceworks and the most of those vendors will have some kind of API connection to teams or to Azure or to 365 at some point. What we'd ultimately like to find and haven't found it yet, and I'm sure it exists is because we're pushing teams so hard right now at lasso, we'd really like to be able to use teams for what it should be used for for team building, especially around help desk and training. And that that would include the remote access features that the one click to dial the client or the you know, the person that's opened the ticket so you can actually talk to them rather than text them or whatever have a common help desk file repository and and a how to and that kind of stuff and the videos integrated but use more of the teams like aspects in a help desk environment. We found some individual pieces. Some of them are quite expensive and we're still trying to find the sweet spot. But that's something I want to talk to folks in Portland about is is some of the teams specific tech support tools that they're using because I think it's it's fairly new. And we're still trying to figure out what the best mix of those particular vendors are. Great. Again, it's our it's just the the prevalence of teams in our community is very large right now. Yeah. Correct. So yeah, so that that that's a that was a great question. Yeah, it would be helpful to know a little bit more about some of the quirks you're having with with Spiceworks. I mean, I think if you haven't, I would definitely reach out to their their support. There's a community as well within Spiceworks. So I think in the past, it's been a while since I've done it directly with the Spiceworks community. But I remember that you would get some some good responses from from users out there. So think about that as a as a resource as well. But but sounds like William's going to be talking about this at the next conference in January. So might be some some good information to come from that. Well, I think you get we could skip. Yeah, so so what else? What else can you do? Again, best practices with help desk. So if you go to the the next slide. So like this I know for most places is is daunting for a number of reasons, budget being budget being one year. So we're having a hard time paying for replacement equipment. You know, how are we going to pay for for user training? And again, this this leads back to the frequently asked questions. You want to you want to try to get the most bang for for your buck. So you could you could base. So one is you could do user training in a variety of ways. It might just be documentation. It could be very short videos. It could be using which I think I have a slide coming up. You know, there are online resources that that and free, you know, Microsoft being one of them where you could steer your steer users to resources for for training. But the goal, the goal here is is you know, so how do we reduce the amount of support request support tickets that come in? So again, you want to look at the data that's coming in. And and think about, okay, so so what do we need to do some some training on and and and then figure out how best could we get that out? It could be just a tech tip. Or, you know, as part of you know, think about incorporating it as part of your onboarding of a new user. So, you know, you realize, yeah, a new person starts and, you know, a weekend, three months and six months in, we keep getting these, these questions, build it into your onboarding process, make it available and frequently asked questions or some other, you know, type of resource place. So they could get this information. Again, you're you're looking at ways to reduce help desk as part of best practice.  What are any thoughts on on this? We, we, we don't do a really good job at it. We're, we have a couple of, we don't do any formal new user training. We do, we do have a kind of a kit that we give them that says, these are kind of the kind of the FAQ model where you say these are probably the 12 most common questions, but then really leave it up to the managing attorney in that particular office to let us know if he or she is having a, you know, having a particular issue. It's, it's, it's a real weak spot in the way that we deliver tech to our new users and it's something we're looking at, but we haven't really found a great, a great formalized process. And it may be, again, something we, we incorporate with a combination of things you've already talked about, the FAQs and the videos and, and, and just really developing it and having a decent relationship with your staff in terms of, again, being non-judgeable. I'm here to help that kind of thing. Right. You know, this is a challenge. This is a challenge for most places.    So, you know, if you don't have the funding available to do it, you know, think about free options.  But they are, you know, especially when you have a board with some people from, you know, private law firms, talk to them, see, hey, you know, could we borrow your IT people?  And, you know, these are some things we're thinking about, you know, we need some help with, or we would like to create, you know, can you help us with this? If you're, if you have someone on the board that's at a private firm that's big enough, they might even have a trainer where you could use them. So, I, as I mentioned, early on the call, you know, a lot of people at Just Tech, we have a previous life working directly with a nonprofit. So, I work for a number of years at Legal Services NYC, and we did exactly that. We had a board member who was large enough to have an in-house IT person, in-house trainer, sorry. And there was a period of time where it was a little bit slower for them, and he'll basically allowed us to use him for two, three days every month for four months straight. So, we built a schedule. We had 16 offices around New York City at the time, and we built the schedule like, okay, you know, let's work on the topics. And it was at the time, you know, areas within Microsoft Office and Windows that we wanted covered.  So, he went to each of the office and, you know, did a presentation, did a training, and then did some follow-up where he was available in the office and went around to use his desk and was able to show them some things that was covered in the training. So, that was a great free resource that we had.  If you could go to the next slide. So, again, we're talking about help desk best practices, and having a password manager is very key. Not only, so security aside, you want to have one, but for that knowledge transfer, or just so multiple people should have access. Let me say in no scenario, should it just be one person that has access to the passwords and no one else in the organization know how to access it. And again, whether you have an in someone wearing an IT hat, whether you have someone that's an actual IT person, or an outsourced solution, there should be a place where passwords credentials are being saved and multiple people have access. I think, you know, to take it one step up, it should be you should have at least one or two people internally that have access to it and access to everything. And, you know, they've got to be really upper management to have access to everything. But role, you know, sort of role based access, I think is key because you know, you don't want someone who really shouldn't have access to everything. All the credentials have access to it. And, you know, the reason why this is important is that if that you have someone get hit by the proverbial bus, or leave the organization, you could be in a really bad situation. If you don't have access to all your credentials, and this goes for website access, your, your, your server access, your firewall access, your switches, the local admin passwords on machines, you know, any, any hosted accounts that, that you have that that require access, you want all that information in one place accessible by multiple people. So you have it and not fall into a situation where someone leaves and you don't have a lot of that information. William, anything to add about that? Oh, it's critical. This is change management 101. Yep. If there is one thing that people take away from the entire webinar, put in a institution wide password manager, it helps with security, it helps with help desk, it helps with attrition people leaving the number of hours that this one piece of technology will save you and the number of headaches is so large. Yeah. And I can't remember 150 sets of credentials. And that's how many sets of credentials we have for different services. And the more cloud centric you get, the more of those credentials you'll have, and you can't possibly write them down and manage them that way. So it's it's a key takeaway. Absolutely. Okay, so documentation. So again, we're talking about help desk best practices. And this ties into password manager, but in a different way. You want to document as much as you can about your environment. Now, we are we are focused on on help desk. So I'm not going to get into you know, some of the networking side of why it's important to document your environment. But on the help desk side, you you you want to document, one example of what you would want to document is, you know, the special kind of one off fixes or or setups that that you have to do as an example might be, you know, you've got two people in the organization that that has an accounting package installed on their computer. And, you know, they they had to do certain things to get it to work properly. You want to have that things like that documented because again, that person leaves the organization who set that up. And then you run into an issue. And they had to do something special to get that set up. You're starting from scratch, only trying to imagine how the person set it up. And sometimes going to that vendors, you know, just say QuickBooks as an example, just going to them might not be very easy to kind of backtrack. So if you have the more you have documented and a place where you know, someone could go to look up, okay, so we had this accounting package on two computers, this is how we had it set up. It really just it goes a long way in being able to provide support. And, and, and you know, make it easier, you know, when when you're transitioning, you know, from one person leaving to the next, or even if you transition to a different solution where you have one person wearing an IT hat, and then you bring someone in to actually do IT, or you outsource it, you know, being able to share that documentation really just just goes along a long way. What way and anything to add on why it's good to do documentation? Well, it's great. Obviously, if the the things we try to document are the are the really strange esoteric stuff that's just you generally don't walk into the rest of it is fairly generic. But the some of the esoteric things that we deal with occasionally. One, for instance, for lasso specifically is, is we have the public defender contract in Oklahoma City. So that's not even part of lasso. But it's part of my portfolio. So I've got a special section in our password manager for secure documentation that says these are all the things that we've done for public defender over the last 12 months. And again, hit by bus scenario. This this gives leadership the the knowledge that that I'm documenting things that we're doing for another organization or another grand tour or another contract. But it but what we document is the esoteric stuff. Yeah, great, great. And yeah, it's like documentation is hard to do. So if you have certain areas that you want to focus on, that's, especially for help desk, that's definitely one of the areas where we want to focus on, not just, you know, the common day to day stuff, but the stuff that you don't run into often. So again, practices for question, and also a comment on the comment side, I'll just share super short story. I'm on the board of a small nonprofit are one of our executive team left. Six months later, we found out that we had a donor management system that nobody had access to. Like just that. There was a question here. What would people suggest for password managers? In the question, they do suggest last pass, which is my absolute favorite. What have you guys used? And what would you suggest as password managers to check out? We're sorry, we're using dash. Yeah, we're using that dash line just to throw it out there. Which one William? Dash line. Okay. So, so from from my, my point of view, so we were using enterprise level password manager, because we're, we need a place to save, you know, our internal credentials, client credentials, and keep that all, all separate. So, so we're using pass portal. And, you know, for if you have an outsource solution, they should be able to provide that to you at a pretty nominal, you know, monthly costs that would be worth it. And, you know, one of the reasons why enterprise level is good is because again, you don't want, you don't want everyone to have access, who has access to the password manager have access to everything. You know, as an example, you know, you might not want help that support having access to your financial systems, you know, admin account. You know, as an example, it just, you might have someone else who's handling that and that's just not something they need access to. So, you know, for us, role based is very important, you know, even internally, our, our health desk staff that supports clients, they don't have access to everything. They, if they're not going to touch say, a client's firewall, then you don't, you don't need, you know, that that person should not have access to that credential. So, look, I'll say there are a lot of solutions out there. I think just having one is, is, is really key. But if you have the resources and, and the budget, you know, think about it, enterprise level password manager, but, you know, last pass is a good one, even though I, I don't recall that, and, and sorry, correct me if I'm wrong. I don't think you could do role based access for that. But that might have changed over the last few years, but that last pass was definitely a common one. I've been using their teams level, which does not have that. I believe their enterprise does, but I haven't actually tried it. I know NJP was looking at it, but I don't think they are currently using it at the enterprise level. I think they're still on the teams level. Okay. But yeah, I think, look, it is hard to, to encompass everything. But I think as, as Sarge said earlier, if you're not, if you leave with anything, I'd, I'd rather have you store everything in one place and people have access to everything versus not have anything at all. So you've, you know, you've got to choose your battles. And, you know, again, I think just having something in place is, is, is most critical. But if you have the budget and resources where you could make that more granular, you should, you should definitely do that. So wow, 210 already. Alright, so, so quickly on policies and procedures, again, you're trying to make things easier for, for help desk. So you want an acceptable use policy. Look, it is a document at the end of the day, people read it, people sign it, doesn't mean they actually follow it, but, but you want to have that in place. You know, this is again, a way to reduce help desk issues.   As an example, again, you know, ways to sort of how this ties into, to help desk, how to contact support.  So everyone knows how to contact support, you know, by email, by phone, if you have that, what, what, if any after hour support, you have, you know, so if you have that laid out for users as, as, you know, part of a procedure policy, you know, it does help streamline the help, help desk support. And, you know, one thing that does sort of tie into help desk and questions that come in is, you know, making sure users understand how to handle data internally and externally.  So if you, if users have this, this information ahead of time, or a place to look at, again, you're, you're looking for ways to reduce the, you know, the number of times help desk is, is contacted. William, anything to, to add on any of these? We're trying to bake all of this into one terms and conditions that everyone clicks through and accepts or ejects when they log into our 365 instance. So I'm trying to encapsulate a lot of this into a relatively short terms and conditions that once you touch data in our 365 Azure tenant that you have to accept. That's probably the only new wrangle that I've seen. Gotcha. Okay. Next, next slide. So again, help desk best practices. There's some tasks that you could set up as, as routine tasks.  If you're, if you're not, say, having all of your files hosted online, is mapping network drives is a very easy and, and common one where I see a place that doesn't have that in place. So where that helps help desk is if, you know, if you're manually mapping network drives, I mean, you end up having an issue with the computer, you know, it loses its connection or that that user has a problem with their, with their desktop and they have to use another computer in the office. Once they log into that machine, it's, it's, you know, it's not following them. So they have to then, they have to basically get that user set up, you know, almost from scratch on a new machine. And if you don't have network drives mapped, you're going to have to manually map those drives for the user. So again, you know, thinking about ways to reduce how help desk is, is contact is, is, you know, the thing, think about how you could automate. So mapping network drives, installing mapping printers and copiers as well. If, if you're large enough and, you know, a common, a common issue of, you know, users needing help password resetting passwords, you can think about options where they could reset their own password them themselves, if they're, you know, locked out as an example. And then, you know, again, for the help desk side, if you're doing any kind of inventory, looking for options to automate the software and hardware inventory. So they're not manually, you know, keeping track, like on a spreadsheet as an, as an example. So this ties into options within a ticketing system, where you again, you're, you're automating the process of, of doing your inventory. William, any, anything to add on on automated? And it is, what help desk ticketing software do you think is best for easy reporting? Like when something comes in, when it's done, closed, popular issue solutions, preferably even with a dashboard. So we use the enterprise level connect wise. And I know there's some of the, some of the larger organizations use that. Kasey is another one. William, what, what are you guys using there that you've had for the last four years? You mentioned we're, we're using grove HQ. So G R O V E HQ calm. And there's things we love about it and things we hate about it. Again, we, we kind of didn't keep anything on prem. So we needed a cloud based treble ticketing system. And this was one of the first ones we looked at four years ago. I'll put it in the chat, but that's currently what we're using. Great. And as William said, there's things he loves and, and hates about it. And I'll say there, there is no solution out there. That's just going to be all pros. There's just, you find something that you don't like about the system or you wish was different. Yeah, I think that's generally the case for, for, for any application, not, not just ticketing system, you know, you're going to have a solution in place that, you know, hopefully covers the majority of what you need. And it makes it a little bit easier. But you're always going to find things that, that you don't like about it. And wish was a little different. I would always recommend to submit those suggestions to whoever the vendor is that supports the application. It definitely is a long shot. But if you have the same kind of thought or concern that they're receiving from a lot of other people that, that might help push that, that change through. And I have seen it a couple of times happen. Unfortunately, not as many as I have not seen it.  You might, you might get lucky on what you're submitting.  Next slide, I guess. Well, actually wanted to, I'm sorry, did you, I think sorry, jumped in with Well, you were talking automation, and you're talking about drive mapping. And I remember when we used to do that. Having moved everything to the cloud, we've, we've pushed one drive adoption and SharePoint adoption, where there is no drive mapping anymore. So I got a little chuckle out of the drive mapping, but we've all been there and done that. So I mean, look, communication, we've, we've, we've definitely talked about a lot sort of throughout. So just making sure users know, you know, where to go to get help, what resources they have available. I would strongly, strongly, strongly encourage people. If you're making changes, you want users to know. Yeah, look, we've dealt with organizations that have made smart changes, but they don't inform their users either ahead of time or with enough time. So you want to communicate with the user. So if there's a change that's happening, that even if it's a small sort of impact, but who can create support requests, you just want to communicate, you know, hey, on, you know, this weekend, we're doing network maintenance.  You don't want your antivirus scanning your entire computer during business hours. So, you know, you set it up as at 2am.    So just sharing, you know, communicating information to users, you know, we have this antivirus that scan your computers at night. We also run updates at night. Please do not turn your computers off. So just even, you know, those kind of communication tips is important. Wow, 225, anything to add there, William, on communication? Yeah, and communicate with the help desk. Absolutely, right. So both works both ways. So an example is, hi, help desk. I've had this issue for the last two weeks. Can you help me with it? And the answer is, yes, I would be more than happy. But why did you wait two weeks to let me know? So yes, communication works both ways. Great, great point. All right, let's go to the next slides. We're running out of time here. So again, we talked about user training and there is some free available resources I put two up here. Microsoft has some great, some great videos, you know, to use. So, you know, that's a link to one of them. And then Adobe also has some good online stuff as well. So, you know, check out what resources they have available. You could, you could do it a number of ways. You could just tell users, here's the resource available. You know, look through and pick the ones that you are most interested in. Or if there's certain topics you see as a trend that would be helpful, you could send that specific link out and say, you know, we see this trend, or we see there's a common issue with, you know, formatting in Word. You know, here's a good video to watch. So, there are a number of ways you could approach that. Anything, William, you were shaking your head on the online video.  Well, I just, I posted the Microsoft Teams document page because Microsoft has not been really good at training end users. And this is actually a decent, a really good example of Microsoft trying to change, to help, to help desk help our end users adopt some of their technology. And I just posted it in the, in the, in the chat. But it's a, it's a URL I use quite a bit because they've, they've figured out, again, the, the short discreet how-to type videos, especially around a complex product like Teams, is, is really helpful. And I reference a lot of our staff, too, because they'll do a, how do I, and I'll go, there's a two minute video on that. I could show you, but watch this, and more importantly, this is a resource you need to have for not only yourself, but your entire office. Great. Great. Okay. I think, I think we're on our last slide. So, so thank, thank you everyone for, for, for joining. I appreciate you dealing with my technical issues. Today, I hope, you know, I hope this was, was useful. Again, you know, some, some people joining this, all might be new to them, and they're looking for, for places to start. It sounds like, at least from some of the questions and comments. Folks already sort of have things implemented and looking for, for ways to fill in the gaps or, or, you know, just look at wanting to see, you know, some of the best practices out there. Yeah, questions to LSN tab, you know, is a great resource. So, you know, you should post questions there. If there are questions I could, I could help with, you know, please feel free to email me. I'm sure William sort of feels the same if, if, you know, there's some way that, that he could help on either what we've talked to today or, you know, just something else. You know, even if it's something we don't know, I'm sure we could steer you in the right direction. So, you know, use us as, as a resource, as well as LSN tab, you know, to help in, in any way we can. The work the community is doing is, is, is very important. And, you know, the technology around it, that work is, is important. Because it's, you know, hopefully allowing staff to work more in question and get things done, make things, make things easier, not, not harder. So, using the resources that you have available is, is important. Yeah, I want to say thank you to both of you guys for doing this. This is such an important area. I want to emphasize what you just hit on, which is the email listserv for LSN tab is one of the best resources out there. Ask your questions there. The community will help out. We'll have this video up within the next few days. And there'll be a summary of all of our videos from the last year coming out on the LS tech lists. If there's trainings that you want to see, my guess is that you'll see a survey here in the next few weeks as LSC starts to set up trainings for next year. So please get feedback and ask for the things that you really want to see people do trainings on. That's all I've got. Thank you guys so much. Thanks. Thanks everybody. Bye.


We got some fun petty drama in the Yankees game tonight. Cole versus the nine hole hitter, Caballero for the Mariners. Did I say that correctly? O2, waist to curve on the outside. One and two, comes at him with a fastball. Bat flip, bat twirl, bang the shoes. Stylin' on foul balls. Leg kick, knee drop, 180 spin. Head bob like you should have got it. One two again, goes slider, little floater. Out number one. All right, that was the first at bat. This one's brought to you by shop.johnboymedia.com. And this dude's second at bat. He strolls up to the plate. He's ready to go. He says, time, time, time. But Blue doesn't grant him time because Cole had already started pitching. And he's like, come on, man. I'm trying to mess with the pitcher here. Break up his rhythms. Now Cole waits him out. He's set. Cole gets on the mound. He's set. The 1-0 pitch is 97 mile-power fastball right by him. Cole catches the ball, gets back on the mound. He's like, okay, I gotta wait. This guy's making me wait. We're playing the waiting game. I'm waiting. I'm being patient. Okay, he taps the bat. He taps the bat. Now as I said, twirls is bat, twirls is bat, twirls is bat, twirls is bat, twirls is bat. Twirls is bat. One, one, and another fastball 98 miles per hour because Cole is just sick of it. He was averaging 96 miles per hour this whole game except in that bats against this guy because he was bothering him so much by just doing all these waiting games and making him wait and trying to use the pitch clock to his advantage. And Cole's just like, okay, dude, I get it. Okay, you're ready. Cool. Now I'm ready. Time, time, time. Oh, okay.  Don't let the nine hole hitter bother you this much. That's what he's trying to do.  Just make him wait a little bit. He's making you wait. You're making him wait. We're having fun. We're playing baseball. We're competing. Just have fun. You're pitching great. He's the nine hole hitter. Don't let him get to you. Batch, roll, batch, roll, batch, roll, batch, roll, and everybody is ready to go. And the one, two pitch, fastball, 99 miles per hour, ramped it up even more, broke the bat was so slow that he couldn't get him out.   I dialed up to 99, I broke his bat. Probably would have been better at the 96. Now this is his third at bat. The first pitch, curve ball for a strike, doesn't like the call walks away. Garrett's like, no way. Yeah. He says, you know what? I don't like that call. I'm going to go take a stroll. Garrett's like, no way. This guy's annoying. And to his credit, that's what he's trying to do. So calls like, stay calm, stay cool.  All right. The 02 is coming. Now he hasn't called time yet. He's just gone for some long walks, stretches his neck. Garrett's like, you ready yet, man? Are you ready yet, man? He says, I did call time. I was like, yeah, man, you definitely called time. We just waited for you forever. Circle, circle, circle, line dash, line dash, dash dash. Steps into the box, does his tapping. Trevino says, hey, Garrett, you want to throw the slider? No. You want to throw the cutter? No. You want to throw a fastball? Yes. Yes. And bam! Above the screen. You want to waste my time? I'll waste your time. Booney's like, what the heck was that? Little duck, little flinch.  Everyone gets scared besides the guy in the green shirt. He just raises his eyebrows. Ball comes flying at the net. Way high guy in the green just went, oh, I'm horrified. Nothing, no reaction. 01, 02. Now Garrett throws ball number two. That was a little cutter away or something like that. The two-two pitch, fastball misses, and now look at this look.  Don't walk him now. That'd be embarrassing. You got to strike him out. Come on, man. Finish it. You threw a ball on purpose. The three-two, strike three swing, ha, ha, ha. With the audio, you can hear him laughing at him and staring at him. Better's like, all right, cool, whatever. But the Mariners coach's got service. He starts yelling at Garrett Cole and wagging his finger at Garrett, saying, don't do that, Garrett. Don't do that. And Garrett says, you don't do that, Scott. You don't do that. Don't throw that pitch. No, no, no, no, no.  We got a manager wagging to the pitcher and the pitcher wagging back to the manager. No, no, no, no. You're right, old man's going to tell me what to do. No, no, no. Don't do that, Garrett. Not allowed. No, no, no, no, no, no, no, no, no, no, no, no, no. Not allowed. Take me through the back and forth there with Caballero. Yeah, I mean, sometimes a high fastball can be a really effective pitch. So, gotta change eye levels. It's not every day we see you pointing at the other dugout and kind of shaking your finger there.  So, I wagged my finger at him. As I said, this one was brought to you by shop.jomboymedia.com. Our crew already got some coal shirts up in the store. We have tons of Yankee shirts, tons of MLB shirts, partnered with MLBPA. Lots of good stuff. Shop.jomboymedia.com.


I read an interesting book on Elvis Presley. The downfall of Elvis Presley was he had this manager named The Colonel who was very exploitative and he took the most successful singer at the time, Elvis Presley, and he just destroyed Elvis Presley. So most of you are going to rise and fall based on the power of people that you emulate and learn from. And some of you will think you don't need mentors, but you're always being mentored by somebody.  You're using subconsciously or being programmed by the algorithm. It's very hard to have an original thought. Pablo Picasso said, good artist, copy great artist, steal.


So two years ago, exactly around this time, we launched this vision in this room and said to the world, if you want to change the world, New Zealand can be your platform for it. We've selected 145 fellows from 34 countries. 39 of those fellows are from Aotearoa, New Zealand. Right now today, while we're here, over 80,000 acres of rainforest will get cut down. Nearly 200 species, God's great creatures will go permanently extinct. What are we going to do? EHF is a bet that entrepreneurs can help penalize these shifts. EHF is built with deep respect for those who turn insights and ideas from the realm of what's possible into our shared reality. Entrepreneurs of New Zealand and the world, we want to work with you. We want to support you by your side to help your dreams come alive. And we can all do it like a surrogate, one step at a time. It is my honor to follow in the footsteps of a great man. Thank you, Brother Matthew, for passing me this manuka stick. Manuka is a small tree, but it is so tough that it defeats even steel sawblades. In this way, it is like Aotearoa, New Zealand, a small but mighty nation that has led the world again and again in ways that have changed life as we know it for the better. This is the nation that gave Sir Edmund Hillary the iron will and wide-ranging resourcefulness to summit Mount Everest and inspire the world. Two years ago, the New Zealand Government, Hillary Institute and Kiwi Connect partnered over a bold vision, the vision of leveraging the coming technological renaissance to green the New Zealand economy and improve well-being for all her citizens. In my new role as EHF Board Chair, it is my privilege to serve alongside the illustrious Bill Cromode of NEXT Foundation, Kate Frickberg of Think Tank Consulting, Aniki Goodall, Chair of the Hillary Institute, and now Wayne Mulligan, Chair of Te Aotearoa, Taranaki Whanui. That, you know, it doesn't matter if we're wrong, what matters is we're having a go. And if we knew all the answers, then we probably don't know the right questions. But the most important thing I want to do is I want to make New Zealand petroleum-free and no petroleum-based products than any of our food services. With this Manuka branch, I vow to give my gifts in service of New Zealand and to this noble cause. Thank you.


Let me ask you a question. Here's another logical question.  You're not going to pass away until your 70s or 80s or 90s. Why should a single woman, a single person, want to do a living trust?  You haven't commingled it. You've kept it in your separate trust. So even though 30 years old, you're not worried about passing away, it's very important for a young person to have a living trust as well, not because they're thinking about death, but because they want to protect their assets.


Welcome we got a lot of materials so we're going to blaze through this hold on it's going to be quick So my name is Matt Porter. I'm with Consulco group and If you're looking for Alan, I'll explain on the next slide why I'm not he and you can probably tell because if you know Alan He's got hair down to here. So This is a introduction to memory management and I stress introduction So if you're experienced Colonel person, you might not get as much out of it This is the this is the presentation that back when when the early adopters Embedded litics back in 2000 2001. We're coming in from working on our tosses I wish I had sat down and put something this this good together because these are the things that everybody Needed to understand to really grasp their system alright, so Real quick about the original author Alan Ott. He couldn't be here. Unfortunately. Good friend of mine and he's a veteran at embedded Linux developer He's a Linux architect at Softiron You may have heard they're an arm server company, but he put together all this material. He's a fellow instructor LF training instructor also gives the the Colonel internals class that Contains a lot more than this material so he did a really nice job on the slide deck and Trusted me to present it as well. So Anyway, just want to give him the kudos for this awesome material. So here we go We're gonna talk about Memory management from beginning in the end and it's gonna be the intro as I said so we start at physical memory if you look at your Low-end systems. I have a single address space and memory and peripherals are sharing that same space they're mapped into different parts of that single address space and The all the processes in OS In in this type of system share the same memory space. There's no memory protection Like we you would often hear about and we're gonna get in all the details of this And you're running a process in that single address space Processes can stomp on each other because they're all shared in there. You have to separate them manually and quote user space or your user application can stomp on say your real-time executive that you're using the schedule So examples of these would be an 8086 Cortex-M part AVR's all these low-end microcontrollers in the old pre-MMU processors So let's take a look So it gives us I know a lot of us are not working on on x86 But it serves as a ubiquitous example if we look at a 32-bit x86 system right Lots of legacy obviously, but it is a common ground We have all of these legacy areas and so forth You have hardware mapped between RAM areas You can see that your PCI physical PCI area memory mapped IO is all in the high part. Okay, so that gives you an idea Physically what x86 looks like now What's the limitation with the single address space, right? You have portable C programs expect that they kind of own the whole thing, right? They don't they don't know if you're you're trying to port several C programs into one space You've got to go set the addresses of this this can live here and this can live in this segment So they don't stomp each other so it's kind of hard to do that You've got to have special knowledge of your actual platform Need to know what your total RAM is And you need as I'm saying you need to separate those processes You have to do all this work and there's no protection right as we said rogue programs can stomp all over things so incomes virtual memory and this is where Things get fun So what is it? Right? This it's a mapping. It's a virtual mapping hence the name virtual so So you map a virtual address a fake address to that? Physical address all right when we look back at that x86 Map that's all physical world and if we can just think in virtual addresses We can have any mapping we want so we map virtual addresses to physical RAM But we also map virtual addresses to hardware devices, right? So PCI GPU RAM on SOC IP blocks, right everything So what's the advantage right? Described how in that flat memory model the single address space we have a situation where you know I got to tell something to run at this address and this address and this address up the end times and Actually have a nice memory map of where everything lives. It's not portable, right? So When we have virtual memory, right? You have one processes RAM is inaccessible to the other processor it's also invisible right so you have built-in memory protection and Colonel RAM is not visible to user space directly The nice thing you have is that memory can be moved right so Memory can be visible to different processes, but you have to actually Set up a mapping for that and the other nice thing is rather than in a single address space Where you have all the memory? Sitting there and you have to manually share it and segment it All right, you can now do things like swapping memory out to disk because the addresses you're dealing with are just virtual The other thing you can do with virtual memory is map that hardware, right? We talked about can be mapped into your process address space Okay, we need help from a kernel to do that on behalf of of user space All right, the other thing is we can take RAM memory and we can map it into multiple processes All right, we're gonna get into that more and that's the the shared That would be a case where like a shared library, right where you're mapping it in the multiple processes And finally with virtual memory we get the ability to have rewrite execute permissions placed on those address accesses All right, so we have two address spaces now, right? We've got the physical addresses we talked about and we saw that physical memory map x86. We use an example and That's you know DMA peripherals whatever it maps out to in your physical world, right? Virtual addresses, right? And those are the ones that our actual software uses, right? When we get to our machine code, whatever whatever architecture, that's our load store accesses, right? Out to memory and those are always using virtual addresses all right, so Looking at virtual memory, right? We have to do a mapping this mapping is done in hardware So there's a piece of hardware that assists with these mappings. Okay, once we have it mapped. There's no penalty for Accessing memory that way, right? The permissions are handled without a penalty So this is all handled in hardware for us and we're going to talk about what that hardware is that does this And of course we use the same CPU instructions of same load stores whether it's RAM or a Piece of peripheral IO, okay? So in normal operation, you're always using virtual addresses all right, so what magic does this it's the memory management unit, right and So an MMU sits between the CPU core and the memory all right. It's often in a modern architecture part of the physical CPU itself if you look and Like retro things you'll find that MMUs were used to be a separate discrete part, right? And we're interfaced and we're part of that set Sold just like say a PMIC as often a an integral separate discrete piece of a architecture so The one thing to keep in mind is that the RAM controller is a separate piece So you got an MMU the DDR controller is going to be a separate IP block tightly coupled though And what does an MMU do right and what it does is it just does that magic of transparently handling of the translation of those load store Instructions into physical addresses, okay, so we map the memory accesses the virtual Addresses to our system RAM that physical address space we talked about right same thing with peripheral hardware No different from its point of view right it handles permissions. He said we got permissions with virtual memory and If we have an invalid access to something it's going to generate an exception and with that exception We can go do some interesting things We'll talk about that in a bit, okay How an MMU works there's an important piece of the MMU called the TLB translation look aside buffer, okay, and so that's just a hardware buffer that has a set of mappings and Those are your virtual to physical mappings. It'll have permissions for that space, okay, and There's a there's a granularity in which these mappings are kept and we're going to talk about that in a moment and The the interesting thing is that you know TLB design is a very architecture specific very part-specific performance sensitive and so you'll see a wide variance in how TLB is are designed how Mappings are placed in if it's software done or it's hardware assisted That type of thing and also Capabilities how many slots they have All right, so this is a quick little diagram of what a system looks like if you're having trouble Visualizing it where it sits you see the MMU between the memory controller, right and the CPU You see that TLB on the side with some entries, okay? All right, so they're saying with that TLB the MMU takes a look at that buffer, right? Is there already a mapping in there when it accesses a virtual address and Then it can look that up and if it doesn't Doesn't find one then it's going to generate the page fault Interrupt the CPU. Okay. Now if the address is in the TLB But you let's say you're doing a write access, but it's only set for read permission it's also going to generate an exception and That'll come back into play as we get into how we use those things in Linux so in Linux a page fault right so you have a CPU exception generated, okay, and This happens when you access that invalid virtual address what makes it invalid, right? It's not in the TLB, okay? and you have three cases, so First virtual address just isn't mapped. Okay for the process that's requesting it second, you don't have the right permissions, right and third would be that it's a valid virtual address, but it's currently swapped out And that one's a software condition So let's dive into we're gonna dive into each of those but first we're gonna get into kernel virtual memory side of things okay So we use virtual addresses both in the kernel and user space But the way that we use them how things are mapped are quite a bit different. So in the kernel we use them obviously and but we have this split in how we treat our virtual addresses and the upper part of our of our Virtual memory map is for the kernel in the lower part for user space and usually when we teach people about this it's harder to think with 64-bit addresses, so we go back to 32-bit and We affectionately call the default spot that it splits between user space and kernel spaces C bazillion That's at that 3 gig location. That is a default So this is what it looks like. So you saw that hugely complex physical memory map of x86 32-bit architecture and lo and behold, here's the virtual memory map We've got 3 gig for user space, right Config page offset controls where that split is set at, right? And so every process gets its own 3 gigabytes in that system It has that whole view so you remember to go back to that single address space If you had multiple processes you had to go link them and all these different spots and manage your processes very manually in this world When we link applications, they all end up at the same place, right? And the kernel just has this one gig in our 32-bit case Okay, so as we said that config page offset controls that a lot of architectures If you have specific needs you might fiddle with that a bit that sometimes happens in embedded stuff and the On 64-bit, we don't have this situation where there's ever a possible need to do that essentially On arm 64. We're at eight bazillion there x86 64 the splits at a different location But you know given RAM sizes and so forth It's effectively something that's that's not worried about in the 32-bit system where that page offset is is Has an effect on how we deal with large memory systems, which we'll talk about in a moment So there's three kinds of virtual addresses in Linux and and LDD defines these these best you can you can look at that and The way the way we define them are And some people use some different terminology historically But in the kernel side we have kernel logical addresses kernel kernel virtual addresses And then we have those user space virtual addresses, okay? There's another special case But most people don't speak about them exactly that way either physical or bus addresses But you can look at LDD 3 the link was in there for a little bit more information. So kernel logical addresses That's the what people consider the normal address space that they're normally dealing with what you get back from K malik is a Kernel logical address, okay, they have a fixed Offset, okay, and so You see a magic number there. That's the config page offset value And that would map to that now that that physical addresses specific to one architecture That could be wherever your base of RAM is and it does get more complicated in in various other Segmented memory systems, but this is an introduction. So we keep it simple So because this is a very simple mapping Logical mapping the conversion is really easy to do so visually it looks like this your kernel logical addresses are at page offset Point down assuming your physical RAM in that physical memory map is starts at zero boom you've got this very simple logical thing and Accordingly you have a very simple set of macros that can convert when it when it's a kernel linear or logical address okay now The next thing that it's interesting when when we have a small memory system, right? So we'll we'll call them small or large and this is really Specific to our 32-bit example, okay? So less than a gig of RAM. It's technically less than that really when when you look at where the splits at You have those kernel logical addresses right starting at the page offset and then going through the end of memory So if you had 512 megabyte, okay, then You would have C up through D all foxes right would be that kernel logical area So it's very simple when you have that small amount of memory. That's where it gets mapped into kernel logical space Okay So things that it includes in logical space like we already said Allocations with K malik Get free pages all of the all of those allocators and kernel stacks and The key thing here is And we haven't talked about how swapping works yet, but logical memory can never be swapped out. Okay There is As we said, there's that fixed mapping we saw how simple the Mac the macros were for that Okay, but what's because of that all everything in that kernel logical area? It's all physically contiguous. So that's important because we need that for DMA. So that's that's why you'll see K malik and those those types of allocations used with DMA will buffers, okay Okay, then it gets complicated if you're on a large memory system something more than a gig of RAM nominally Right, we run out of space right our page offset was at sea visit How are we gonna map that all into? Kernel we can't okay So there is We run out of room there and then on top of it we have to have the space for use by V malik Memory which is our kernel virtual address range and we need to keep that so we're gonna talk about that in a moment, okay? Once we go above that gig of RAM nominally, it's actually less then we have stuff Mapped into the kernel virtual Memory area, and so that's the the high mem support Again note that when we're taking the 32-bit model we have that problem with 64 bit We don't really effectively have that problem until we're gonna get ginormous amounts of RAM on the system Doesn't seem like it's gonna happen tomorrow in that space. Okay? All right, so Kernel logical Mapping right we saw a page offset and then in this large mem system, right? We don't have enough room for that so the additional RAM is going to be Mapped separately in that kernel virtual area So you don't have that logical mapping so those things you can't you can't use any of those simple macros on them at all Okay, so how do we call that? low memory is that Directly mapped set right and then the high memory there is Is you know not physically contiguous it gets mapped in on demand we only have that situation on 32-bit and The the key thing on the low memory as you saw and you can see it visually you go back to the these These sections that you have that one-to-one mapping there right, but the rest of it you do not Okay So let's talk a little bit more about kernel virtual addresses So the easy part was that logical set very simple right so kernel virtual addresses I usually call it V malik space like a lot of people and so keep in mind of that and That's that area above that logical range of its managed dynamically and So those are used for non contiguous mappings. So what's the practical case for that? Insmod right load a module Memories allocated it needs to be virtually contiguous for that module But doesn't need to be physically contiguous right so what you'll find is the module does V malik it's got that virtually contiguous area and But the actual physical backing RAM could be scattered anywhere Okay The other piece is memory mapped IO you're using IO remap and friends will get into that right and That also ends up in that space so All right, so quick look at that. We've got logical addresses physical RAM Whoo Yeah, and so that that's what that looks like and then you've got your virtual address space up there right your modules are getting installed your IO remaps all of that all right So keep in mind as we said that the key there is it's non contiguous right you can't can't Can't rely on it for DMA at all. That's the main point here that Yeah, so this is this is reiterating probably maybe too much is This emphasis that on the 32-bit machine, right? We have a very constrained space in that in the the the logical address space if we have You know 768 mega RAM. Okay, so there's less space for for kernel virtual addresses. So Those are tunable and you just don't deal with this problem on the 64-bit system All right, so let's jump into the meat of user virtual addresses because this is where it gets more complicated So our user virtual addresses, right? That's what our applications or our processes are our mapped into Okay, they're all below page offset if you remember that memory map we saw below the 3 gig mark in our modeled 32-bit system right and each process has its own mapping What I mean by mapping is its own view of virtual address space, right a thread shares mappings and Things get a little bit more complicated with the clone because there's a lot of options and you can choose how much you're sharing and so forth but that's beyond intro level and So one of the key things is Colonel virtual logical addresses, right? They have that fixed mapping. Okay user processes are fully used in the MMU and the only time the only time that you actually use RAM is when when When you're actually touching it will get into that right the memory isn't contiguous It's a lot like that V malik space and kernel you can't rely on anything being contiguous just because it looks that way from the virtual address, right and The nice thing is you can swap out remember kernel logical Virtual addresses, it's not swappable memory and the memory can be moved around on you. So That's virtual world All right, so What what does this fundamentally mean since things can be moved around on you it can be swapped out You can't use it for DMA right can't can't allocate can't malik memory And then try to DMA to that virtual address All right, it's not not going to be a stable backing behind it All right, how does this work every process has its own memory map? You can go look at struct MM right, there's pointers to that in your task struct for your process and That's where that whole mapping is kept of those Pages talk about pages in a moment every time you do a context switch that memory map gets changed and That's where that overhead comes right your context switching overhead where you have to go change that mapping Okay So again back to our map here we've got this this view of the 32-bit world and Every time we change the process this whole set of mappings in here into the space is going to change So back to the MMU All right So we use that to manage those virtual address mappings and so I already hinted at page So how does this done it works on the granularity of a unit called a page? Okay? and Some architectures people always hear 4k Some architectures most architectures. They're configurable There's some advanced features some very large page types. We're not going to get into that Today, but here's some common ones right 4k 4k or 64k and arm 64 Like I say, we're not going to talk about huge pages. That'd be a more advanced topic, but let's just assume 4k for This talk since that's what's most most architectures are defaulting to so that's our unit of memory that the MMU can work with right? We're lined on that page size anytime we do any allocations or mappings All right, and then we have this concrete concept, which is the page frame Okay, and that's page size page align unit of memory of physical memory so anytime we say page frame that means in that physical memory map, okay, and When we talk about a page that's the unit for virtual addressing purposes that the MMU is dealing with Okay, and so you'll see that abbreviation pfn throughout the memory management code That's your page frame number right referring to that page frame physical unit Okay, so MMU operates on pages right memory map for process is going to have this huge list of mappings, right? Big space a bunch of scattered page frames all over the place, right a range of multiple pages and so What does the TLB need to know right the TLB when when it actually gets loaded with a mapping right a virtual address the physical address? So page page frame, right and then a set of permissions right rewrite execute Back to our view of that right just as a reminder All right So as we were we touched on earlier If we act access a region of memory right that we don't have mapped we're gonna get a page fault exception Okay, and this is normal right these are good things we want this page fault exception Okay, and I mentioned that TLB is very in size, you know some of the embedded stuff They have 16 entries in it. It's not much when we know that our page size is 4k, right? And so that's got a lot of churn in it. Okay, and so when we context switch We have a lot of page faults as we start touching Virtual addresses that aren't mapped right so your process gets swapped or context switched in you start Executing code. It's touching you get page faults that exception because we don't have a mapping right and We also have a concept a lazy allocation. We'll talk about in detail here All right, so this is what it looks like visually in between you've got your virtual address It's hitting the TLB, and then it's able to touch those physical page frames right through those mappings that are set up All right, so mapped page ranges All right, so contiguous set say the text for your application your process some data area that's mapped and Those are going through the TLB to access actual backing page frames for that area And then you'll have some on map space that maybe hasn't been executed yet Notice that's the allocated frames on that side that's going through there. All right So just just as I mentioned with kernel virtual addresses and that V malik space It's not guaranteed to be contiguous. Okay in in user space virtual addresses All right, so don't rely on that. We already said that's why you can't use them for DMA All right, and one of the reasons for that is it makes it much easier to allocate memory if you get into how the internal memory allocators Work and and think about how how fragmented things get this allows you to go put together a large allocation with a lot of scattered page frames, right and and Almost everything you do it doesn't require physically contiguous page frames backing your code All right so As we're saying when we looked at that that virtual address and it's and we said the virtual address space Say in 32 bit in that 3 gig area We said one of the cool things was that each process gets its own address space So what does that mean you hear that all the time? It means that when you look at the virtual address space and say you look at that task struct and that mm You're gonna see mappings that have that same virtual address But they're pointing to all different physical memory addresses all over the place So if they're in there at the same if you have things scheduled Running next to each other. They're using the same virtual address right they get scheduled in but it's mapped Right to a different page frame each time so but they don't have to know about that backing and so Here's an example Process one with this set of Virtual addresses map through all these different page frames right in the blue and then process two has got these same virtual addresses and He's he's touching completely different page frames right just visually representing that okay Now we get into shared memory right well we need to for IPC purposes shared memory is a common concept a posix concept Normal concept in most OS's and and so shared memory using MMU and we saw how we can have the same virtual address with different page frames right We can have different virtual addresses pointing to the same page frames is essentially how shared memory works right so simply map the same physical frame to Two different processes right the virtual addresses don't have to be the same and Now you have shared memory right two different processes completely different virtual addresses But they're touching that same page frame as they get context switched in okay, and how does that look? We get the shared physical frame down there in green right we've got this virtual address mapping to it It's touching that shared frame right this is a 4k shared Memory space and then this completely different virtual mapping in the other process pointing to the same frame boom we've got shared memory Now So that was a case with with different virtual addresses okay The M-Map system calling may be familiar with right you can get at a specific Address to share the memory so that's That's a different case okay, and it can fail All right, let's talk about lazy allocation So one of the things you will notice when you work on a Linux system or a classic Unix system is that the kernels not going to allocate memory directly Well, yeah, you saw your your call actually come back successfully Right you got virtual memory, but it didn't actually allocate the physical memory of those page frames that back it right? And that's what we call lazy allocation So this is optimization right the kernels are going to wait until you actually need to use that memory So if you're allocating a four megabyte chunk of memory for your database And you haven't touched any yet. It didn't really allocate anything for you right? If you if you never use it you never touch it it never allocates anything All right, so how does this work? So when we we request that memory it just Creates this record of the request in the page tables talk about page tables in a moment returns the process And so you've got that virtual memory set aside in the user space process Okay, once we touch it our old friend the page fault comes into play right? We already learned that we're going to get an exception right because there's there's no mapping there right or it's only set to read permissions right and We're going to go do a page fault handler so Kernels going to use page tables See that the mapping is valid in this case in a lazy allocation right allocated virtual address space But it's not yet mapped in the TLB, okay? At that point it's going to allocate those page frames a page frame a series of it whatever the request needs to be satisfied with okay, and Then it's going to update the TLB. It's architecture specific how that happens of course with that mapping And then he comes back from the exception handler and the user space program continues. So You your malloc got you that Virtual address space and return quickly but when you went to touch the memory all of this happened behind the seeds right the first time You went to dereference that pointer and update it with a value So that's what's happening behind the scenes okay, but you're not aware of that key point here right? But you will see it if you're running Benchmarking and you see that lag right it's appreciable right and you can use you can use tracing tools and see how that's Happening visually The other thing if you have time-sensitive Things here right you know that you have a fast path You can go Preallocate that you may have used mLoc or the family mLoc calls That will go ahead and preallocate these things so you don't have that lazy allocation situation So as we said getting into page tables tlb entries could be the entries into tlb Can be a limited resource right we can't just map the whole world of our address space in there right? So we have a lot more mappings in that struct mm for our process than we have tlb entries So the kernels got a track all that so it has a set of data structures. We call the page tables, okay? and you can look in struct mm and VM area struct to see how those are done and But it's essentially a hierarchy that leads you down to that 4k page Right and the associated mapping to page frame number and the permissions right so everything lines up with what needs to get Loaded into the tlb's Okay, and also has metadata in addition to that about is it valid or not and so forth and some other housekeeping Flags as well, okay? So When we have something the page table in a tlb we so we have a valid mapping Right, and you touch it the hardware since there's nothing in a tlb yet is going to generate that page fault right? CPU doesn't have the knowledge CPU the mmU right only a kernel does All right, so our page foot handler runs right it's gonna traverse these page tables Find that mapping for the virtual address right page granularity Select and remove an existing tlb entry create a new one with our address and the correct permissions and so forth and Come back to the user space process. Okay? all right swapping and And Good alright So swapping we're used to our systems. We do deal with our desktop systems our development systems Where we have a lot of swapping out to our disk when they're doing heavy builds, right? I'm running low on RAM and You know how this works is the mmU is the thing that enables this okay, and so you're gonna run out of that 16 gig RAM you have under these heavy builds and You're gonna context switch and it needs more memory and it's gonna take those page frames that were backed and It's gonna take the contents of those and it's gonna push them out to your storage right and then When you need that data back and you've been context switch back in it's gonna read that back off that slow storage and Bring it back in that's the big picture right so Low-level details, right? It's gonna do that on a frame base basis, right? It's gonna copy a frame to the disk remove the tlb entry and then that frame is free to be backed for another process right So when we need it again, right CPU generates page fault right common theme here, right? We we flush that entry out of the tlb, right? so now it's gonna generate a page fault and then when we we hit that page fault process sleeps we copy that frame from the disk into an unused frame and We update that page table entry and then wake the process back up Okay, so it's gonna be slow process right we got to go out to that block. I owe or throttled by that bandwidth now And so when we restore the page to RAM, okay? We're not necessarily getting the same page frame. So again, we have this virtual dance going on here, right? There is no persistence or affinity to that original physical page frame So you need to get rid of this notion that paid, you know physical addresses matter, okay? You will use the same virtual address though, right? Because those mapping stay the same in user space. All right, so you don't know the difference. So your code's executing along you yield the processor It gets swapped out You context switch back in it could it'll it'll redo that that mapping same virtual address And your code continues on at the same virtual address But a completely different backing as it the freight page frame context gets copied back in and then mapped in okay? Again, this is that low-level detail why we said we can't use user space virtual Addresses for DMA. We have no persistence of the physical backing that the DMA engines in the peripheral hardware need All right, so what does this look like visually? We've got this frame that was selected By the kernel to be swapped out to our disk. We've got this wonderful trash can looking cylindrical discs thing here and We copy that frame out to the swap media We invalidate the TLB entry page table entries invalid now, right? Okay, and Now there's there's no entry there So that that frames freed up so now you can free it back into the allocator pool But the data is preserved out there on disk. That's in your swap partition. All right. All right now We go we get we get context switch back in right we're back and running same process We try to access that same virtual address. We were just running when we got so rudely taken off the CPU and We get the page fault thing we've been through the page fault dance before and we just rock on through that We get copied from so swap this cylindrical simple disk thing and Put back into that page frame that we got allocated Create the TLB entry But I had one more animation yeah, and Then we return to user space Now we can access that virtual address. We've got the same data. We had before we got swapped out. All right, so I'm actually running this on time beyond so I win. All right It's 95 slides So user space We've got several ways So so we've been through that whole stack all the major pieces of how everything's happening in the background now now Let's see how this maps into you know our apis we have in user space right and So we have several ways That we allocate memory right We've got all our family of Alec things and I've referenced them a couple times verbally right we know that we can M map To directly allocate and map pages. We often see that to map some peripheral IO If we're hacking around not doing proper kernel drivers We have break and s break where we can modify the heap size right So first off met M map right One way that we allocate a Bunch of memory from user space right You'll see it if you if you if you live the world of running s trace on things You see lots of M map happening right when the files are getting Opened and so forth So if you use map anonymous you get you get allocated Normal memory the shared flag allows us to share that memory with other processes All right, so break. Why is it called break? That's the top of the program break legacy terminology right and so Effectively you increase the heap size with that as we're saying okay Now lazy allocation going back to our whole lazy allocation technique, okay We have a situation with with if we look at M map dot C and do break That it's implemented a lot like M map. All right, so it goes in it modifies page tables We talked about how that happened right where we modify the page tables and then we wait for a page fault Okay, and the other thing you can do is you can pre fault Who we talked about with M lock right and not have that issue where with with accessing the memory you have this long lag Relatively long lag where it actually has to allocate that big big chunk of page frames for you All right, so you can you can take that cost up front With M lock and then have relatively deterministic behavior once you're actually accessing the memory the implementations of malloc and calc Sort of the same thing They're gonna use break or M map depending on how big the allocation is and that's gonna happen behind the scenes right and If you if you are astute you can modify that behavior with malopt You could set the threshold parameter to say where where one kicks in or not that's often used in system tuning, okay, and then finally a stack If a process goes beyond a stack right CPU is also gonna trigger a page fault, okay One of the special things the page fault handler does in this case Right is it's gonna detect that you got an address just beyond beyond the stack It knows where that's at right and then it can allocate a new page, right? So it allocate another pfn Go into the page tables map that in drop it in the TLB and Remember pfn could be anywhere. It's not physically contiguous. It's just virtually contiguous. So It's faulted in Execution continues on and it's able to you know drop stuff on that segment of the stack You can see how that works and do page fault That's the arm version And So quick summary Like I said introduction. So if you're already a kernel expert you probably know all that but we went through physical memory, right? We looked at a stock You know x86 familiar memory map we talked about virtual memory three types, right? Kernelogical kernel virtual user virtual which ones are contiguous or not, right? We use Kernelogical for DMA We went through user space addressing how Processes will not have contiguous physical memory and how Swapping page faults work to do lazy allocation and so forth Like I say we cover swapping and then how those user space APIs map on to all that So that's it for the intro. I've got one minute For questions Yes way in the back Okay, so so first part first part of the question let me address that so the question was well If the kernel always has the mappings, right and you're talking about that kernel logical Mapping that has why do we have to wait for this expensive mapping to user space and that that so to answer that And hopefully I'm answering the right question The reason for that is those those Kernelogical mappings if we just use those it would be just like that single address system without an MMU and I can tell you that There's there's systems that in the 90s that had MMUs that running our tosses like VX works They would map with the MMU just flat address space because they had to have the MMU on for performance reasons But you were you you don't without without having your own process space, right? You would have to link everything in its own address space and everything so Kernelogical addresses are nice and linear and easy to think about but You have to do these remapping For user space to have that nice world that we enjoy of that protected per user process address space where you just Write a program link it and it'll run in any context, right? If we had all one mapping of the Kernelogical spaces be just that single address space You'd have to link your program at zero and one bazillion two bazillion and manage them Not stomping on each other as you allocated the memory. I Hope that answers the first part of it and I'm out of time, but we can talk about the second one Yep, sorry 95 slides, so