Yeah. Hi. My name is Narendra Ghate. I'm a designer from IDC. Our professor, Professor Natkan is here, so we are very honored to be here. I've passed out 97. So last 20 years, I've been in Tata LXC. I'm leading the design studio there, and we're doing multiple projects right from UX and in AI, which is something that we'll talk today, to transportation, service design, product design, all of that. So it's a fun place to be, and I hope we have a good discussion today. Thank you. Raghavan, Director of Products at Adobe. I've been in the software industry for longer than that. I'm sure some of you may go all the way back to the days of fireworks when that was a cool tool to use. So I managed that, and after that, it's been fun with Muse and Dreamweaver and now into the more modern world of XD, which is a new tool that we have introduced. I'm really looking forward to how the other panelists and all of you feel about changes in UX and how we as a tooling and solution provider can meet your needs. I've already had an introduction, and pardon me, as you can hear, I'm quite sick and you may be getting sick of me because I've been on stage twice now so far. But I do work at IBM Design, helping clients take advantage of emerging tech, and the keynote I gave yesterday and the workshop were both about a new book that I've written about an emerging tech that I'm super interested in. Awesome. Let's start with this question about why emerging technologies are the next frontier for UX? What kind of roles and opportunities UX designers can play in this field? We can start. Mia, I can start. I am a super proponent of designers leading the charge into emerging tech, largely because we should do what we always do, which is bring that human centeredness to it. The prior talk was actually all about that, and I believe that's like if we just let the tech go without user centeredness to it, disaster will follow. I think what's world without changes? The world is changing whether we like it or not, and it's I think up to this group here to humanize it. I think that's what the real value of UX is to make the technology or the other advances be more value add to the general population rather than just to minorities. That's how I see it. So, Javadev heads the society general on emerging technologies. So, can you give a quick intro? Hi, I'm Javadev Sambedu. I'm a part of Sockchains Emerging Technologies and Trends division. So, we focus on technologies that will come mainstream five to 10 years for the bank. We focus on blockchain, machine learning, deep learning, IoT, and augmented reality. There is a common sentiment in the industry that emerging technologies can be both boon and pain for consumers. Chris, what do you think about the positive of an emerging tech? Since I have the mic, maybe I should answer. See, it's slightly long-winded answer, so please bear with it. If you see what's happening till almost five years, from industrial revolution till about five years ago, what was happening was that the designers or the people, for example, were not actually fully utilizing the technology in the products. For example, you had to be trained to use the product better. Now, next five years as AI comes in, what's going to happen is that the technology can do far more than what people really... Sorry, it's the other way around. People expect far more than what the technology can really offer. And I'll just give you an actual incident that happened today morning. And Alexa has been, or Amazon Echo, has been introduced in India just now. And I got mine yesterday and I introduced it today. I mean, I sort of installed it today in my house. And I was very happy because, you know, it was doing a fair bit of complicated things like follow-up questions and stuff like that. And I was thrilled. I told my wife, see, it's working. My wife asked her, it's a question. So today, what's the traffic like on the way to Whitefield? It couldn't answer because the Google Maps is yet not integrated in Alexa.  The point is, the people expect far more very quickly because of these buzzwords like AI and smart technology and things like that. And then when it does not perform, you really feel that the product is not up to the mark, the brand is up to the mark, not up to the mark. And for the next five years, I think the role of UX designers would be to how to humanize almost the other way around, how to make people understand technology better so that they don't expect, or they don't let their expectations run away and they think these products are magic and let the products mature.  And that's where I feel UX, people will have a large role to play to just make this gap that is widening between users and product, come closer again. I mean, I do have a point of view. When you talk about anything that is emerging, it is disruptive by its very nature. And anything that is disruptive can be scary. You may not really, you can be on either of the sides. You could be benefiting from that disruption or you could be made redundant because of your current skill set or the role that you have. I guess it's all about how do you react? How do you make the most of it to even convert a potential, bend into a positive moment for you? So that's how I see it. I think I feel quite similarly. We have a lot of problems that we're facing in the world and they are massive and they are global. And unfortunately, the way that cultural attention works, it would be difficult to use old tools to try and solve those problems. Emerging tech gives us both the opportunity to rethink the problems, how we address those problems. And just because of the nature that it is emerging, people will pay attention. And if we can design them right, we can really use those to tackle some of the global problems that we have. So I know I'm a bright green optimist, but every time I think or look at a new emerging technology, that's my first take on it. Well, how can we use this given the massive problems that we've got? All right. So Jaya, what are the promises and challenges you look for from emerging tech? Okay, specifically each of the technology that we are tracking, I mean, we look at it from what are the industry trends and identify some of them, which we think has maximum impact. For example, one of the things that we've been pioneering is from last two to two and a half years, we've been looking at blockchain and the kind of disruption that it can bring. Because specifically, instead of waiting for other companies to disrupt us, as a bank, we ourselves are disrupting. As a lab, I have absolute autonomy on the kind of products that I can choose, which business line of our group I wanna go for. And we start exploring with this. And in many cases, you can clearly see, because we are doing this, the only one who's gonna be majorly affected is us. But instead of being worried about it, we look at this as an opportunity that we can invent a new kind of a product which does not exist in the market today. And that's the kind of mind, it's more of a mindset than anything else. And that's where I think emerging technologies helping us have this conversation. And it's the same. In fact, unlike any other technology, I mean, we work with IoT, as I said, augmented machine learning. But blockchain has been the only place where we've ended up collaborating with our competitors. That's never happened until now. And the number one bank in the world in commodities is working with us. We are the number second in the world in that. And this has never happened in the history because we never see eye to eye on topics of collaboration, let alone on technology. But in blockchain, we've realized, if you collaborate, both of you, I mean, have a chance to win. So that's the kind of disruption what we're seeing, which we've never seen something before. Great, Raghavan. So you guys are really, really groundbreaking products you're making. You are disrupting yourself by creating brand new products, future projects. And especially most of your products are targeted for designers and the developers, animators and whatnot. So what are the promises and challenges you look at from this emerging tech perspective, building products from Adobe perspective? The pace of changes, the pace of disruption, innovation has just significantly increased from the days of, if you look at Photoshop, what was changing from a core value that it was providing? Not much. There were obviously innovations in that, but from a design perspective, there was not much happening 10, 15, 20 years timeframe. But then I think in just the last three, four, five years with I think the value of UX becoming so critical, the design world has totally transformed itself. Last year's technology is no longer the hottest of the coolest. The innovations are at like almost on a daily, monthly, weekly pace where it's hard for a tooling and a solution provider for us to think about like what could be coming, forget about today's problems. You have to build solutions that would be value add for the next generation. That's tough, but that's fun because instead of just solving the same thing over and over and making some incremental changes, now you are truly looking at IoT or machine learning and how do you really bring that in and add value? It's not easy, but I guess that's what is keeping the team really excited about it. Yeah, as you rightly said, this technology is actually moving very rapidly. So in this fast-paced technology of tradition today, how designers can upgrade their skills? Do you think our designers need to learn code or what breed of designers is industry looking for? I'm sure we'll have a good discussion. Yeah, I am a strong believer that designers need to understand beyond just pure visual aesthetics and into the technology from a code perspective. And just take the example of such a mundane thing compared to IoT or more esoteric things, such a simple thing as website design, right? I'm sure all of you at some stage have done that or continuing to do that. If you're designing the experience for a website and if you don't know the technology behind it, which framework or which framework is gonna be used by the developer, I don't think you're gonna succeed. Empathy for the end user is one, but if you cannot collaborate and work closely with the developer to whom you are handing off the design, it's a, I think it's very inefficient. I've seen that happen so many times and I just really feel that if the designer can not actually code, but if you can at least relate to it and can understand it, that's significant. I think the days of somebody who was just purely drawing boxes and thinking his job is done, I think that's just not how I see it. So Narendra, your perspective. You are leading more than 100 designers and basically you're working on it from autonomous car to a lot of projects. Yeah, sorry. I'm going to be frank and harsh and we can have a fight, but I'm very clear in my mind that designers should not code. And I'll try and defend it. 10 years ago, maybe. Now when AI is going so rapid, you draw these boxes, put them colors, you press a button and the code will get ready. Automatic generation of code is already here, right? So what's happening is that technology is getting so advanced and it is so complicated that you really need to be deep into it to really be able to do anything with it. I mean, we heard Saurabh's talk in the morning. I mean, if you want to create personalities and actually give those kind of algorithms into that code, you really need to be very good at it. And you cannot be both ways where you're also doing a bit of design and a bit of coding. The other thing that is happening is like I said in my earlier remark, the gap between what people expect from the technology and what the technology can do, the gap between what they think they understand of the technology and what the technology can do has increased vastly. Now when you open your car bonnet, you really do not know what's happening inside. So the designers actually have to help the technology and the engineers so that the product can be made more usable and friendly. It means much more harder work on the coding side so that you reduce the load of trying to sort of what the end consumer is supposed to do. And my opinion is that technology is getting so complex that you cannot have both the hatch at the same time. Okay, what about you, Jeff? No, definitely I think a certain degree of understanding the process is absolutely critical for the engineer. They may not really need to know the nitty gritties of how to code, configure, and things like that. But what we've seen, for example, the biggest problem what we had when we were building a commodity trading platform was not how the screens would look, but frankly how the user was looking at naming a port. You know, the port name was called Vladivostok, but in Russian it starts with a W and with English it starts with a V. The bigger problems we had from the design elements were things like that, because the designer did not get into the mindset of what, who they were designing it for. But I agree, we don't really need to get down to the nitty gritties of the code, but definitely a very, very strong association with the tech team and understand what they're trying to solve will really help in bringing out a overall better experience to the end customer. That's where it's. What to add? I'm just saying that understanding has to be both ways. So even now, after so many years, I am half the time evangelizing and training or educating my customers and that, in terms of what is design. So design is not aesthetics. Design is not showing how the colors would work. The fact that a certain thing has to be worked in a certain way is where design really comes in, because that's what the solution creating is happening. And because designers are not constrained by the technology difficulties that are up there behind it, they can really imagine a solution that is really the ideal solution. And then we can figure out how to really get to that and what is possible in stage one and stage five. So I feel designers can actually think of a complete solution, and partly also they have to keep, but I agree with you, they need to understand at least what's happening behind the scenes. They cannot be completely oblivious to that. I'm probably more on this side of the stage than that side of the stage. Yeah, fair enough, fair enough. I mean, I wouldn't trust an architect who had never applied a nail to a board, right? Some familiarity with the medium is critical to knowing what it can do so that you don't paint a picture that is absolutely unattainable. At IBM Design, the designers end up acting as facilitators between business and user advocacy, which we do, and development. And knowing just a bit of code, not how to launch an enterprise application, but just a bit of code and the vocabulary and the concerns of developers means that that conversation is easier, means that that facilitation is possible. So I would not advise designers to become coders, but yes, get familiar with the medium. Yeah, they need to know what's happening. Great. So let me ask you this question basically. The world is moving towards automation, right? So there is a lot of buzz in the industry, like a lot of jobs get replaced. So is it same for the designers too? Do you think designers' jobs get replaced by AI and robots in the future? Jya? I think the role of the designer becomes all the more critical today than ever before, because today we are moving more and more into a world where algorithms are dictating things about how you would buy things to where you would be getting your cab to a whole lot more. But if we don't humanize these elements properly and keep the customer in the center of what you're trying to do, because at one point I remember there was a study where if your battery of your phone was dying to about 1% or 2%, and at that point where Uber was applying a surcharge, they used to check that element that your battery was running out of 2%, and automatically placed a surcharge because there was no way you could go back and ask for another part. Now if you keep allowing algorithms to find the most optimal one, they will do what you ask it to do. If you ask it to bring it more profitability, that is what the algorithms will go out to design. And today, as designers, we have a lot more bigger responsibility because you need to humanize this whole thing. Is it ultimately only about increasing the revenue for you? Or is there a more larger impact that we as an enterprise should do? Because you can even help that person out with that lesser battery without applying that surcharge because you don't really have to do that. That's where I see the role of designers becoming more pertinent today than ever. And it's not about the worry about, will they take away jobs? We need to completely turn the question around to see, what are the additional elements? What are we, who are we designing this for? Is more relevant and that's my take on this. But yes, I totally agree with him. And see, I'll give you an example of what I think is going to happen, especially for UX or designers, careers going forward or what they're expected to do. The core humanity of understanding humans, understanding the subtleties, is something that AI will need to learn from people like us who tell them that this is what people expect and hence you have to repeat that. And I'll give you an example which is almost impossible for an AI to understand right away. So how many of you all, for example, listen to music while sleeping? Some of you, right? Have you all noticed that when you turn off the lights, the music seems louder and clearer? It's a pure perception of the human person that suddenly the sound seems better because the lights are off. There is no way an AI can understand that. And there are many, many, many such small, small perceptible nuances of humans that makes us humans. And it's only us designers going back to the core ethnography and core understanding of people is where we'll be able to understand it, quantify it, maybe make it into a platform and translate it so that some of the AI systems can then start appearing more and more human. So it's going to be our job to make this AI system more and more human by being core to understanding what people are. Let me look at it from the point of automation because again, like I said before, automation is inevitable. It's already happening. And something a lot of you would have done or would have seen done in your world which is redlining a design, right? How much time and effort has gone into doing something that probably every single designer hates doing. But everybody had to do it. Now there are tools which have totally automated that. So you don't need to really spend the precious time of a designer wasting time doing redlining and handing it off to be put into production. Just imagine that at significant additional levels where it's just inevitable that more and more parts of things that a computer can easily do whether with AI or other deep learning capabilities is secondary but the key is automation is gonna just move up the chain till how long and how much will be taken over I think is secondary. All I will say is for this audience just keep in mind that things that can be automated will be automated, it's just a matter of time. So just always move up the chain, re-skill yourself and make sure that you can think about how to step ahead of that game. Chris? I'm gonna tell a scary story. I worked with a client who had developed a system that broke online advertisements down to their atomic elements and then built what we would call a narrow AI nowadays such that it simply did A-B testing across every single place and every person that it delivered that ad to. So it would try a picture of a puppy with Helvetica and the color red with a mother of the age of 35 in Des Moines and see if she clicked on the ad. It would try the color blue and times with a financial advisor in New York City and see if he clicked on that ad. And it would slowly get better and better as following what the results were, right? Which in this case was simply clicking on the ad. So over time, the AI was doing the design of these online ads. Now the kicker, that was 10 years ago. I think as a designer your job is, well your future will partially depend on what it is you sell. If you sell that you are pixel pushing, your job is not long for this world. If you are a strategist, if you are a user advocate and a systems thinker, you're fine for the three to five years that we've been talking about. Yeah, so how you sell yourself and what you do, what you think you do is gonna be really important. So Narendra, a question to you. So what is one session you can give to the UX India audience to stay relevant and transform their skills? I sort of partly answered that. What I feel is happening with lots and lots of technology including other products that are getting easier and easier to use, that people tend to use products and create beautiful things and feel that that is the design. And they are missing the core thinking and the core skills, observation of empathy, of deep thinking that was done much more earlier when these tools are not that easy because you tended to sort of finalize your design in your head before you start turning on your computer. Now it is like you're designing on the computer. This thing of being able to be true to yourself in terms of how you think and how you design and how you are able to relate to and observe things around all of that is something that will really help you in the future because those are the ones that can be applied to any new tool that will keep coming. I'm sure he's working on the next version which will be easier and faster and will do a half a job anyway. So this thing of knowing that as a designer you're able to think creatively, able to think holistically. If you're able to sharpen that and like he said, to be able to present it in your own self and for the world in terms of how you are, that will really help you. I think it's all about storytelling.  So I don't think that will ever go away and that is what this group has to look at. You are dealing with real customers, real people. As long as we provide a very personalized solution that treats your end users as people, I think things are gonna be fine. Chris? At IBM Design we talk a lot about the loop which is about making and testing and reflecting and honestly I would give that as advice to individuals as well. To participate in communities of practice to make whether that is storytelling through videos or speculative designs or whether that's actually picking up a blue mix account and trying out one of the 15 Watson APIs but in making is where you'll learn in participating in communities of practice is where you'll get feedback and of course making a habit of just coming to conferences and reading every single day to be familiar with what's in the zeitgeist. Yeah, I think that is true. In many cases we are told when you're designing something get into someone's shoes but one thing people don't tell us is if you have to truly get into someone's shoes you need to leave your shoes out first because we try to get into someone's shoes with our shoes on and that is the way we try to empathize with people. If you truly want to build a product you really need to get into someone else's shoes and build it for them. For that you also need to keep a tab on what are the things that are coming up in the market because that can, there is no limit to how free you could be when you're designing something but you also need to know what is available out there so keep yourself abreast of what are the latest things that are happening but truly if you need to empathize leave your shoes before you get into someone else's. Cool, great. I think like it also, I have learned a lot of stuff with the great panel here. So as designers, fellow designers we have to remember that, you know, so emerging technologies keep emerging all the time, right? So we should be little frontier and actually start learning technologies. Designers, as panelists said we don't need to code as a designers do but we should understand what's coming up so that like you can understand in terms of what technology offers and what is the experience we can create, right? Or design fundamentals are still same so whatever technology is coming we are building on the top of it or looking at the empathy perspective from the experience perspective and end of the day design is all about problem solving so we are, there are tons of problems to solve and we are all, we are solving and technology actually helping us to take the momentum up, right? So, and because there's a lot of disruption happening in the industry and we should proactively start learning so as they said, keep reading a lot of stuff, go to conferences and maybe like, you know, talk to developers like, you know if you keep talking to developers you will understand the terminologies or like the problems what they are facing and we can provide and design a better solution from a user perspective and I hope you all learned along with me and I would like to thank you all the panelists for, give a big round of applause for all these great panelists. Thank you so much, sir.


Everything behind me is awful right now, but you know what, it is out of focus and we are good. Today, you guys, is an exciting, exciting day on the channel. I am finally getting around to making over my little office nook area in my bedroom.  It's kind of like a little wall cut out in the corner, and I do believe that it was actually an additional closet space because there is a clothing rod in there. However, I'm converting it into a little office editing nook station for me, and that's what I've actually used it as since I started living here. However, I didn't get around to actually making it over. I just had my desk shoved in there. I do have a little bit of a direction that I'm going to take in here, and that is because today's video is sponsored by Walmart. They have an incredible DIY and done section over on their website, which you guys is absolutely amazing. I actually wanna share it with you guys right away so I could talk about all the different features that Walmart has to offer. They actually sent me all of the supplies for today's video, and Walmart is truly an incredible website or location if you wanna head out to find the DIY supplies for your next crafting project, but their DIY and done section on their site is really built out nicely, and I'm gonna share with you guys what I got and how I am planning to transform this office now. Whether you are new to DIY or you are a DIY expert, Walmart is your one stop shop for all things DIY, especially on their DIY and done page. They have such an incredible page that basically showcases so many different areas throughout your home and how you can easily DIY these spaces to make them your dream space. Here we're in the little kitchen section here. They show different faucets, cabinet hardware, back splashes. We even have bathroom updates for just $15. They have everything from small to large items, including vanities, faucets, wallpaper, paint, the tools and supplies that you need to take on any DIY project ranging from small to large. And I love how everything is just categorized so nicely. You could see all the different projects and shop by category as well. And they also have some incredible how-to tips sharing with you guys different DIY projects and just little upgrades and updates you could do around your home. And this is what it looks like when you click into the different sections. This right here is the best DIY supply section. They even have those floor pop tiles, you guys, that I always use super retro friendly. But what did I get for this space, you might be asking. Well, I got some paint, of course. This is the Rustoleum Studio Color in the color called Stellar. And then because Walmart is my one stop shop, I had to get my paint roller kit as well. I also got this really cute pendant light, which I love and I got a new drill, a cordless one. My current one is corded, so I'm so excited by heart. And then I got some stain for a DIY project. We're gonna be working on a little bit later. And then in terms of decor, I got this really great kind of hanging planter system. And then a couple of baskets for organization and the top shelves.  And I am so excited to be working with Walmart on this project. As you can see the before of the space, it just so bland and boring. And I'm really excited to DIY it into a fun space. Whether your project is big or small, Walmart has you covered. Make sure to click my link in the description box below to check out Walmart's DIY and Done section to start creating the home of your dreams while not breaking the bank. So some of you might be thinking to yourself, Drew, why do you have this ostrich stuffed animal? Drew, why do you have this ostrich stuffed animal? And Drew, why do you have this real ostrich egg? Well, I actually have a very, very secret hidden talent that nobody has seen before, but I figured why not debut it right here on the Lone Fox YouTube channel. So I'm gonna share with you guys how I can easily run like an ostrich. And that's why I have those because my mom thinks that I am part ostrich. So here you go. All right guys, so how you do this basically is you take your hands and you put them under your legs like this and you latch them here. And then you start from over here. And you have to. Gosh. It's an ostrich. So I went ahead and cleared out the entire office area. And as you can see, it's pure white. I actually had it painted white just to go along with my room scheme because I wasn't exactly sure what I was going to be doing in here. However, I have come to a conclusion that I wanna use this color here. This is by Rustolium. It's studio color and it is an egg shell finish. And the color is called a stellar. Now it looks a little bit brighter on camera, but it is a super, super dark kind of gray toned navy color. It's kind of similar to the accent wall in the dining room. If you guys remember that wall and I absolutely love that wall so much. I always love walking past that area and kind of looking at it. So I figured, why not do the same thing in my room? So I have like a little touch of that in here. And I think the contrast of the moodiness of this color will really contrast nicely with like the light airiness of my bedroom. I also forgot to share with you guys this because I've only actually opened this once as well, but there is a little bench seat in here, which is kind of fun because you could see some old hardwood floor under there. When it comes to painting with dark colors, one of the most asked questions I get is how to prevent streakiness. And I'm just gonna tell you guys right now that is something you really cannot prevent. The only way you can fix it is with layers. And this paint in particular, the Rustolium Studio Paint from Walmart was such an incredible first coat coverage. I was actually able to cover this entire space with just two coats, which is pretty great for a dark, dark color. A lot of times dark colors, I typically feel like need about three coats to get that full on streak free finish. So I loved this paint in particular and it comes in so many great colors. And on top of that, you can literally have it shipped right to your door so you don't even have to leave, which is awesome. So guys, it has been about a week since I last saw you and there's a reason for that. I actually do not like how dark this space is. As you can see, it is just so overly moody. Like I don't know. This small little space is where I work all the time. I'm constantly editing videos in here. I'm constantly working out of here, doing voiceovers, responding to your guys' comments and something about this space just does not feel right. And I slept on it for a couple of days and I was like, that just does not look correct. And it's directly in front of my bed as well. So I was always seeing it at night being like, I just have to change that to a lighter color. Now the color itself is beautiful. Like it's such a dark, dark navy color. So if you wanted this for like a nice contrast in your home, the color and the paint is incredible. But I ended up ordering a new color from Walmart and it is from the same exact brand because I actually really like this paint. It is called Birchwood and I'll pop up a photo of it on the screen right here for you guys.  And I'm going to go ahead and I'm gonna paint this room entirely back to that color. Well, it wasn't that color to start. It was just white.  But I wanted to share with you that I do not always make the right choices. Like there is so many times where I have to go back and repaint sections or just colors don't turn out correct. And a lot of times people don't really show that on their channels or show that across YouTube. But I wanted to show you guys, I painted this dark and it's gonna be such a pain in the, to paint it back light. But it's just not working for me. Like it's just not my favorite thing. Again, the color's amazing and the paint's great. It's just, this space is so small. I feel like it needs that brightness element, you know? So let's go ahead and get to painting. I just have to pop back in one more time about this paint. Can you even believe how full coverage this first coat of paint is over the top of such a dark color? That's all I have to say. Like it truly is such an amazing paint. Good morning guys. It is day three of this makeover and that's because I spent the first two days applying six coats of paint to this entire little office nook. But I am finally at a place that I love. I just knew that dark color was just too dark for this small area. It was such a beautiful color and I'm excited to actually have that can of paint in my stash which I'll hopefully use in a future project. And I did paint a majority of it off camera just so that you guys didn't have to watch me paint for a long time but I'm gonna flip the camera around. And guys, this is the finished color in here. It is so, so pretty. It's that birch wood color that I was sharing with you guys by Rustoleum and wow, I just love it so much. It's like a grayish warm taupe tone but it's still super light to where I also went in and used the existing white color to repaint the windows and repaint all of the trim. And I just love this. I love how bright an area it is but you also still get that contrast between the wall color and the trim which I love but the first thing that I wanna do today now that it's fully painted is actually DIY the desk which I am so excited about. It's going to be very similar to this little bar console that I created a similar kind of design detailing around the edge. I'll pop up an actual wording for what that's called. So let's go ahead, head over into the living room where the desk is and get started on that project. I'm sure a lot of you guys know that I had to do a large scale DIY project and this one turned out incredible. I am so happy with it. This is my old desk that I've had for years and years. I just wanna update it and turn it into something new. So I started off by pulling off the handles and grabbing some sandpaper which you could also find on Walmart's website.  I wanna take off all of the old finish because it does have kind of a glossy finish on top. And if you need to vacuum off that sandpaper I am not going to judge you. So that's what you're gonna wanna do. Just go across and just give a nice sanding making sure to sand in the direction of the grain of wood. And I'm also gonna be using these quarter inch wooden dowels along with this Rustoleum wood stain in the color called Kona, which is just so pretty. And what you're gonna wanna do is actually cut down your quarter inch wooden dowels into two and a half inch strips or whatever height of desk you're working with.  So I actually cut out a ton of them. They ended up looking like little noodles in a bowl which is totally fine with me but they are little wood chunks. So these are actually going to be, originally I was going to nail them but as you can see here, nailing doesn't work very well. It also splits the wood a little bit. So I ended up just using a wood glue which worked just as well. I'll link below the wood glue that I used for you guys. And the glue was applied with a paintbrush. I added a generous amount on the backside. And then just like the tutorial that I did a while back for that console I'm going to be doing the same technique by using a dowel as a spacer. So after you glue one down you're going to wanna place a dowel with no glue next to it and then place a dowel next to that one with glue and that's going to be your perfect spacing. So you don't have to space it out at all. You can just use the sticks themselves as spacers and they're kind of gonna sit in place as you can see here. And then once you end up gluing down quite a few of them and it's set up for about 20 to 30 minutes you can go back and remove all of those spacer dowels and just work your way down. Now when I reached the drawer section here I only applied glue to like the middle of the dowel because as you could see the dowel actually overhangs the top and bottom of that drawer front which is totally fine because once I close the drawer it's actually going to look like a seamless front almost with some hidden drawers which you're gonna see in just a minute here but I'm working my way down applying and gluing all my dowels on and just adding a spacer in between and then once they are set up and dry I'm removing the spacer section. All right you guys I went ahead and I applied every single dowel across the front of the desk and it looks perfect. It only took about half an hour to do so and I'm letting the wood drip and I'm letting the wood drip. As you could see these pieces are actually extremely sturdy on here. I was doing a bit of research online and they did say that wood glue is actually sometimes stronger than the wood itself so I'm hoping that once this is fully dry it's just very very strong and all the pieces are mounted properly but I'm gonna let this dry for a while now then we're gonna come back and stain the top of it. Alrighty I am back and we have our stain in hand this is the color Kona and I can tell you guys right now this stain is so beautiful it is 100% my new favorite stain color and on top of that this is my new favorite stain brand as well because the stain inside of there almost is like a gel like consistency so when you apply it on top of a surface it almost gives you time to move it around and spread it as opposed to just soaking directly into the wood which is something I do not like with stains because sometimes you could apply too much in one section and then it's automatically darker than everything else but this stain really allows you to work with it and kind of spread it out and work as you go so as you can see there's a little bit of streakiness on there but you can easily go in with an excess paper towel and wipe away the extra and kind of build it up if you still want that darker color and then on the front section here with our little detailing that we added I went in with just a nice kind of dense paint brush and added all of that stain across every single strip of dowel the top and bottom and all the drawer fronts as well. I just installed the light and I am so obsessed I love the black and brass with this colorway I think it is so pretty how great does that look? And of course it does turn on and you can turn it off I am just right underneath it but whenever I'm in this space anyways I'm typically sitting and I'm working at my desk area so the light looks perfect in there I just love it it is so cute I just added the desk and I am just in love with this I think it is so cute with the color in here and the light so far everything is coming along so nicely and the way you actually open these drawers if you were curious is you just have to pull the tops here so that's all you have to do to open and close those drawers there I'm probably gonna go back and make sure that this is like stained nicely but it's pretty simple to actually open and close them I'm gonna go ahead and see you guys in the morning for all of the styling details where we're going to style all the shelves add some plants in here add my computer all of my work necessities and film the final transformation or makeover I guess the final clips of it so I will catch you guys in the morning When I woke up today I knew I wanted to do just one more DIY project because I really wanted to incorporate that blue paint color that I originally painted the entire space and I also did need a piece of artwork for the space as well so I figured why not DIY one so I grabbed a little bit of filming paper and cut it down to size and I'm using some masking tape in a variety of sizes to go in and create some geometric lines and as you could see here there is literally no rhyme or reason to this at all I'm just going in and using the tape to kind of tape off different sections and create a mask because wherever that tape is sitting the paint is not going to touch and then any of the open sections it's going to touch of course so I'm using that Rust-Oleum Stellar paint with kind of a chunky brush here to basically dry brush on that color and the reason I'm doing this is because I really want to be able to see those paint strokes because you guys are going to see in the end once I remove all of the tape that it's going to have very clean lines so I want the actual area of paint sections to be not clean if that makes sense so there's kind of a juxtaposition between these very clean edges and then also like these very kind of brush strokey sections as well so I'm going to go ahead and remove all the tape now Good morning everyone it is the last and final day of the office nook makeover and I am just loving how it is turning out so far as you guys saw I installed the light yesterday the painting is all completely done I just love the color in here and then you guys also saw that I created this really cute DIY artwork using that blue paint that I actually ended up painting over just because I really wanted that little element of like indigo in there let's go ahead and get to the most exciting part I guess you could say of any makeover which is the finishing touches and the styling I was super excited to get these storage baskets from Walmart as well because I really needed to organize a lot of my electrical items including my microphones, my headphones, my hard drives I also wanted to use one of them for more personal mementos because there's nowhere in my room that I'm able to store these items for some reason I just don't have any space for them so I ended up putting them in a basket in my little office area and I just love the way that these look up there and then I also ended up finishing it off with a couple of books and just decor elements that I've already had laying around my house and in my little office space prior Interrupting the styling just a little bit to share with you guys this incredible drill that Walmart sent me it's by Heart Tools they have an entire line of tools through Walmart but on top of just sending over the drill it came with so many little bits and accessories as well which I can so appreciate because most times you don't get this kind of stuff it even has a little level in here a box cutter, this little plier right here so all around you get all of these pieces and I actually do need to go hang up the artwork that I created so I'm gonna go ahead and use this to hang it up and see how it works we have the space in front of me and I am so happy with how it turned out it was such a small little area but I packed quite a punch in here I think and I think you guys are going to love it it is completely done and I guess I can reveal it to you guys so let me go ahead and share with you guys my brand new editing nook office space so yeah here it is guys that is the finished office nook makeover transformation and I hope that you guys enjoyed this video because it was a fun one for me I really loved transforming the space of course it wasn't my favorite painting it so many times I just wanted to share with you guys that it's not always your first design choice that can be the one that you kind of go with I always have a couple of ideas in my head of how I want to do things and then typically I narrow it down to be one of the options and this time I narrowed it down to be one that I just truly didn't really love in this space it just felt a little bit too dark I did love the color which is why I wanted to incorporate it in the artwork of course but I just did not love how dark and just like dreary and it just was not bright in there I thought the window was really going to help but overall once I lightened it up with the new color I just love the way that it looks and it just coordinates back to my room perfectly with the kind of creamy tone gray walls and also the green accents and plants and such I also do want to thank Walmart so much for sponsoring today's video and sending me so many incredible goodies to make over this space you guys they have such a vast array of DIY project supplies whether it be for your kitchen, your bathroom, your bedroom, your office, the front of your home they truly have everything you need from paints, stains, tools, furniture, counter tops, sinks, appliances, anything you need Walmart has it so I'm going to link their DIY and done section below for you guys so you can take a look at it so I hope that you guys had an incredible Christmas please let me know what you guys did in the comment section below I would love to hear and then I also hope that you guys have a great great new year's I cannot wait to see what 2021, 2021 wow how's in store for Lone Fox you guys thank you so much for all the support as always and I will catch you guys in my next one have an amazing rest of your day bye guys


Well, well, well, welcome back, good gentlemen, ladies, friends, elders, sly thee toves. This is going to be an exciting read. Really exciting. I'm sharing with you today Thomas Merton, the poet and writer of the 20th century who died relatively young in 1968 in a year where many people like him, who were public figures, who were stood made a stand against the Vietnam War, which he had recently done in a letter and over Vietnam on overwhelming atrocity. Many of these people, like Malcolm X, Thomas Merton, Martin Luther King, they were assassinated. Now you're going to say, wait a minute, Merton wasn't assassinated, he was, he died by an accident. Well, okay, that might be true, but look into it. That's not a normal kind of an accident that happens. Plus there are two versions of Merton's death. There are two different ones. Both happened in a hotel room in Bangkok and both involved a fan, but the two versions, one he was electrocuted, the other the fan hopped off of its rocker and lodged itself in his side. No one's, I haven't heard anyone try to address this discrepancy. No one in public or publishing or media or anything like that or even anybody in the Christian faith that he was a part of. So this is a great poem by Thomas Merton. I hope you've, well, you've listened to the introduction to it, a long introduction, but it's a long poem, it's about a five or 10 minute poem, and it's worth reading. It's from the collected poems of Thomas Merton right behind me there, all right, Cables to the Ace. I read you the sonnets of a blind captain.  He celebrates the division of electric communions, gold and money, tinkle like chimes on the top of a building, about to terminate itself. It is the hour of melted wagons in the night of the city. In the caves, the secret voices of bulls have nightmares. The ocean climbs into the hallways of the eye until the dawn of the mornings. And they are there, both of them, the sun and the outspoken rifleman imprisoned in wholesale mud. Poseidon eats the flags on the stage of the wars. He ponders the veroons which he loves. He considers the regulations of shipwrecked outlaws. He meditates on the seaweed business. He hears the sound of pearls. The uncertain dawn catches fire from too many people breathing. The burn is terrible.  The decorated escalator of futile questions leads to the judge who has a bench on the most informal layers of opinion, a message from the women. You can wait for us in the shadowy mirror. You make us adore the products of the stars, the dynamic Babylonian construction firms. The proper words come down the escalator from the floor where the angels have their sports games. Now they find themselves among the swallows. Plato is there with the girls. He listens to them. He encourages them. He travels incognito. Art becomes benevolent in the circuses of winter. In the valley of tears, the championship game of the smile. To the abyss, the eyes of fire, the lamps with bony wings. The escapes around the equator. The brown sun of the antipodes is equipped with arrows. It pursues the naive bird into a cablegram decorated with storms. It is time for the nerve, which electrocutes the mass flash of feeling. Feeling is recorded on a memorandum pad.  The bird rests, finally atop a thunderbolt. The ascetic beauty of the awakened arsenal. The little bearded consul of copper chews up the silent smoke. He thinks about flying buttresses, the broken hearted bridges, the goats going crazy. These are some of the children of his age, these fickle sailors. He has seen the fire of their nostrils, the flame at the bottom of the pier, the amateur. He dreams of his coat edged with blood. He walks through the targets, stuffed full of pins. He inhales the rising salt. He enters into the traffic of weddings. He is swallowed up there, stuffed with pagan songs. The red sea of the pharaohs, where the spectacle vanishes in an unemployment of fireworks. The sudden disappearance of the wise man, hieroglyphics riot in the temples of the octopi. The masked high priest walks again to the end of the rivers to search for the lost Moses.  They have understood the wisdom of the egg. They offer you patience of ecstasy. They smile at you, and intimately, at the moment of your choice, do not choose the neuter. The sold forehead cannot look at you, oh flowers. You clues of pleasure. The turtle is proud of her jewels. She stretches out her palms to show them to the rain.  You have embalmed us like silkworms. Go look for your apocalypse in the unexplored subways. I sit in my green field like a quiet diamond. I engulf the blue domain of naked air, light, and all the letters add up to this. Music is a joy invented by silence. Daisy's, a geography of little unknown girls present in the grass, charm the baby monsters. I hear the great red bell of summers, the messenger of deep times, cloud and testament. I am quiet about the ghosts of tears. I complain about the useless muscle which cries, me, the frontier, a little further. It is green clear, the summit, nothing. The little silent girls enter into the shadow by the door of the chosen that was published by a New Directions paperback 1968.


So Before we really get into this as Cynthia mentioned I have updated this I originally gave this talk two years ago at PG con For 9.4. So this is updated for 9.6 There are some code references in the slide notes I will have the URL where you can grab the presentation from it at the end In terms of the talk there's there's a lot of different Talks and blog posts and stuff that's out there about vacuum how to tune vacuum what about cost away all that kind of stuff and The one thing that I haven't seen out there and we're you know where I really wanted to go with creating this talk was something that really Walked through What the heck does vacuum actually do? I mean we all know that it gets rid of dead tuples and Put some space back in the database for us to make use of but beyond that You know, we don't we don't know what what the internals are actually doing So that's what I want to go through in this talk is is to really focus on What does the code of vacuum look like and and what is it doing? Those of you that that you know are They find this stuff interesting or have additional questions First of all, if anything's unclear because some of this gets a little technical, please just ask away No need to wait till the end But I would also Highly recommend go ahead grab the presentation. It's got references to the source code And for those of you who have never looked at postgres source code It is some of the best commented code. I've ever seen anywhere So even if you're not familiar with C code, there's no reason to be put off by that Just go ahead take a look at it what the code actually says if there's Anything that you want to dig into further? so Here's the laundry list of what we'll be talking about We'll be going into a pretty fair amount of depth Mostly about the internals will touch a little bit on auto vacuum as well So MVCC and Bruce I think Bruce gave a talk on this yesterday. I don't know how many of you saw it You could easily spend four hours just discussing MVCC But just as a as a very brief overview Postgres uses multi-version concurrency control Most modern databases have shifted to some form of MVCC Because the alternative is that you have to lock rows every time you touch the row That really really hurts performance and it becomes very difficult to administer that Administrate those systems What this means in practice is that in Postgres you when you delete a row The delete doesn't actually happen at that point the row gets marked as having been deleted and Later on vacuum has to come in and go actually clean up updates Essentially turn into a delete and an insert so an update never happens in place It's always creating a new tuple in the database And in this case I'm I'm differentiating between row is something that you would see from a query Tupel is what's actually in the table itself So eventually these old tuples do have to be removed from the table to make space for new tuples coming in and that's what vacuum does So another portion that's that's somewhat related to MVCC is the commit log the commit log tracks the status of every transaction either the transaction is in process or It committed or it aborted Multi trans multi x-axe store information about If you have multiple transactions touching a row at the same time The row header does not have the ability to handle that so as soon as you have multiple people touching the row Simultaneously we have to generate a multi x-act so that we can track everybody that's touching the row Vacuum is also responsible for removing Old data from both C log and from multi x-axe So this is this is something where sometimes people will see The C log Subdirectory which which by the way the names of a lot of these directories are changing in 10 I'm not sure what Commit log is being renamed to I do know that the PG X log directory. That's going to be called PG wall But the people have occasionally come to the mailing list saying the C log directory. It's huge Why is it so big and it's because they just hadn't been running vacuum? So because vacuum hadn't been running yet. There were transactions taking place commit log just kept growing and growing so Before we can really discuss what vacuum does we have to look at the question of what can actually be vacuum Vacuum can only remove a tuple from the storage from the table If no other transaction running in the system can possibly see that so The database will track Over time what is the current oldest running transaction? What transaction has the XID that is the farthest the transaction ID that is farthest in in the past In addition to what's happening if you have streaming replication set up The other effect that you can see here is that streaming replication replicas can also affect the oldest running XID in the system because if you have a very old running transaction on replica It may need to Access data that vacuum would like to remove So if you have hot standby feedback turned on your replicas will be providing that Transaction information back to the master and the master will hold off on vacuuming as long as you have An old transaction either on the master or on the replica The second aspect of what can be vacuumed is that In order to perform a vacuum The code has to obtain an exclusive lock on the page that contains the data so That's normally only necessary when An update or an incident or a delete is is happening something that's actually going to be Modifying the table, but there's there's actually additional considerations for for vacuuming so Normally Multiple back ends can hold a reference to a buffer and if you're doing something like a deeply a Many layers deep nested loop join for example the outer side of that Nested loop the the the outermost driver may be sitting on a single page for a very long period of time Seconds, maybe even minutes potentially even hours That is going to block the ability to do cleanup on that page so In normal operation if we're not if vacuum is not doing a forcible freeze When it comes to one of those pages It's not going to attempt to up to clean that page. It's going to skip that page This was changed. I think maybe 9.2 this was changed somewhere in the in the version 9 series previously you'd have a vacuum running and Somebody be sitting that they'd be holding a pin on the page vacuum and come along and say I need to grab the exclusive lock Vacuum would sit there and wait anybody else that came in and tried to read the page would block behind the vacuum So suddenly vacuum became very disruptive to the operation of the system So what happens now is is that vacuum will just skip over that if you run a vacuum verbose It will tell you in the output if it had to skip over Any pages because of the inability to acquire the cleanup block No, no, it just all it does right now is is hold account so when I When I first came across this and and was looking at this in 9-4 it occurred to me that You know, this could potentially be very problematic this could really prevent vacuum from being able to do its job and I started a discussion on the mailing list about Well, I honestly at this point I don't remember I either started a discussion about we should at least report it Or I maybe started a discussion about maybe we shouldn't do it. We should fix this And the discussion went back and forth and ultimately we decided well right now We don't know if this is a big problem at all You know, maybe it's an issue that should be addressed further. Maybe it's not So right now we just we don't really know so certainly Anyone that does run vacuum verbose if you run vacuum verbose and it says it's had to wait on Page locks, especially if it had to if it skipped a large number of pages We would definitely like to hear a report from you on the mailing list Because that's that's that was the that's why we added that tracking So at least now we've got some amount of visibility as to how big of an issue is this in the real world If it turns out it's a pretty common occurrence, then we would most likely look at okay Let's let's try and find some ways that we can do better in that area so the between both of these between the oldest running transaction stuff and then the the issue of the The cleanup lock What this boils down to is that long-running transactions will prevent vacuum from being effective So if you have a system that's doing a lot of updates on some tables You want to try and avoid having long running queries in that same postgres instance and it's important to note that the X IDs and the MX IDs those Happen at the instance level so you could have one database It's got a really fast churn rate and a separate database in the same instance that has long-running queries The fact that they're in two different databases doesn't matter the long-running query in database a is going to present problems to vacuum in database B so Transaction IDs and multi transaction IDs are both limited in size. They're both 32-bit integers the X IDs Transaction IDs are created When a transaction modifies data or if it has an exception handler So if you create a save point inside of a transaction That's going to generate a new X ID Where that is very easy to get into is if you have a plpgsql function that has an exception handler Under the covers to make the exception handler work plpgsql creates a save point So you need to be careful with plpgsql functions that are executed very frequently We've I've had a customer that was using a trigger for partitioning and They had overflow partitions So if if they didn't have a partition created for a particular piece of data the That would go into a special overflow partition and the trigger that was handling this Was using an exception handler to detect that. Oh, we don't we don't have a valid table That meant that every single row that was inserted was creating its own Transaction ID for the save point for the exception handler So as they're loading a billion rows into the table all of a sudden their database is shutting down to prevent wraparound So how many of you have ever experienced the wraparound shut down not fun is it? And and it always happens Friday at five o'clock in the afternoon So yes, the the long and short of this is that When the database is getting close to those rolling over it needs to do an aggressive Vacuum and that is the term that is in the code. So that's why I'm using it You'll also hear this reference is a anti wraparound vacuum In fact auto vacuum when it's doing an anti wrap vacuum. It says so in PG stat activity and You might also hear this referenced as a vacuum freeze Well a vacuum freeze is a little different all that vacuum freeze does is it sets the freeze limits to zero So that vacuum will Be as aggressive as possible and freeze as much data as possible So Similarly similar to how long-running transactions prevent vacuum from being effective extremely high rates of Updates of XID consumption for Sherlock or concurrent foreign key checks The for Sherlock and the concurrent foreign key checks. Those are the common ways that you that you need to generate an MX ID Those can cause freeze problems Freezing I want to be careful because freezing is not quite the same thing is anti wraparound Freezing and by the way all that freezing means is that when you take a tuple and you freeze it the X-min The the inserting transaction that created that tuple that used to be a real transaction ID It's simply set to a special transaction ID, which I think is two And that that special transaction ID. That's the fruit. That's a frozen transaction ID That means that that data is visible no matter what your transaction ideas Yes, it will remain that way forever until you update or delete the tuple When you update or delete the X-min will stay marked as the frozen But now you have an X max that's saying. Oh, well as of this transaction this tuple is no longer valid And then at some point vacuum will come and remove that tuple So there's vacuum freeze min age and vacuum multi x-act freeze min age control how old a X ID or an MX ID has to be before they get Before they might be frozen before they may be set to that special frozen X ID The freeze table age gucks settings those control how often an aggressive vacuum happens so as vacuum is running if if we come across a tuple that that matches these conditions either of these conditions if Vacuum can it's going to go mark the tuple is frozen, but it's not going to be paranoid about. Oh, I have to go do this these limits These control when auto vacuum or regular vacuum is going to go into paranoia mode So once a table is old enough that it hits one of these limits That's what's going to trigger an aggressive vacuum And that vacuum is going to very assertively go through the entire table and Make sure that every tuple that needs to be frozen is frozen. Now when it does that it's When it does that it's going to be using these limits to determine which tuples to actually freeze So there's there's a difference between these two settings And and similar to how long-running transactions cause problems. Well, they also prevent freezing so sounding like a broken record, but heap-only tuples were added in I think it was 9.0 Because vacuum is is Really expensive. We wanted a way to try and avoid it and that's what heap-only tuples do So if you perform an update and the update does not change any index values on the row and in this case an Index value means obviously if the column is referenced in the index or If the column is referenced in a functional index for example, if you've said create index blah, blah, blah on Lower some text field and lower so that the index is actually on the lowercase value That text field is obviously referenced in the index Finally indexes you can have partial indexes where you have a column referenced in the where clause That is a reference column in the index. So if you are performing an update and The update does not touch any columns that have been That are referenced if it doesn't modify that the update statement can actually say oh Set this index column equal to this index column that doesn't matter what matters is that the value itself did not change So if the value doesn't change on any index columns, there's no need for the update statement to create rows new rows in the indexes Those new rows in the indexes are What makes vacuum so painful? So basically because new rows didn't get put into the indexes the old row a new row will be created in the heap But no indexes will be pointing at that row. They're only going to be pointing at the old row that means that So so that creates a hot heap only tuple tuple Those tuples when they become old enough that they could become back. They could be vacuumed That can actually be done by just a regular plain old insert update delete statement that just hits the page and goes Oh, I I need more space So when those commands run They're first going to check to see can I free up space on this page just by removing old hot tuples When that happens vacuum just never gets Involved it it has nothing. There's there's no need for it to be involved in that process. So What that boils down to is avoid referencing heavily updated columns in indexes So vacuum itself There's four major variations. There's auto vacuum and manual vacuum and then there's With manual vacuum you can do a vacuum full which just completely rebuilds the table from scratch It's it that will actually just go and rewrite the table It's not really a vacuum. It's it's basically Create table as select from table Switch the names have all the indexes in place. That's that's really what vacuumful does. So we're not going to talk about that anymore today Vacuum is just a regular manually run vacuum and then Vacuum freeze if you remember those age limits that I was talking about a second ago What vacuum freeze does is it sets all those limits to zero? So it just says be as aggressive as you possibly can about Freezing this table and forcibly Read the entire table to make sure that everything is frozen. Yes. Yes So auto vacuum generally doesn't need to be tweaked Nowadays It's It's pretty good on its own for a moderate size database If you have a database, however, that is very large has a large number of tables You should certainly look at Increasing the number of auto vacuum workers There's some others tweaks that you might have to do My colleague Scott me it actually has I don't think he's giving the talk at this conference But I'm pretty sure you could find it on Google He's got a presentation about how to go through and basically tune auto vacuum to be as optimal as possible But for a lot of medium small medium databases auto vacuums default settings are going to do pretty good One thing I want to specifically warn people about I've seen Many customers. They'll see an auto vacuum running. They're like, I don't want auto vacuum running in the middle of the day So they'll set auto vacuum nap time to 24 hours Well, that just means that auto vacuums not going to do anything except once every 24 hours But you have no way to control when that 24 hour clock starts and Generally speaking it's going to that timer starts when you restart the database, but over time. It's going to slowly shift So first of all, it's not really doing what you want it to do second of all, there's other problems that you're going to run into because that check if that's Only happening every 24 hours. There's other back-end processes that should be running more frequently that now really aren't Running so that's really just not a good a good thing to set up If you do have a Workload that has you know high points and low points like maybe every weekend is slower or maybe overnight is slower It is very very helpful to run a manual vacuum during your slow period Let it do everything that it can and then during the day when you have higher or whatever your fast your heavy period is Auto vacuum then has less work to do so auto vacuum can focus more on oh, these are tables that are really getting hit I'm going to try and keep these clean And it's not going to worry as much about other tables The other thing the other place where Manual vacuum can really be very beneficial is if you have Tables that are getting hit over and over and over again, and you know that these tables have a high rate of churn The problem that you run into with just letting auto vacuum just trusting auto vacuum on those tables is If all three of your auto vacuum workers end up vacuuming three large tables And they're going to be busy with those for hours or longer Well now your tables that see this really high rate of churn. No nothing's vacuuming them So those tables are just going to blow bigger and bigger and bigger What's worse is the index is going to blow bigger So if you have tables where you know you have this high churn rate go ahead and just set up a cron job I usually just run it once a minute which may sound really aggressive But the thing is these are tables that are small So if you have a small table with a high churn rate But you vacuum it once a minute the table is going to stay small and as long as the table is small the vacuum is going to be very very fast Vacuum full as I mentioned it it basically just completely rewrites the table One thing that you can do if if you Find yourself in need of vacuum full some people have had luck with PG repack There's also been some complaints or certainly some complaints about it in the past So certainly be careful But what PG repack does is it will rebuild your table for you in the background? And it only takes a bare minimum of locks where necessary. So it's it's kind of like a vacuum full It's kind of like a vacuum full But it's it doesn't need the exclusive lock so vacuum One thing to note is that you cannot run vacuum inside of a transaction vacuum has to start its own It starts a new transaction for every table that it vacuums And it expects to be able to do that. So you cannot run it in a transaction With how postgres works that also means that you cannot run it from inside of a function There is a vacuum DB shell command. That's really just a convenience wrapper around essentially doing P sequel vacuum the one nice feature vacuum DB is that you can tell it To fire up multiple parallel vacuum processes so if If you've got a database and you just want it vacuumed as fast as possible vacuum DB is the best way to do that so Starting the the actual code walkthrough What vacuum does is for each table? It's going to scan the table itself scan the heap and as it's scanning when it finds dead tuples It puts those into an array and memory After scanning the heap it scans all the indexes and the indexes go and remove Their reference to those dead tuples After scanning the indexes it goes back to the heap and removes all those tuples that have now No longer have any reference from indexes And then finally if the analyze option was specified It will run analyze Once that's done once it's handled every table it's going to handle it will take a look and see if it can update the Columns in PG underscore database that frozen X min and dat min X I dat min Mx ID Those those are the columns that get checked for are we in danger of a wraparound so keeping those up to date is obviously important One big gotcha With vacuum so remember that vacuum first scans the heap and it makes this list of oh Here's all the dead tuples Well if maintenance work memory is set small or if it just isn't large enough that means that any vacuum Will have to scan through the heap and then once it runs out of space to remember dead tuples It has to stop scanning the heap scan through all the indexes Go back to the heap Remove the dead tuples and then it can start looking for more dead tuples So if you have a really large table with a bunch of dead tuples, and you have a really small Maintenance work mem you can end up scanning through the indexes over and over and over again Just trying to remove this work The the only bright side to that is that it's once Once it's gone and remove the dead tuples from the row Well, at least it's gotten that far, but this is something to be careful of It does not explicitly tell you that but if you carefully read the output you'll see So on a normal vacuum it will talk about the indexes doing the scan and then it's Does a summary at the bottom of what was actually done on each index So you see each index mentioned twice if you see an index mentioned more than twice That means it had to do multiple passes so The the vacuum command when you when you actually call vacuum itself It's going to be entering the vacuum function. That's That's what I just described here It for each relation then calls vacuum rel Vacuum rel will vacuum a single relation whether it's a materialized view a regular table or a toast table I Actually, I take it back vacuum It well, so vacuum rel it depends on whether it's it's called by auto vacuum You can manually vacuum a toast table normally there's no reason to But in any case vacuum roll will handle both a When so when vacuum rel vacuum rel needs to get a lock on the relation itself Normally, it's just a share lock unless it's a vacuum full If it's an auto vacuum and If it's non-aggressive Then if vacuum rel cannot get the lock on the table, it's just going to skip it So what that means is that if you're doing something like an altar table command and Auto vacuum happens to fire up and go I'm going to vacuum this table auto vacs not just going to sit there and wait It'll say oh, I can't get the lock move on So vacuum rel it does a bunch of mundane setup work And then it calls either cluster rel for vacuum full or it calls lazy vacuum rel once those calls return Vacuum rel will call itself to vacuum the toast table unless you've told vacuum not to do that It will not do that for auto vacuum because auto vacuum always handles toast table separately lazy vacuum rel is is where the Kind of the the work really starts happening. So the first thing that it needs to do Is it determines the XID and MX ID limits? These are the limits at which first of all What is the newest XID that I can possibly consider visible everywhere? So that's You know anything that's been to the any dead rows that are older than that transaction They can be removed. It also sets up the freeze limits So based on those settings that I mentioned earlier. It's going to determine any Any tuple that's not dead, but it's insert XID is older than This limit it's it's going to get frozen Once it's determined those limits, it's going to scan through the heap Using lazy scan heap, which I'll get to in a second After lazy scan heap is done. It's going to If there's enough Possibility if there's a possibility of reducing the size the physical size of the table It will call lazy truncate heap to try and do that It will clean up the free space map It essentially vacuums the free space map And then finally it's going to update PG class and log statistics if necessary lazy scan heap so we've now gone from vacuum to vacuum rel to lazy vacuum rel now We're down to lazy scan heap This is where the the real meat and potatoes when it comes to the table scan comes Comes into play so for each block It's going to skip ahead if it can and if it makes sense to do so so The block skipping depends on whether the vacuum is aggressive if It's an aggressive vacuum meaning it's a vacuum that is trying to prevent That's trying to satisfy the table freeze age requirements Only blocks where all tuples are known to be frozen can be skipped So this is the vacuum freeze map Feature that was added in nine point six. This is brand new Previously before nine six. We don't track We don't have a bitmap that tracks whether a block is entirely frozen This is problematic for things like data warehouses where you can have really really large tables that are basically a Pend only so the only vacuum the only work vacuum will ever have in those tables is at the very end of the table But every time that you have to worry about an anti-rap around Vacuum is gonna say oh, I need to scan the entire table So now you've got a 10 terabyte table that vacuum is gonna slowly chew its way through well starting in nine point six We record What blocks have nothing but frozen rows and if the block is entirely frozen Even if it's an aggressive vacuum, it doesn't need to touch the block. There's nothing for it to do So that's an aggressive vacuum if it's not an aggressive vacuum Then any blocks that are all visible so we know that there's no dead tuples on the block Those blocks can be skipped as well So non-aggressive vacuum skips frozen as well as all visible an aggressive vacuum only skips frozen Oh the the other part of the of the skip logic is that? It will only skip if it's going to be able to skip ahead at least 32 blocks That's basically an attempt to not fake out the read ahead logic in the operating system So if it can only skip a few blocks ahead, it's just going to sequentially hit each block anyway So once it's once it skipped ahead if if warranted It's it's going to it checks to see how much Maintenance that says work mem. That's inaccurate. It should say maintenance work mem is Left so the array of remembered dead tuples if that's getting close to being full It's going to stop the scan hit the indexes Remove the tuples and then come back to what it was doing After that check it attempts the cleanup lock if it cannot get the cleanup lock and it's not an aggressive scan It's just going to skip the block. It'll just go on to the next block It will increment the counter that says here's how many blocks I skipped because I couldn't get the cleanup lock Once it's got the cleanup lock it's going to do page pruning this is the exact same process that hot pruning is So this is the same function that's called in an insert update delete when it gets a page and says, oh, I need The page Doesn't have enough space. Let me go see if I can free some up after that It's going to record The dead tuples now there is one exception to this dead tuple thing I've been talking about of remembering in an array list of dead tuples if a table has no indexes There's no reason to do that There's no need to go remove tuples from an index because you don't have any indexes So in that case as vacuum runs through the table It's just going to immediately Remove the dead rows from the table and continue on If necessary, it's also going to do tuple freezing It's going to update the free space map And the visibility map that actually yes, that does happen for each block. So at the very end it determines Oh, if every tuple in this block is visible Market is such in the visibility map If every tuple in here is frozen market is all frozen in the visibility map Finally after it's scanned through the table its Updates the statistics the vacuum statistics and it does the final pass through the indexes and The final pass of cleaning the heap lazy vacuum index. So this is called Lazy scan heap is what we're just talking about lazy. It calls lazy vacuum index This is where the index vacuuming happens Because postgres is extensible and allows you to create new index methods Theoretically without changing back-end code. I'm not sure if anyone's ever actually pulled it off that way But because we do have different index access methods each index method has its own requirements for what has to happen during a vacuum so The one thing that is in common is that The the indexes will have to scan through the index entirely we don't currently have a method for I Have a heap tuple. I have my row in the table Let me go find the index, you know in this index worse the stuff in the index that's pointing here We don't have that ability if we had that ability vacuum would suddenly become a lot simpler But because we don't have that the different index access methods. They just have to scan through the index Completely and every As they come across each index pointer They have to go check to see does this index row Point at a heap row that's going to be deleted The list of dead tuples it's I'm pretty sure it's a binary search because the the tuples are going to be in order in the list so So the the long and short of this is that it's very difficult to reduce the size of a loaded index This is definitely true for bee trees other index methods may not be quite as sensitive But by and large when it comes to index bloat Just don't let the index get bloated if your indexes do end up bloated your best bet is to re to do a reindex So lazy vacuum heap after After the call to lazy vacuum index has happened Lazy scan heap then calls lazy vacuum heap This is the function that actually goes through the heap table Takes the list of oh here's all the dead tuples and one by one it goes through the heap and removes the tuples themselves As it's going through and doing that as it hits the end of each page It will defragment the page and then it updates the free space map Finally there's a last step for dealing with indexes Index each index access method has the option of defining a lazy cleanup index function This is run after We know that we're done. We've we've finished vacuuming a Particular table and now if the indexes want to do anything in particular at that point they can go do it For bee tree all it does is it just simply cleans up the free space map the brin index. That's when it does the It's it's renormalization Honestly, I didn't look at the other ones Finally Vacuum will Updates if it's necessary. It's going to update PG database that frozen X ID as well as the That min Mx ID What this function does is it looks to see Have I actually been able to advance? Have I been able to change either value if it has been able to change either one of the values? It goes and truncates the commit log files It updates the So there's a separate internal copy kept in shared memory that mirrors the information in PG database. That's updated and then PG the multi-exact files are Those will get truncated, but that happens during the next checkpoint. So the checkpointer is actually responsible for that So that's everything on vacuum itself Real briefly on auto vacuum. There are two parts to auto vacuum. There's launchers There is one launcher and then there's multiple auto vacuum workers The launcher every time that it wakes up and that's controlled by vacuum nap time It looks at the list of databases and it prioritizes them by whatever database is most in need of Of transaction freezing Usually there are none then it prioritizes in by most in need of Mx ID freezing and again, usually there are none and then after that it just looks at what was the last what was the Which database have I looked at least recently? So There is no prioritizing by the vacuum launcher of oh, this is a really big database I need more workers over here. It does not work that way Multiple workers can work on the same database at once. So if you have a cluster What's kind of a common pattern is if you have a database where you only have one real database There's a couple other databases. You'll have your postgres database. You'll have template one Those will get visited But what's going to happen is it'll start off Say it visits your large database fires up a worker there wakes up again in another minute says, okay Now look at postgres fires up a worker in postgres database. Well, that worker is going to return very quickly Next time it's going to say well, I haven't looked at template one So fires up a worker there again, that's that worker is going to return very quickly.  You have three just three workers So what this means is that if you are running an instance that has a large number of databases You really need to set the nap time to be small In fact, if you're running more than say 50 or 100 databases you probably want nap time to be down in the order of one or two seconds Unlike a regular Vacuum an auto vacuum worker will self terminate actually I think it I'm pretty sure it actually just stops what it's doing on the current table But auto vacuum is Intended to not interfere with normal operations The best way to tell whether auto vacuum is doing its job if it's being effective is to Simply monitor how many auto vacuum workers are running you can see that in PG stat activity and compare that to Auto vacuum max workers if you frequently See that all your vacuum workers are in use. They're all doing something That's a strong indication that auto vacuum cannot keep up with what is happening in your system The auto vacuum worker Once it goes into a database it gets a list of the heap tables and materialized views that need to be vacuumed or analyzed After it goes through them it gets a list of toast tables that need vacuuming Temp tables are ignored For each relation it attempts to get a lock and if it can't get the lock It just skips the relation unless it needs to be aggressive due to anti-rap around After it runs the the vacuum itself it'll update That frozen X ID Sorry, it does that. It's at the very end Right before it's ready to exit and then once it's exhausted its list of work it then exits The key point here is that auto vacuum does not prioritize tables within a database So if you've got a table, it's desperately in need of vacuuming it may end up way at the back of the list So this is part of why it's very helpful to run manual vacuum as well to give Auto vacuum less work that it has to do in some of these situations So ultimately auto vacuum can become ineffective for high-demand tables if too many large tables need to be vacuumed at once It'll essentially get stuck on the really big tables and then it cannot go Visit the other tables Vacuum cost delay, it's pretty well documented c-section 1944 the critical idea here is that once you hit the cost limit The vacuum will sleep. So it's a little bit unintuitive increasing the cost limit speeds up vacuum increasing the sleep slows vacuum down That's the one thing that's a little bit tricky You definitely want to be careful. I've seen customers where they're going Oh vacuum is creating too much load on the system. So they turn vacuum down. They slow vacuum down Well, what's happening now your your system is getting bigger and bigger and more bloated and more bloated and then you get to a point where oh now I have to do an anti-rap vacuum and I'm in trouble because I just don't have the the speed to dedicate to vacuum So be very careful not to slow it down not to throttle it too much If you have a system that has very fast rights, so like if you have a raid controller with a battery backup You may well want to set the page dirty setting to be lower than page miss Because on those systems writing a page is going to be cheaper than reading a page So Enclosing long-running transactions hurt vacuum high transaction rates Those can those accelerate the need to do freezing Try to avoid referencing indexes so that you can get hot definitely make sure maintenance work mem is large enough Trying to reduce a bloated index is next to impossible So just don't let them get bloated and auto vacuum can only do so much You really want to try and give it a hand with some strategic manual vacuums Unfortunately, I am out of time. So I will be happy to stick around outside to answer any questions That is my github repository that has all my presentations. This one is just called vacuum. Thank you