Just because you Feel alone doesn't mean you are alone Just because you feel unworthy or damaged Or unimportant Doesn't mean that that is true But I know you feel it sometimes you have to Feel the feeling and ditch the story that comes with the feeling that wants to attach itself to the feeling Because the meaning you give to the feeling is Where fear steps in and gathers around the meaning we put to a feeling that we have So today I know you feel it I do too. I know we feel lots of things, but it doesn't mean That it's true So sit with the feeling and Ditch the story that wants to attach itself to the feeling


Hello and welcome to the Grand Line Review, your source for everything one piece except for today, because this is a mere announcement of the second Grand Line Review Reverie, which this time will be taking place on Sunday August 26th at 10am Eastern Standard Time, 3pm Greenwich Meantime, and sadly for us Australians once again, Midnight Australian Eastern Standard Time. For those of you who are unaware of what the Reverie is, it is a gathering of celestial dragons in the Marijois section of the Grand Line Review Discord server to come and discuss, well, whatever they please. This discussion takes place both through voice chat and text chat, so if you don't have a microphone or you're just shy, don't worry because you can still participate. To gain the celestial dragon role simply head over to the Grand Line Review Patreon and become a patron by selecting the Lieutenant or above tier. Then head over to the Discord server and you will have access to the Marijois section. From there indicate if you wish to attend in the Reverie Registration Channel. And just to reiterate, the Reverie will be taking place on Sunday the 26th of August at 10am Eastern Standard Time, 3pm Greenwich Meantime, and Midnight Australian Eastern Standard Time. I look forward to chatting to you all once again. This has been the Grand Line Review, and I'll see you next time.


Welcome folks, we at the Dow Industries finish up $6.52, the Nasdaq up $4.60, S&Ps up $112. Gold contract down $46, trade in $19.96, you get silver down $0.87, $26 an ounce, light sweet crude up $14.50 at $109.50, notes and bonds at $108.76, trade in $126.20, the 30-year full point plus 8 ticks at $155.26 and $Kingdollar. $Kingdollars down $106.67 ticks at $97.996, Euro at $110, yen at $115, the British pound at $131 to $1.00 USD. We get out and we take a look at the S&P first, what you're going to see, you get an expansion of price, you get a contraction of volume, did $105 million out here today, went downtown, yes they went $164 million, you're going into $137 million and we'll see whether this buy can even make it to $4.33 million, that's only five points where we are, my take is that we're not going to get over the swing point of $441 million. NDX100, same type of setup inside the NDX, pre-cues, the pre-cues actually had a bigger contraction of volume, $59 million shares, we get downtown with $116 million, bottom line, your swing point up here on the cues of $350 or $334, my take is that we'll try to get up to about $341 million, maybe you're up $11.64, yeah maybe six more bucks and then you're going to be right back downtown, what you've had here, if I put this on a weekly now what you're going to see is this, is that on the weekly, we had come down hard with that high volume low that hasn't been tested yet, that's $318 million, that's $381 million, well we already got $265 million and we're on Wednesday, that is saying that we're going to have higher volume going into a low volume high, that is saying see ya, don't want to be up, gold, gold contract out here, they whacked the gold contract out here today, it comes down with 327,000 contracts, bottom line though, guess what you're already in an ABC structure on the way up, yesterday we went up with $427, down on three, 27 and let me just, was it $427, $448 actually, $448 and gold is in a confirmed ABC structure on the way up, the dollar, this one's going to be a little tricky one because the dollar, that could hit hard but that 104 is kind of hanging out there, you know we'll see if the dollar gets back in here at 97, 988, if you get out another 1,000 ticks you're back inside the lower range and then, then you'll see commodities really go to the moon man because the commodities have gone to the moon folks with the dollar being high, okay, this is why it's so cool this market, this market was perverted today meaning but it's, the market is always right.


Good day everybody. Dr. Sanjay Sanyal, Professor Department Chair. So this is going to be a demonstration of the aorta and the inferior vena keva. This is a supine cadaver. I am demonstrating from the right side the camera person is on the left side. This structure that we see in front of us, this is the inferior vena keva and this structure is the aorta. The aorta as we know is to the left of the inferior vena keva. Let's take a look at the extent of the aorta. This is the aortic hiatus where my instrument is pointing right now. This is at the level of T12 and the aorta divides into the two common ideagartaries at the level of L4. So T12 to L4 is the extent of the aorta. Now let's take a look at the branches of the aorta. The branches are divided into what are known as the vascular plane concept. What is this vascular plane concept? We have three unpaired visceral branches. This is the first one. This is the celiac trunk which is coming at the level of T12. The next one, this is the superior mesentric coming out the level of L1 and this is the inferior mesentric at the level of L3. So these are three unpaired visceral branches. The celiac trunk supplies the foregut, superior mesentric supplies the midgut and the inferior mesentric supplies the hindgut. The inferior mesentric continues into the pelvis as the superior rectal artery and we can see that and it supplies the rectum. The next set of branches are the paired visceral branches. The first set of branches are this. This one here which I have lifted up and this one. These are the renal arteries. In this particular cadaver, there are not one but two separate distinct renal arteries coming from the aorta. The first set are the ones which I lifted up just now. They are supposed to come out at the level of approximately the level of L1 but in this particular cadaver we see yet one more artery coming from the aorta on this side as well as on this side and they are coming separately and they are entering into the kidney on both the sides. So that is the first set of paired visceral branches. The next set of paired visceral branches are this one here. This is the left testicular artery and this is the right testicular artery which comes out at the level of approximately L2. The next paired visceral branch we can see only on the right side this one here which I have lifted up. This is the right supra renal artery. We do not see the left supra renal artery. The supra renal glands as we know receives branches from many sources. One of them is from the aorta and that is what we see here. So these are the paired visceral branches. The next set of branches are the paired parietal branches. The first set of paired parietal branches are the inferior phrenic arteries which we cannot see because we have to remove the fascia and the liver. The next set is the subcostal artery which goes below the 12th rib that also we cannot see in this dissection. But what we can see are the four lumbar arteries. To see that let me push the aorta here and we can see the branches coming from the aorta. These are the lumbar arteries. On the right side and when I pull on this side we can see branches going on this side also. So these are the lumbar arteries and they are the ones which supply the posterior criminal wall. So these are the paired parietal branches. Now let me mention some clinical correlations pertaining to the aorta. One of the important clinical correlation is aneurysm of the abdominal aorta which is usually a result of atherosclerotic damage to the tunica media. The aneurysm if it does occur it occurs usually between the origin of the renal artery and the bifurcation of the aorta into the two common iliacs. So this is the location of the aneurysm of the abdominal aorta in which case the inferior myocentric artery arises from the apex of the aneurysm. When a patient has an aneurysm of the abdominal aorta it produces an expansile pulsation sometimes which can be not only felt but can also be seen on the anterior abdominal wall especially if the person is thin. If an aneurysm as seen by ultrasound if the diameter of the aneurysm is more than six centimeters it is more likely to rupture. So therefore we have to treat it before it ruptures because after rupture mortality rate increases exponentially. In a thin person we can normally feel the pulsation of the abdominal aorta at this level where my finger is located we can feel it against the L4 lumbar vertebra where the lumbar vertebra forms a normal laudotic curve. In elderly person or as the age advances there can be calcification of the tunica media and we can feel an axial crackling which we can feel and we can feel a little bit here also. That is known as monkeybergs medial calcific sclerosis. This is a plain excerpt of the abdomen to show monkeybergs medial calcific sclerosis of the abdominal aorta and the iliacs. Superbocentric artery as it crosses over the left renal vein it can compress the left renal vein and also in a male the left gonadal testicular vein and produce what is known as the left renal vein entrapment syndrome and left testicular varicose. Let me mention some clinical applications of the abdominal aorta. The applications pertain to aotogram and angiogram. The usual route for angiogram is the femoral artery. The cannula is inserted through the femoral artery it goes through the external iliac arteries goes to the common iliac artery and then it enters into the aorta and after that under a C arm an immediate intensive fire we can cannulate either the inferior eccentric artery the superior eccentric artery the celiac trunk and get the respective angiograms. This is a selective celiac artery angiogram and this is a supramicentric angiogram to show the distribution of the vessels. We can cannulate the renal arteries and we can get the renal angiogram. This is a selective renal artery angiogram to show the segmental renal arteries and not only that we can continue the cannulation and we can do even coronary angiograms. This is a left coronary angiogram to show the left anterior descending and the left circumflex and this is an RCA angiogram to show the right coronary artery. So this is a very useful application of the aorta and its angiograms. Now let's take a look at the inferior vena keva which I told you is on the right side of the aorta. Abdominal extent of the inferior vena keva is much longer than the aorta. The inferior vena keva starts in the abdomen from its cable hiatus which is at the level of T8 on the right side and it ends at the level of L5 where it divides into the two common iliacs. However the visible extent of the inferior vena keva is much less than its actual extent. The visible extent is only from here to here. After it enters into the liver after that the inferior vena keva is not visible. Neither is it visible when it goes through the diaphragm, neither is it visible when it goes immediately after that into the right atrium. So this is the actual visible extent of the inferior vena keva. The hepatic veins which enter into the inferior vena keva inside the liver are also not visible. Tributaries of the inferior vena keva match the branches of the aorta but there are some differences. For example the inferior vena keva does not receive any counterpart of the celiac artery or the supramacentric artery or the inframacentric artery. Instead the supramacentric vein and the inframacentric vein which are the counterparts of these arteries they drain into the portal vein. There is no counterpart of the celiac artery. There is distinct renal vein and we can see the renal vein here. We can see the left renal vein is much longer than the right renal vein because the inferior vena keva is on the right side and the left renal vein receives the left esticular or the gonadal vein and it also receives the left suprary renal vein. The left renal vein is also used for splenorenal shunt because the spheric vein is situated just one centimeter above the left renal vein. The left renal vein can get compressed as I mentioned by the superior misentering artery. We can see the vein on this side also and here we can see the renal vein is arising as one trunk and it's immediately dividing into two and entering into the right kidney. The inferior vena keva receives the right suprary renal vein. However the left suprary renal vein does not drain into the inferior vena keva. Instead as we can see here it drains into the left renal vein. Inferior vena keva as I mentioned a little while back receives three hepatic veins inside the liver. However the hepatic artery actually is a branch of the celiac trunk. Now I'm going to show you something very unique. I have retracted the inferior vena keva to show the lumbar veins. There are four lumbar veins which drain into the inferior vena keva and we can see them on the right side. Fifth lumbar vein opens into the external iliac vein and that we can see here. This is the fifth lumbar vein. These four lumbar veins which we are seeing on the right side they have a communicating vein which runs up like this and it is a very thin walled vein and it is located within the fascia so therefore we have to see it very closely here itself and we can see that vein where it's being traced by my instrument here. This is called the ascending lumbar vein which has communications with l4, l3, l2, l1. This ascending lumbar vein as it goes up just below the 12th rib it unites with the subcostal vein and after that on the right side it becomes known as the azegous vein which then enters into the thorax through the aortic hiatus in the diaphragm. We have a similar situation occurring on the left side though it is not visible where instead of azegous vein we have the hemi-azegous vein which is also formed in exactly the same way and this also enters into the thorax and then at the level of T8 it opens into the azegous vein. So we can see the formation, the beginning of the formation of the azegous vein in the form of ascending lumbar vein here. This rains the structures in the posterior abdominal wall. So these are all the structures which I wanted to show you pertaining to the aorta and the entry of vena keva in the abdomen in the posterior abdominal wall. Thank you very much for watching Dr. Sanyal signing out. Give it always the camera person. If you have any questions or comments please put them in the comment section below. Have a nice day.


I know there are memories, of course you reflect your journey maybe when you're growing up. Is there a moment maybe you feel like, wow, this is me, this is Kenna talking to me directly? Yes, yes, the part particularly, pa le kwa hukas, I was once a huka.  Nika kula tiagas once. I figured this is not for me. I think I'm built out for corporate. Nika chana na yo manheno, tiagas once. Lafu nao nao tu alkuwa display apu, ki dogwa. And then manheno ya kujaribu. Kuwambia mungu, one more chance. Lemi jelu, if God was to count the chances of us from Him, my friend, I guess yes, abika. In Ichebiasu, when God was anikuwambia, janasi kuwa serious, leo nikuwa serious. So those parts of second chances for me are a big deal because I was given a second chance and I try even now in my personal life and in my social life and in my corporate life to give us many second chances to as many people as I can. So the whole idea of second chances for people, it's a very big deal, you know. And going back to your roots and saying, Dear Lord and humbled of this, for me that touched home for me because I mean most of us tumipiteo njiya. What we think is this was ghetto tumipiteo njiya. What were group of schools aja piteo. Lakin ni that for me was what held home. Right, I'm standing next to one of my boss in the industry. Of course you've been supporting creative in the industry. You're here to learn, you're here to network of course. You're here for one reason. We're here for the happy country, happy nation. How was the experience? It was totally awesome. I mean it's a fresh breath of air in terms of entertainment. We've seen things we've not seen in the industry in quite a while. Water stuck with me is a toa jahe kwa menu. And the reason it stuck with me is because Mazen jahe is one of my favourite meals. Believe it or never that I'm kikuyu. But in real I mean he's touched on our daily aspects of life. There's so much of the life he was living there that most of us have lived through and it's really nice. And I think the best part of it was the reaction from the audience. It was really awesome. I mean guys have really accepted it. I started an interview without you introducing yourself. Kindly reintroduce yourself. Oh let me introduce myself. My name is Redan Mungai. I'm the group general manager of Odibets Global. Odibets. Yes. Now I know their memories. Of course you reflect your journey maybe when you're growing up. Is there a moment maybe you feel like wow this is me. This is Kenna talking to me directly. Yes yes the part for the part particularly pale kwa hukas I was once a huka.  Nika kula tear gas once. I figured this is not for me. I think I'm built out for corporate. Nika chana na yo maneno tear gas once. Lafu nona otu alkuwa display up o kidogo. And then maneno ya kujaribu. Kwa mbi amungu one more chance. Let me tell you if God was to count the chances of us from him. My friend. I guess yes a bigger. In Isheviya subun. God was a nikuwa mbi ya janasi kwa serious. Leo nikuwa serious. So those parts of second chances for me are a big deal because I was given a second chance. And I try even now in my personal life and in my social life and in my corporate life to give us many second chances to as many people as I can. So the whole idea of second chances for people it's a very big deal. You know. And going back to your roots and saying dear Lord and humbled of this. For me that touched home for me because I mean most of us to me piteonjiya. What we think is this will get to me piteonjiya. What were group of schools out there piteya. Lakin ni that for me was what was what held home. All right now for the campaigns you guys have been doing over the Christmas. You know what were kulembuzi. Christmasi. What was she na mbuzi. Of course the one who had won a she na mbuzi sana. Lakin ni kuna otu wama on asema. There is more you guys can do than mbuzi. I don't know muna pangani ni more than that. For now tuna se matuna kufana kusinda mbuzi. But 2024 is gal law full gal law full of surprises. As well as say I mean we are back to our roots. Community is our roots for us. So we will be doing a lot of community based things for us. Growing the community. Building the community. Muhes, muhes, muhes, muhes.  Nice to see you. How are you doing? Ni kufiti. Kufiti. You are well bro. Nimefroyo kuwa nabana. Yeah, fitting. Sava ni kwa katikaji interview. He is tall in the show. Sorry. Fa wile ni kwanenda kusema sema. Kuhi para. Tuna sema beti ni jama.


But often times when we hear the word death, we connotate it with something negative. And it's important that we enter into a state of contemplation and reflection and understanding of the Qur'an and the Ahadith of Ahad-e-Bait al-Mas'alaam to really recognize the reality of it. And a couple of questions that we want to pose during the course of our conversation and our upcoming lectures, hosted by Taqadain, is number one, why is it that we need to talk about death? Why can't we just talk about life? Number two, what's the benefit of contemplating and thinking and reflecting about death? Number three, how do we make sure that our transition from this world to the next is seamless, whereby we're able to invoke the mercy and the compassion of Allah SWT? Number four, is death really something that I should fear? Upon reflecting and contemplating upon these questions and deriving Ahadith from Ahad-e-Bait al-Mas'alaam, we see that during the course of the course of our conversation, we see that during these nights and days during the holy month of Amal-e-Lan, we can be amongst those who make the necessary preparations so that we're able to reap the mercy and the proximity of God and hopefully enter into paradise in the company of Muhammad and Ali Muhammad.


Snastrunk? Hey, welcome back to Snastrunk plays Robotrack. I am diligently checking absolutely everything in this room So I can freaking find What I'm supposed to do next What is this again? Oh, yeah, let's avoid this guy. Oh wow. I thought that would be a battle It was not there is nothing in here Can I take a nap? I like it in RPGs when you can just I think Chrono Trigger does this What other games do this some other games do that where you can climb into any bed and you can replenish your health Of course, I don't have to worry about my character's health. It's the robots and Robots don't sleep Or do they? No That's a serious question your robots. I Don't know that's I don't want to get too philosophical. Let's check out Let's not edit out these and oh, this is a new enemy right here It's that illus smoke And he did not last long, but I miss the ho ho somehow and yeah, his attack is pretty good It's what sucks about these battles. You just you're just left sitting there What do I do there we go? All right level up level 10 So that should be 10 more points for each robots and I don't really have anything special to Apply let's actually before I do maintenance. Yeah, I can actually expand The hit points and then do maintenance that would be the smart thing, right? I think I guess Yeah, whatever that looks good Keep them as a kind of a balanced But with a lot of power With him I want to increase the two strengths with our guard and charge and we're good. I suppose I could Make the third robot pretty much anytime I want it to you now, but I don't know Don't really need to yet And yes, you can only make three unfortunately be pretty cool if you could make like a ton and have like you know swap them out That would be kind of like a next-generation version of this game. Oh Yeah, since I leveled up to level 10 I can go up here now To the book here if I'm gonna have to fight this guy And as well just leave it in whatever Sick of editing folks not gonna lie to you Let's just use a bomb Good get out of my way and there we go shield three I'm gonna have to apply that to give that to Leonard eventually and increase his his defense See when it oh that sucks, but when a capsule is in your range like that. I recommend going for it and And and then attacking So it looks like I got poisoned here, and if I haven't shown that off yet, it's obviously just like any other status I Don't I honestly don't remember if I've shown this off yet You go to your items like this and use clean and I'm also going to use a cure because his Hit points are pretty low So now that I learned that ability I only need one more level before I can Before I can get to learn all the abilities I can learn here Drunk a got clean good for him. I know I've said that a million times Out of my way, please Might as well just go see what's down here Nothing is down here. It feels like the front door should be right there, but it's not kind of weird get rid of that guy and Cure that's good Let's see what's in this store The hell is that thing frog robot goofy? Yeah, okay. Oh, he's talking about Rask the my ancestor or whatever and I don't think I look like him, but if you say I do and sure there's a sword too on this grave here Why not? Is there anything else in here Anything at all? It feels like there should be more in here. Why would this robot be in here?  Nothing at all really nothing, okay Whatever Sneak by this guy and there's nothing else on the floor here. I'm not sure what it the significance of that I'm not sure of the significance of a lot of things in this game If you like feeling lost Then this might be the game for you and I haven't explored this yet. It's another empty bedroom. Sweet. I think did I already go to that bedroom? I honestly don't remember It looked exactly like the other ones. I'm getting mixed up and Yeah, Larry took beating in that fight Get these things out of the way here. Oh There's another door over here cool. Oh, I'm back With the doll and all that Can I go back in?  You got to go all the way back around if you want to go back in there. God damn So what do I do now? Do I go back to bed? Is there anything new if I go back to sleep? Okay, the answer is no I'm not gonna make you watch the dream sequence again because it's the exact same thing. That's so dumb This game has its I've said this a million times already, but it was good things about this game But there's definitely flaws. That's for damn sure. Okay, so I Pretty lost. I'm not sure what to do not gonna lie to you Obviously already got that okay I've been lost plenty of times though. I remember um I went to Europe my only trip to Europe. I went to was this again. Oh, yeah, I Was uh, it was like my third day in Switzerland in Lucerne around the lake We were staying around the lake my mother and I and I was feeling pretty confident about where I was and What you know just just finding my way around I'm the type of person that loves to wander around aimlessly and in real life just like in video games But I ended up taking a wrong turn oops There must be something up here Okay, let's let this guy get out of the way. Yeah, I was feeling pretty comfortable. I decided to take a walk. I Took a wrong turn. I took the wrong turn at Albuquerque as Bugs Bunny would say I ended up on this hillside away from the city Next to this neighborhood like soccer Thing and of course the clouds darken everything comes together and Or the clouds come in it starts hailing. I'm out by myself. I'm 18 years old. I have no money I can't speak the language and or anything like that and Yeah, that's probably the scariest experience of my life to be honest with you didn't know what I was gonna do Don't tell me this dog can climb this chain down because that's the only other way you can get this dog down here Oh, come on.  That's a rip-off Well, at least I can progress with the game now That's ridiculous. Anyway. Yeah, when I'll never forget going to Asking for help these kids Oh There he goes these kids Just the look on horror on their faces I was like does anybody speak English and they were just like oh and ran in the house. It was funny Well, where are you going? Yeah, what what she runs into the wall cuz she's goofy No, uh-huh. Yeah, don't stand up too quickly there dude gonna hurt your hip again Your hip is still drunk I've received the light sounds like some kind of religious proclamation Anyway to finish the story Okay Room in the West Hall Like what do you have to say? I'll finish the story some other time. We're coming up on Okay, the West Hall gotta remember that we're coming up at the end of the episode here basically I Got lost in Lucerne, Switzerland a lady that understood English laughed at me and Let's help equip the light here and she bought me a bit bus ticket back in the town, but we'll talk about that next time Thanks for watching. Have a great interview day


3, 4, 3, 2, 1, and 5 5! Welcome back to my end of the morning That is a stretch routine Before we start working out We told you today is fitness edition And we have fitness enthusiast But they're here with us We have Ms. Samba Drum rolls for Ms. Samba Come here for To boom up people Because he learned by himself And then Zane's with Davo As a trainer as well So we tried to balance the sexes It's a man's world And a woman's world as well So let's do this As I introduced But before that Hilda Yes, good morning everybody So it's about time we get fit today Today I'm very excited For the very first time On the morning We are not sitting down In official clothes You know We're ready to jump a little bit Let's get ready for the weekend I wish I could get ready Like this for the weekend Anyway, yes So let's get ready for the weekend Let us know where you're watching us Go to New Phillip We can see you partly on Twitter Asante Sanna for your comments Make sure you do slide into our Twitter At h54channel Hashtag Social Friday Hashtag Why in the morning On Facebook at Y254 On Instagram Did I mention Instagram? Y254 on the squad channel Yeah, yeah, yeah On the STV channel 376 Signet H24 Star Times 54 African across County to county African across Please tell us where you're watching us from So we can give you a shout out Represent your county Represent your country As well So The main people today As I've introduced Zins to Anzena The ladies Ladies first Yeah, ladies My name is Brenda Samba AKA Ms. Samba I'm a fitness enthusiast By notes Alright And I'm also a trainer I train people in the gym And I also have a fitness company Basically what we do is We get contracts from companies We have a group of trainers That we work with So I have a partner Shout out to Alex, by the way We send our trainers To companies to go train For example, this is a company Right This is a corporation This is a corporation Yeah, you have employees So we send our trainers To come and train You're the employees, right? Right, today Yeah, like today Yeah So today we have gotten Pro Bono service Exactly I'm doing this Exactly Oh, yeah The name of the company Is Fit to Beats You can follow us On Instagram Fit to Beats Facebook Fit to Beats Alright Thank you Thank you I'm a trainer And I'm a trainer I'm a trainer I'm a trainer Alright I'm a Zaki Jami Alright I'm a midambo Yeah, my name is Zen Midambo I'm actually a fitness trainer By profession I'm a strength and conditioning coach Your camera is that one, John? I'm a strength and conditioning coach Specifically I do personal training I'm not best Somewhere specifically But mostly I do freelance training So mostly I come to where Your training at Or I can defy it To a specific gym that I know Alright, nice one So you're going to do home training Yeah, we also do home trainings Alright, you also do home trainings Carry what you have in the morning And I'm really looking forward to this It's been a while since I hit the gym My gym is like I'm a midambo I'm a midambo I'm a midambo I'm a midambo I'm a midambo I'm a midambo I'm a midambo You know what I mean I'm a midambo I'm a midambo And the side of town So here are some things you're curious about Yeah, before going for a walk There'll be one person coming through towards the end of the segment It's running a bit late It's interesting that Gideon talks about How men act In the quest Of trying to be great Powerful and strong They're ruining bedroom Clouds So we're going to discuss this later on But now, it's about time we get to know me somewhere in Zayn over here.  Yeah, I do personal trainings and also group classes in specific gyms. Well, the concern that people have is certification. Why should I trust you? What do you have any certification? What have you gone through? Why should I call you? Why should I entrust my health and my body to you? Yeah, exactly. The issue of certification in fitness training is such an important issue. Previously, most trainers in Kenya were not satisfied. So most gyms just took in trainers because they would see a guy's build up, he's looking strong, so they assume there is a trainer. So I'm a certified trainer. I'm a strength and condition in court specifically. I'm certified by the International Rugby Board. I'm a level 1 coach, so I'm still studying. I want to get up to level 3. Alright, so you used to play rugby? I used to play rugby. I also still play rugby. You also still play rugby? Not a lot. Alright, but not a lot as I said. Yeah, yeah, yeah, because of work now. Alright, so... What is rugby? International Rugby Board. International Rugby Board. Yeah, it's the body that runs rugby in the world. Yeah, it's like FIFA. FIFA does football. International Rugby Board. You learn things on my end of the morning. This is cool right here. Alright, so... So I'm certified by the International Rugby Board. Is it for rugby players only? It's sports fitness. And anyone can do fitness. Anyone can do sport fitness. You only transfer now it from sport fitness to guys that are not into sports. It will still work for you. It will still work for you. Put it in the lane. In the lane. Manual limits. Exactly. Intensity in your body. Intensity in your body. Yeah, in the train, you can do rugby. You can train in the village. They will come and lose weight. Yeah. You can lose weight. Alright, so I'm going to start training. I'm going to start training. Anyway, until the next time, someone is always... Let's talk about certification. Why should I trust you? Let me just... I first got into fitness when I travelled to London. So I went to London to study and I was really passionate about fitness. So my sister took me to the gym. Shout out to Linda, by the way. She bought me my first gym clothes. She took me to the gym and stuff like that. So I decided I don't want to get certified. So I did a short course while I was there at Slife Fitness Academy. So, yeah, I have to be certified because I'm working with corporates. So this is where you're going to do a proposal for the corporate people and you're not certified. Alright, so is this the first thing you should ask a trainer when you were considering to get a trainer? Is it the first thing you should consider? Are you certified as a trainer? Yeah, you have to consider if the trainer is certified because they have to know what they're doing. They have to be an expert. They mess around and make you get an injury or something. So they have to know what they're doing. You just don't go to quacks. Say no to quacks. Say no to quacks. Say no to quacks. Alright, so another thing. Times are quack out. People will say, what is the best time to work out? Is it in the morning before I go to work? Is it in the evening? Is it in the morning before you get back to work? Is it in the morning before you get back to work? There's really no research that has proven that they are best times of working out. But for me, I would suggest working out in the morning. Because when you wake up in the morning, you're active so it's better to do it when you're still active. Rather than in the evening, when you're coming from work or from your daily activities and you're tired, high chances are you won't pass by the gym. So you'd rather do it in the morning? People would not drink in the morning Exactly, you'd rather do it in the morning To avoid the issue of the night The night of the night The night of the night The night of the night The night of the night So you'd rather do it in the morning As fast as you can So I wanted to find out from you guys Let's talk about your individual daily workout routines I can imagine for you to be a trainer So what do you do in the morning The first thing when you wake up Because when you start at this segment You could see us stretching over here That's actually what we were stretching So please let us take us through your morning Workout and what you do Immediately I get out of bed In fact here you are not just supposed to To just come out of bed Like who are jumping out of bed Just jump out of bed See me, comment after See me once Like haters Yes, what is the first thing I was informed they are procedure First of all I walk to the gym After I get to the gym I run on the treadmill You have to warm up before you get into the intense workout So after warm up I go to the strength area The weights area So I like to train my chest, my back I incorporate it I incorporate all the exercises I incorporate all the exercises Chest, back Thighs everywhere But my favorite chest Back you have to do If you are working out you have to work out all the parts of your body But my favorite workout is the ab workout I really love doing the ab workout Because I like to wear crop tops And I like to show off my tummy Me too I do ab workout every day Every day I have to do the ab workout Because I have to show off What is your routine My first session My first session in the morning is always with clans Depending on what I am doing If I am doing a group class I will definitely work out in the clans If I am doing a one on one session With a clant I will try and test a bit Of what the clant is doing When I am demonstrating for them workouts I have to show them exactly what is being done But I have my specific times for working out I finish in my morning clans first And then I set about one hour For my own workouts Because maybe I want to do different things From what the clant is doing So it depends on whether you have a clant Or not I have a lot of friends of mine Who come to my channel They come to my channel And I say I work out every day Because I work out every day Is it the same for you guys Who have clans I say I am going to buy this I think I am a lifestyle fit I am a lifestyle fit I am not the type of person who has clans I have a lot of friends Exactly I have a lot of friends I am different I have clans But I have a lot of friends I have a lot of friends I have a lot of friends I have a lot of friends I have gone to court every Sunday Mostly, weekends To inform about schools I work at Mashub Glyzake Mashub Glyzake? Sina N Tigans I want to really pay people the rest Fine My workouts are based on I don't like working single muscles I would come to the gym and I would put my arms on the floor I would put my arms on the floor I would put my full body on the floor but my favourite sessions are always leg day and I would say leg day is Monday Leg day, Monday It's a story How can you bounce me on a leg day? They skip leg day They skip leg day They skip leg day They skip leg day They skip leg day They skip leg day They skip leg day They skip leg day Anyway, what do you think? Well, there's always the topic of food and diet Diet and nutrition when it comes to working out When it comes to this organic food and vegan diet What's your take on it? Most people are going vegan and for what reason? What's wrong with all the niamachama we are eating? Can you take us through your personal diet? You can't work out without the diet First of all, a vegetarian diet is a diet that doesn't have meat, fish and chicken If you are a vegetarian, you don't eat fish But there are two types of vegetarians There are some who choose to eat the dairy products The eggs and the whatnot But people tend to eat vegetarians Because a vegetarian diet is a diet that is low in fats It's low in fats and high in fiber So it's more healthy But also if you are a vegetarian, you need to know how to replace the meat So you need to know how to replace the meat with the nutritious veggies Like soya, legumes, beans You have to know how to replace But also you can eat meat Because it's proteins and proteins are very healthy You can't eat niamachama Yeah, proteins You have to eat proteins as well But there are... Hold on Like I said You have to know how to substitute If you're going strictly vegan You have to know how to substitute this diet Like if I'm going vegan Oh my God, I don't know what's happening If you're going vegan Know how to substitute Like if your nutrients are missed out Because you're not eating meat You have to know how to substitute them Because at the end of the day, your body still needs to stop Exactly So for proteins, you have to know how to eat beans, legumes, soya beans You have to know... How do they know? You have to know what? You mean what? You have to know what I'm not sure I'm not sure You only know what?  You guys are trained and certified trainers for fitness. So let's talk about how you deal with your clients. When you have a client who, I can imagine the fitness journey is not easy. You can get somebody who really wants to lose weight but they want to give up. When they get discouraged, what do you usually do to keep your clients moving? For me, first of all, I have to know what interests my client. You have to study the client, know what makes the client happy. They're coming to the gym to lose weight. You have to be their friend, you have to know what motivates them, what makes them happy. You have to build a friendship with the client because the client is buying the trainer, not the training. So you have to know how to relate with your client. So maybe if I have a workout session with a client and they want to give up, first of all, I'll remind them why they're here in the first place, like why you came here in the first place. And then we also have, for me, I have a machine. It's called a body composition analyzer. Basically what it does, it measures everything in your body. Percentage of water, your BMI, your muscle mass, your metabolic age. By metabolic age, I mean, for example, you can be a 50-year-old, but your metabolic age is 90. And you can be a 26-year-old, but your metabolic age is 60. So it simply means you're a 50-year-old, but your body is functioning like the body of a 90-year-old. So this really motivates the client. When you tell them you're 50, but now you need to get you back to your 20s. So they go really hard in the gym. So that's what I do, that's the technique that is a psychology. That's the technique to keep them motivated always. Alright, how do you keep your clients motivated? Most of the clients you will get as a fitness trainer are people that want to lose weight most of the year. So it's a journey, and you try to talk to them and explain to them what it takes. Mostly on the reading scales, the readings might not change immediately, but there are some small, small gains that they can start appreciating. Previously they were not able to go upstairs easily. Oh, appreciating small gains. But now they have changed. Maybe they had some nice clothes that were not fitting for them. On the readings they might start seeing the waistline now coming back. So he's up to the window. Yeah, hope that men's fears didn't go a long way. You're killing it. Okay, like a man. In the final one, you are weight loss.  But you have to be patient. Work hard and remain consistent. Work hard and remain consistent. All right. It's because they look good. Is it true first? And how do you handle it? Of course, what we are saying is, but now you see fitness training is a profession also on its own. I'm not saying it's a profession. That's why before we started looking at the story and the certification, we were definitely certified to have some responsibility. Responsibility. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes, yes. Yes. It's just hard to communicate to them. I think we just keep it professional. Always. We just keep it professional always.  Like, I getАs shocked that some people won't approach you and just be professional. Be professional and be anabond. Just tell them you are here to train. We are not here to have a relationship. You have to be tough. Yes, exactly. Before we can take our break. I have one final one before we can take a break. I can see Gideon Onoka is in his studio already. So before we continue this session, we can allow him to join us. I would like to ask you guys for everybody type. Everybody. The big, the small, the young, the old. What are the three most essential exercises or workouts that everyone can do and should do? Most essential, according to you. For me, I would say lunges, squats and push-ups. And you might show us lunges. This is a demonstration. Can we do that? Let's do some lunges. You have to do the right form like that. You can also just do this. You don't have to move. You can just do the stationary lunges. Then you can also do squats. We have lunges, we have stationary lunges. We have squats. We are moving on to the next exercise. Then squats, this. Try the right form. And then push-ups. Push-ups. Oh my God. You need to make one more. Wait, one. Below your shoulders. Okay. It's not working. Are you going to play rugby? One. Go. You can do this. You can put your legs like you can modify it. Yeah. One. Two. Three. One. Two. Five. Okay. It's as happy as that one. It's as happy as that one. I'll have to catch up with it later. We'll head straight to the break. Don't forget to interact with us. YT54 on Twitter. YT54 on Instagram. YT54 on Facebook. Don't forget the hashtag. Social Friday Fitness Edition. We have Mr. Onoko himself. He'll be joining us in a few minutes. Sorry. Let me finish it for you. We'll be back.


Great. Thank you very much. So hi. I'm Chris Pettis. I'm the CEO of Postgres QOxperts We're a small Postgres consultancy based in Alameda, California There's our company website. There is my personal website. That's where the slides will be after the after the conference my email address and my Twitter handle so Corruption. Yes, it will happen to you of it if you run any database of any size Eventually at some point something will happen and you'll have database corruption The good news is it's super easy to recover from corruption Step one is you restore from the last good backup and then you receive the praise of a grateful nation and you're done Right so it questions. Oh You don't have a known good backup or that's a shame Unfortunately, even good backups can have hidden long-term corruption So you might restore from the backup and it has exactly the same corruption that you just were trying to get around Or they could be too old, you know, the a three-month-old backup is probably not that super useful Or you might be hit by Postgres bugs so we're going to talk about preventing corruption finding corruption fixing corruption if you can and First let's talk about preventing corruption The Postgres is very trusting. It basically assumes the file system is perfect that everything it writes the file system will come back But just as it wrote it It can't recover from any sound bad data data, right unless you're very very lucky Nine nine point three introduced checks page-level check sums So at least you get a warning that the data that's coming back is bad So use nine three check sums if you possibly can Just remember Hardware is cheap. Data is expensive. Use good quality hardware. Make sure your hardware properly honors f-sync and 10 There's a tool that ships with Postgres that are to test this It's more common than you think that hardware does not honor f-sync. This is especially common among consumer level Built PCs and very high-level sands often do not offer f-sync it honor f-sync end-to-end because it greatly flatters their performance numbers Um Generally you want to avoid network attached devices for the PG data for the actual database and for the backups There is an unfortunate very common sort of trope that of the backups will go to MNT slash backups The problem is when something goes crazy and starts destroying files on your system The backups are just another local file as far as it's concerned and it'll destroy those as well So have that that air that little it's not exactly an air gap But a data gap between the the system and your backups make it make at least go through an scp or an r-sync and Of course backup backup backup what only exists on one drive you do not truly possess Make sure you follow the right backup protocol for your technique Backups on postgres are not super complicated, but there are multiple steps and you want to do them in order correctly every time Make sure you issue your PG start backup before doing a file system level backup And test your backups Because if you don't haven't tested your backup strategy, you don't have a backup strategy what you have is a great deal of wishful thinking One thing I like to do is what I call a prophylactic PG dump is even on a gigantic database Just PG dump the whole thing to dev null So who's running a database that's over like a couple of terabytes How many bad pages are there? Do you know? No, you don't you don't know if there any bad pages out there because it's very eat because probably every single page on that Database has not been read in a long long time The nice part about this is it reads every row in the database and if one's bad, it'll tell you It's great fight for fighting lurking corruption You know of course if you're you can save the dump file do it I mean if it's 12 terabyte database you probably don't have that lying around so that's okay One downside of using PG dump is it doesn't read the indexes So if you want you can write do some scripting around PG stat tuple extension That does read the indexes and then you can literally pick up and rattle every single page in the entire database and that's very handy So why you know what why why is there evil in the world? Why what causes corruption? So The underlying storage failure and this is this is not uncommon Sometimes sit down with your the spec sheet on your hard drives or your SSDs and do the math of how many uncorrected sectors per billion They say there will be and how many of how many sectors there are on your drive And you'll come up with a number of like 23 Which means there are 23 uncorrected disc errors on a brand-new just out of the bubble wrap hard drive Um, it could be the bad disc. It could be a bad controller controllers do go bad One of the sometimes disc controllers have do not have ECC memory in them.  Sometimes this will be an issue for example on your battery backed up controller Do you know the battery is still good? The you or you have bad RAM especially non-error correcting RAM non-error correcting RAM, you know a huge amount of a non-error correcting RAM will have a bunch of errors in it just because We live in a world where there's radio activity all over the place Sometimes the hardware is working as designed, but how it was designed was not so great For example deferred or entirely missing f-sync behavior because to flatter a band Perforate bandwidth or latency numbers. Sometimes they will not full. They'll consider The f-sync primitive will stop at the point that it's written to the local memory cache on a sand for example But it hasn't actually been committed all the way to permanent storage A network attack storage that doesn't handle detach very gracefully. This is also pretty common If you're using soft RAID there are plenty of edge conditions in soft RAID that can cause weird things to happen You know the the nightmare scenario is one part of a stripe of a stripe set doesn't attach in which case your corruption is a little The database corruption is the least of your worries And sadly post-crisis does have bugs like 9 2 and 9 3 did have a series of unfortunate replication bugs We've been free of those for a little while They're not common is the good news. Who's on 9 6 1? specifically 9 6 1 Who's on RDS at 9 6? Because 9 if you're on RDS and 9 6 you're on 9 6 1 it has an index corruption bug so that you happen like Who's running 9 6 1? Or operator error Backups that did not include critical files For example, there was a you do a backup with a dash that on with our sync that has a dash dash exclude that is star log star That that misses PG underscore X log and PG underscore C log They don't follow the backup protocol. So the backup you don't have all the wall segments for example from a backup It's amazing how many backups just forgot that you have external table spaces This is one of the reasons I tell people don't use table spaces in post-crisis unless you must because they greatly Increase the number of places you can just go wrong Or you type our MR dash RF in the wrong directory Who's done that? Come on be honest. We're among friends here. Yeah, I sure have Or it was a really easy problem and then you started to recover But it was three in the morning and you hadn't had your coffee and bad things start to happen Like you delete the wrong files to free space Like the git lab incident Where they made a bad situation worse because they thought they they copied the bad secondary over the good primary They were very honest and transparent about what happened. So kudos to them for that. There was no they were They they did not try to cover their mistakes, but yeah There was the you and the problem here was there was kind of an unfortunate situation That turned into a disaster because they were sleepy, you know And they had been fighting this problem for a long time and they did the wrong thing and it happens to us all Okay, so you've hit this corruption thing. What do you do or before you hit the corruption thing? So by good hardware, who knows what hardware their cloud services are running on? Okay, you're lucky if you do because most cloud providers don't tell you because what they want to do They because it's a capital-intensive business. So they keep their hardware acquisition is expensive And have multi-tier redundancy. So have a secondary Have have continuous good backups someplace else Make these backups and test them One thing I really suggest recommend doing is priming developer laptops and workstations from Develop from prod backups because of the backup stop working. You'll hear about it And stay up on post-cris releases. Don't defer these. It's really alarming when I go into Some place and they're running 9 3 0 You know, that's what 12 releases back minor releases back something like that Okay, so basic corruption recovery techniques First of all save all the parts stop post-cris QL. Don't just Do a file a full file system level backup and keep that backup safe May just Take a snapshot of it because you're going to start modifying things but you want a zero point to start with and Make changes methodically and document every step use a wiki use slack use a paper notebook use something But don't just keep making changes it because halfway through you'll go. What did I do again? And that could be very very bad.  Take a deep breath move slowly because a crisis a crisis is a problem Plus a plus bad planning storage space is really inexpensive So no matter how big your database is make sure you have a place to do a full file system level copy of it Either on that device or some other device ideally a different device you can mount cloud storage makes it You know cloud providers make this a little bit easier because you can always attach something new but Because you will want to make these copies Okay, the most common kind of corruption you'll run into post-crisis index corruption This will either you'll get errors out or you'll start getting queries in they're performing impossibly wrong The the first thing to do is Start a transaction and drop the index and see if that fixes the problem The nice part about it is if you're in a transaction You can roll it back and the index comes back for your session because DDL changes in post-crisis are transactional unlike every other database in the world If it did fix the problem just rebuild the index, you know commit build the new index you're done If that didn't fix it's probably not index corruption. It's something else or you've got the wrong index In post-crisis 10 they're coming in post-crisis 10 There's a new tool called am check that detects malformed indexes, which is really cool It's also available for pre 910 from Peter Geegan there It doesn't repair the corruption. It just tells you that the index is bad it will never it is designed to never report a false positive, but it is it is possible to get a false negative out of it because Corruption is an unbounded problem Bad data page usually because some errors flying out of in the post-crisis log saying or you're getting an error When you try to run a query and it says bad page You will it may be a checksum failure if you're using checksums use checksums Complaints about bad headers things like that. So can you do a PG dump of that table? Can you pick up each row and shove it out to disk? So you're you want the output there to be clean If that doesn't work, you can turn on this There's a global setting zero damage pages which you can set in inside a particular session That will if it hits a bad page it will issue a warning rather than an error and it will return a zero page Zero in post-crisis means there are it means no data So you've lost whatever was on that page, but at least you can keep going and scavenge what you can But sometimes they'll hit these really bad data pages like Postgres credit like the Postgres back-end crashes when it touches that page Can you select around them? This can be a really tedious process of course of passing starting crashing starting This is why you want to work on a copy? You can use the copy command to get just the good data out to see if you can isolate down to the bad pages Or you can create a new table from a select that selects around the bad pages If sometimes you can delete the bad rows just by CT ID Who's familiar with the CT ID column in Postgres? Sometime it when you're logged into postgres do a select CT ID from table name. You'll be surprised. There's a column You didn't know you had called CT ID. It's a unique identifier for every row in a postgres database And you can use it to identify a row if you have no other way of doing so. Please don't bake this into your application It's if I ever see an app with CT ID in it It's like and worse yet somebody say oh Kristoff said it was a okay idea So finding them you can write a little plpgsql function That iterates through them one nice thing about plpgsql is you can trap errors And so they don't automatically terminate so you can you can loop through select the row and just catch the error and Do a and emit a log line that says there was an error So just catch and log them so this is really nice technique This is very handy for finding toast corruption The toast table for those who are not familiar is a hidden table a semi hidden table You can see it easily enough if you look for it That exists alongside every table for holding large objects like big text fields and things like that the toast Structure seems to be a little more vulnerable to corruption than the main structure And so you frequently get errors out of it So this can be very handy because you actually have to pick up each row and make sure it decrypts decompresses the whole toast structure Okay, but none of that worked like Postgres won't start at all So what do we do? The problem is every corruption bites nature is a one-off and you post this was not designed to handle every possible corruption thing cleanly so Be sure you know the extent of it And be sure you can step backwards. This is why it's very important Well first there are no real recipes You do have to explore a bit on each case and remember work on a copy because you are going to be mucking directly with the binary structure of the database So First collect some data. Are there errors in the message or syslog? That are indicating in a hardware in OS problem. You know is is are you getting this huge stream of error saying? I can't read a block off the disk Is your own killer terminating back ends? That's very common Are you seeing these disc IO errors? Just try cp-r the whole thing to a dev null Can you actually read the files and which files can you and can't you read? Again, so if you have these very very bad data pages where Postgres crashes You might isolate the page and use DD to zero out those those blocks I don't give the formula here because I'm a little worried about turning this loose into the wild But that is one one possibility. It's easy enough to find the formula for how to figure out where in the file to do it If you do anything like this like turn on zero damage pages or use this technique You you need to rebuild all these index the indexes because the indexes will be pointing to data That doesn't exist anymore very common as wall files are missing and corrupt you start Postgres and it says sorry I can't find one of these wall files and Stops and what we'll refuse to start it up Or you went on vacation and you're responsible for the database server and the system ran out of this base And they called you now to say it won't start and We just deleted some log files Which ones? Yeah Yeah, this is this is a common enough problem that postgres 10 there we're changing the names of the directories They are no longer going to be called PG underscore X log and PG underscore C log They'll be called PG underscore wall and PG underscore X act specifically for this reason However, there is a tool that will that will get postgres to start up even in this case, which is PG underscore reset X log it tells the postgres, you know those wall files you need you don't need them trust me Everything's fine. Just start up, please and Read the instructions on this carefully.  You want to make sure you're pointing the right end out and If you do this, there's a very high risk of inconsistent data. It doesn't write those wall files just to amuse itself It does need them for a crash recovery. So You will end up with a database that probably has some kind of inconsistencies or missing data So check it very carefully afterwards There's PG underscore C log, which is you know, it's the this extremely important directory No one ever looks at they did they don't hear about it because it's really small It's just maintained automatically by postgres everything, you know, they don't worry about it until you get one of these It throws out an error that says it couldn't read from read from a PG underscore C log file what PG underscore C log is is it's basically an on-disk bitmap of the state of Every transaction that postgres cares about and I'm not going to go into detail about what it cares about It's not every transaction in the history of the postgres database, but it's a very large range of them So it's a bunch of 8k files with Unpleasant hexadecimal file names that's a strip then with two bits per every transaction that postgres is interested in The good news about corruption here is it's rarely subtle. It'll suddenly you'll start hearing getting these awful messages being thrown out Usually there's a missing file or a truncated file or the two most common here You can patch it you can use DD to replace it with all zeros file 00 of in this bitmap means the transaction is in progress Which means that previously committed transactions can disappear if you do this? So again be prepared to do more cleanup, but you can probably get the database started at least And the worst possible thing is one of the system catalogs gets corrupted It's giving you an error that it can't read PG underscore class It's really hard to recover from this This may be a no matter how old your backup is is what you're stuck with situation unless you call somebody like us And it requires some expert attention to do some recovery or scavenging some of these are really easy like there was a Case where PG underscore namespace which holds things like the schema names It had a it had an entry where its parent was itself All it did was break PG dump the database ran fine And so I just patched that one column in that one row and everything went fine So that one was easy however serious ones where the database won't start because of the system catalog corruption is Are very hard to recover from if post-crisis loses the system catalog It can't interpret the tuples on disk because there isn't enough in the there's no information in the tuple structure itself Out in the rows to say what what what the columns are or anything like that? That's all in the system catalog. So you at that point you have a big bag of bites Okay, so let's talk about I'm going to talk about some actual real-life things that happened to me as a consultant and how I fix them So here is the error Oh Bad see a log entry. Oh, that doesn't look good. So how did this happen? The they had a primary and a secondary and the switch started going bad It was like dropping packets and the packets were bad and the packets were failing checksums And you know one of these things memory, you know bit of memory and the switch went bad so They promoted the secondary of the secondary to be in the primary Unfortunately, the secondary was a much less capable machine in the primary and the site started falling over Because the secondary was been getting crushed by the load. This is why you don't do this, but okay The that's like telling someone who's dying of lung cancer to stop smoking. I mean, you know, you don't you have to have a good bedside man or sometimes So they initiated a beef your primary a beef your primary and then they initially they Initialized it back from the secondary just the right thing to do But then these errors started popping out when it started up That doesn't look good the problem is they ran the all this back over the same bad switch and So our sink our sink dropped some of these files. It wrote errors for them But they were in a hurry. They didn't look very carefully and of course It's our sink is emitting this giant long list of things and they didn't like scroll back and see this that oh by the way I just dropped a few C log files and Everyone was in a huge rush and they pushed the button and it brought back online The good news is they hadn't decommissioned the secondary So we were able to just pick up the missing C log files and drop them in and the system worked Yay We were very lucky that C log file was available Had things gone in slightly a different order and they had decommissioned the prime of the secondary or started Re-initialized it as to turn it back into a secondary It would have overwritten those C log files and they would have been stuck so no matter how bad a disaster is rushing can make it worse and Just make sure you're not introducing new problems in the rush to repair the old one if you're doing an arson or things like that Get out of that. I love seeing output. I don't know.  That's a bad habit. Don't do that Because you'll get a lot of output you have to grovel through to see the one error That that will cause the server to start fail Especially if you're in one of these to this extent possible minimize the amount of data you have to process in your in the human brain Okay, next one. Oh That's not good See toast toast about the toast table. That's one of these magic toast tables Just so you know, that's the rel file node of the of the table that it's associated with so if you can go query for it in PG class So a new primer was provisioned by promoting the secondary perfectly capable machine That wasn't the problem and these errors started popping up almost immediately, you know within seconds of the of it starting up But it was only one row on one table and only some queries spooky and The problem is we we isolate we could isolate easily isolate exactly the one record do a select star where primary key equals this And it works fine. Oh My god now what? So we okay fine. We'll reindex the toast table now that didn't work But if we iterated through the table did a select star from table and and picked up one row at a time that did hit the problem Hmm. Okay The problem is there were two levels of corruption There was a bad toast entry and There were two rows with the same primary key one of which had a bad toast a entry one of which didn't How could two things have the same primary key you ask? The problem is and the index scans only found the good row Just because it happened to be first in the index But a sequential scan found both and report the error So we found the CT ID of the missing row deleted it iterated through all the other corrupt rows and rebuilt the indexes Read the release notes because this was directly related to an existing bug in Postgres But the hosting provider Had an upgraded Postgres, so this is like every consultant's worst nightmare Which is You do a job you're done, and then you get this. Uh, we have a problem call Again a lot of this these these problems new primary was provisioned by promoting from a secondary Put in service in the old primary was dec was decommissioned everything looked fine for a few hours until there were missing rows and Some rows were duplicated. It was as if People are familiar with MVCC on Postgres So imagine you do it because an update in Postgres is effectively an insert plus a delete Is it inserts the new row deletes the old row, but imagine if that old row hadn't actually been deleted That's what we were seeing and There were no error messages. Everything looked fine. Could not get the thing to produce an error message Postgres bug Since fixed the problem is under very specific conditions, which this client Particularly happened to have C log values were not being transmitted from the primary to the secondary. So we were ending up with bad transaction states So somewhere marked as rolled back that whether it shouldn't be somewhere more marked as committed when they shouldn't be And it was really kind of a mess The good news is there was enough information database There are date timestamps and things like that to delete the bad rows and the old database was still available Thank goodness. So we were able to reconstruct the database from internal information inside the inside the user schema So we had to write some handcrafted scripts, and I never want to do that again. That was really painful so first don't exclude that could be a Postgres problem and Do thorough sanity checks on promoted primaries the the issue here was they just They just threw the switch and all everybody went home because it looked fine And no one checked it because the problem as soon as if you actually go in and do a bunch of checks like count star on the different tables it was really obvious there was a problem and Have you know? One of the nice parts about this is they understood it wasn't our fault You know that of that we're not we don't sell Postgres And so make sure they understand that open source software will have bugs And you can't always you sometimes have to work around the bugs because you can't always just call the vendor and get a fix Yeah, so this was a really horrible situation I get a call on a Friday afternoon from a dentist's office and The database was running on desktop hardware and the disk did not RF sync and the power went out with a UPS That hadn't been tested for a while So postgres started up correctly no errors everything was fine, but when you touch certain tables the back end crashed And these tables were central to the application. There was no getting around using them There were there was this garbage sprayed across all of these tables clearly the It was in the process of committing to of writing to these tables when the thing went down And also the sequence is interestingly enough were corrupt as well And you couldn't do a system. Why PG dump no way as soon as they hit these any of these corrupted things everything went south and You had to go through and do a touch crash recovery to find the damaged tables to find the particular parts of the tables that were damaged So I did a schema only dumped to get the blank database because system catalog was fine dumped all the undamaged tables and The good the where they were super lucky is while these tables were essential to the application They didn't change very much There are things like email templates and user interface widgets and stuff that was really not central that Didn't have a lot of important data in it things like the patient data was all in tables that were okay I Did have to do this kind of wonky transform because the old back because they'd upgraded the application since the old backup and It had a different schema, but we got it back and Monday morning dentist comes in. Oh, no, everything's fine blink blink And I had to read manually reset all these sequences from the high point in the data because the sequences were crushed So don't use desktop hardware Um, you know location is too small to have a secondary Even if you only want to run one server use something like wall e that sends backups and wall segments into the cloud Just for you for just to do it so that you have at least you have a disaster recovery backup somewhere and Even really old backups can be useful in an emergency like this had they not had that old backup. They would have been stuck so Who's seen this message without the uke's Anybody actually ever seen this message? Yeah, that's this isn't this fun So the client said there's just too many auto vacuums going on auto vacuum auto vacuum. I was with this database It's always auto vacuuming stuff So, okay, what we're gonna do is set auto vacu these parameters really high And now there are no more auto vacuum-free jobs. They stopped. We're so smart. It's great and then on Halloween Actually These wraparound warnings started appearing in the in the law where it says oh by the way, you only have X number of the X thousand were Transactions to go before shut down due to to prevent X ID wrap around but they weren't monitoring the logs Vlogs are full of these messages, but who logs at the Postgres text logs blah blah blah They did notice but it was too late because this is a really busy site in fact This is Halloween and this site was related to the election So because at this point the excitings were being used faster the auto vacuum could freeze the table Even if we so go in and start manually freezing the highest of the ones that were threatening a shutdown We couldn't freeze it fast enough with the system online So eventually they hit shut down mode and there is nothing quite so exciting as going into postgres in single user mode So what I had to do was manually vacuum the oldest tables to get the system back online and Once the system and started up we I turned loose a script that Said that did like 12 auto vacuums in parallel on the oldest tables because they had several because one of the other unfortunate things about this Database is it had over a hundred thousand tables most of which were close to shut down point so So needless to say I'm doing this and I'm on slack with the client and they're like saying It's really it's this it's that and I just say yes, there's a lot of IO being done suck it up Because there was the otherwise you just hit shut down mode again Which brings up something that I also talk about my disaster my general disaster recovery talk Which is there is that this unfortunate thing in most organizations where? Organizations have this idea that the reason that you the line engineer has not fixed the problem Is someone with a sufficiently grand title has not yelled at you and so you'll get start getting calls from vice presidents You didn't even know existed Saying why haven't you fixed the problem put your phone on silent reduce the number of input channels because if everyone's yelling at you You can't fix the problem so This is an unfortunate thing I that people will sometimes increase these auto vacuum settings because they don't want these These very expensive auto vacuum freeze issues of appearing the the thing is that's okay to do if You do manual vacuum freezes If you do manual vacuum freezes, you can't do just one just crank up the settings though The other thing to do is always monitor the logs There was plenty of time to fix this had they been looking at the logs so if the Use use log stash use something to capture errors this had been going on for days So it wasn't something that you needed to that you need an alert on for example every day run the logs through PG Badger everyone use PG Badger and Email the results to you and just go over to the events tab and click on and see what the errors are because there were thousands of these things and This also sometimes it's also interesting when you go and say why is my application generating five thousand duplicate key violations an hour This can be surprising to the app developers And don't go in and kill auto vacuum processes, especially auto vacuum freeze processes Not a vacuum freeze process is one of those in PG stat activity that says to prevent XID wraparound Think you'll deal with it later. The future is now just deal with it Okay, to summarize just remember basics use good hardware Test your backups Don't have a don't have a backup strategy that's untested Make sure you make sure you can recover and make sure you write down the recovery steps because you're gonna be doing it This at three in the morning with no coffee Stay up on the Postgres news at least subscribe to PG announce and Read the release notes the Postgres release note I people have a bad attitude towards release notes sometimes and some of them are some post not Postgres's But some are really dire because they'll bury the lead, you know, they'll say we added all you know All all save buttons are now are now green and down at the bottom will be fixed giant data corruption problem You know Postgres's release notes aren't like that. They release their The the first the first page screen or so of release notes is very valuable Monitor your log output Postgres is often telling you things. Don't ignore it and get plenty of rest because You know, don't don't try the these problems can be made much worse if you're doing it on a fuzzy head and Thank you and Questions. Yeah, it's um the build calm and it'll be on the front It's I'll do it after the conference, but it'll be oops The good news is control data is actually is actually recreatable if you need to But because it's a fairly fairly small file The that sounds like a problem with either base backup or our sync or whatever you're using choked on that one file It's I'm not aware of a bug that would do that although, you know bugs are It might have it might have died at that particular precise moment, you know at a When it for some what can happen sometimes is on a crash It will because it will it will rewrite control the PG underscore control at that moment If the crash is sufficiently bad that it doesn't get through the process It could have unlinked it but not relinked it which it can happen corrupt statistics can happen that way Also, one thing I suggest is if Postgres crashed It's not a bad idea to go look at the stats and make sure that they are not completely insane You may have lost some stat data because you know just like look at PG underscore stat that user tables and PG underscore stat user indexes because the way Postgres works is it sucks them all into memory updates them in memory and then commits them and Commits them to this PG stat file As it needs to and then when it shuts down it writes them back to the database to save them In a crash that write process could could corrupt that the stats data You may lose some stats and that's normal.  Oh, yeah, that's true. Yeah Well to find defend it works exactly as documented. Let's put it that way If you if that's your only answer do it is basically the you know is It works correctly with the information available. Let's put that way So oh and just just you know the build calm. That's where the slides will be in a day or two Any other questions? Great. Thank you very much I just I just gave this talk at Paris last week and I ran over I I have I have no explanation Thank you so much Well, there was one of the examples was The backups were the backups. Oh great. Thank you. The backers can be very old You know was is one problem Sometimes you I don't think I gave an example of this But there are times when the backups can where the corruption lives in them is the corruption you're fighting is still in the backups Because for example, there's some bit of corruption You just don't hit it very often or but then the database you you change the app or you change the database in some way And you start hitting it all the time, but the corruption has been there for months, and you just didn't know about it Oh if the system catalogs are Well, it is backed up Yeah, it's just part of the database is but if the corrupt if but you could have a corruption event Well, one of those gets deleted or corrupted in some way because they're just a set of table. They're basically magic tables in postgres They're all the PG underscore tables Probably PG underscore class is the most is the one people are the most familiar with and And PG underscore attribute which PG underscore class holds like the indexes and tables and PG underscore attribute holds the fields Those are backed up like anything else in the database. They're just perfectly normal tables Their PG dump doesn't back them up explicitly because they're recreated when PG dump runs But but but they're not you so if you you like grep through a PG a PG dump, you won't see them But that doesn't mean they're not there it's just they're automatically recreated when you do the create tables and things like that But the problem is that their Corruption on those it's hard to come up with general rules on how to solve it because it depends on how bad the corruption is What type it is? I mean we have had ones where PG like PG class was damaged to the point They couldn't be read and there's not a lot you can do there, you know Unfortunately, you know because you the postgres won't start and it's almost impossible to get it to start at that point You're looking at like bite level groveling through the data Occasionally we've had like dropped in an old PG class and tried stuff like that But that's pretty that gets pretty hairy. So yeah, but no there they're there for sure backed up, you know If you do a file system level back up, they're just part of the file system So they just they just happen if you're doing a If you're doing PG dump, it will rebuild them as part of the restore process Yeah, use it to use it to prime staging is one thing I or developer machines You do it once in a while you don't have to do it every day, but you should do it once in a while Yeah, because otherwise, you know your backup process could be broken and you just never know