Welcome to the 21 convention Miami, Florida. Our next speaker has been in the business for 38 years working with men on high intensity training. Man, that's as long as I've been alive. So some pretty cool stuff. Let's bring on stage Mr. David Landau. Thanks, man. Thank you. How many of you gentlemen out here? No high intensity training. Anybody doesn't? You don't know high intensity training. I'll explain that to you over time. It's funny, the speaker before me was talking about this wonderful, he had a wonderful nutrition talk. And if I ate like him, I probably would be twice as big as what I am today. Maybe. That's funny, on the way over here, I was taking a back roads over here from Aventura, Florida where I live. And there was, all of a sudden there was a police blockade. There was, and it detoured me away. It was an Ironman, there's an Ironman going on today. And goodness, it took me about an extra 45 minutes to get over here. Then rude, I had to go to the restroom and I decided I needed something to eat. So quite contrary to the previous nutrition guy, I ate a honey bun. I enjoy honey buns. That's basically why I like to eat them once in a while because they taste good. They're not the ultimate nutrition.  And the fellow I'm gonna talk about Arthur Jones who is the instigator of high-intensity and high-intensity principles and things of the light as far as strength training goes. I used to go visit him. And Arthur was just similar to me. Matter of fact, I used to visit him something like from the year 2000 until he died in 2007. I used to go over to his house and I went over several times. He was my mentor. He was my mentor. I had one more mentor of the late Jimmy Flowers. He's another conversation, another time. But Arthur Jones was my mentor. He was a guy that I was fascinated with. So when I used to go visit him, his nutrition was so god-awful. I wouldn't even get in there. It was like chicken fried steak and eggs and grits and bacon and all sorts of greasy foods. He really didn't care about nutrition as far as that's concerned. Matter of fact, I used to sit down the couch from him. That was my favorite spot. I would let him rule the roost as I sat with several people we brought over, including Drew Bay, who's gonna speak later on. And he used to have a big container of Hershey's nuggets. And I used to ask Arthur, can I have some of those nuggets? And I ate that too. So as far as nutrition is concerned, and with my clients, by the way, I tell them one thing. If they're interested in nutrition, I can actually graph their calories as far as that's concerned.  So as far as nutrition is concerned, if you're interested, eat less of what you enjoy. It's your life to enjoy, so eat less of what you like. And I know that's not scientific enough for you guys, but that's the way I have to look at it as far as each individual. And that's the way Arthur looked at individuals. You wanna run that DVD? Here's Arthur Jones. Think about this, a big game hunter and soldier of fortune who still carries a pistol that will kill 600 elephants, 73 men in his lifetime, but will feel sorry only for the animals. He's real, his name is Arthur Jones, a self-educated man, too powerful, too rich, and too aggressive to be called eccentric, but whose tough talk about his projects, his passions, and philosophy of life is sometimes offend the faint of heart. Here's Arthur Jones, one of a kind. Our three things of value. Younger women, faster airplanes, and bigger crocodiles. This alligator is 10 times as dangerous as one that's never been around a man. This has been coming here for the last 10 years. They've been feeding him, he has no fear of man, and this kind of alligator hurt him. This is a Western diamond back that just turned 22 months old, one of our experiments in attempting to produce a hyper strain of rattlesnake. Exercise to be of value must involve overload. And if it involves overload, exercise is not very pleasant. It's hard. If it isn't hard, it has no benefit. A bodybuilder hard at work in a bizarre contraption called a normalist machine. It's the cornerstone of Arthur Jones' fortune. Jones, a lifelong student of exercise physiology, began tickering with his invention more than 30 years ago. The first machine for sale was built in 1970. At the time, Jones was broke. He made the prototype with $2,500, borrowed from a sister. Since then, thousands of normalist machines have been sold and the business is entirely owned by Arthur Jones. He's become a millionaire many times over. There were other inventions in other times and other places. This film was shot with a camera that mounts on a device designed by Jones. He loaned the African continent in those days working as a filmmaker, mercenary, a wild animal exporter, a soldier of fortune and pilot, everything from choppers to jets, Jones' macho style and open fondness for younger women provoked the inevitable question.  No. I am a male chauvinist. I think women are wonderful. I think every man should own several. Lake Helen, Florida. Jones lives here with his fourth wife who shares his passion for nature and animals. Jones has the best of all worlds. The headquarters of his organization, his main office, is only a stone's throw from where he lives. Jones is an insomniac, so he often works late at night, at home, alone. His politics are also a lonely pursuit. Politically, I'm about 64,000 miles to the right of a pill of the hunt. Independence, Virginia, where novelist equipment is manufactured. Another plant is now building in Texas. His phenomenal financial success has forced Jones to wrestle such money problems as taxation and inheritance. You pay tax in proportion to the amount of money you make. If you make no money, you pay no tax. So far, I have made very little money because I invest it in the business. I spend it, and I would much rather do that than give it to the federal government and let them waste it. You can't eat money. You can't take it with you, and leaving it to people is a terrible mistake because it simply destroys them. It is my clear intention to leave nothing to my children.  Jones is hard. So is the regiment demanded by his exercise equipment. Thousands of institutions use novelist machines. One of those, Pennsylvania State University. Strength coach Dan Riley. Well, we're finding that we're producing a superior athlete, an athlete that's better conditioned, less susceptible to injury. We take less time to train, giving the athletes more time to study. The problem is very few athletes or coaches know how to utilize these techniques. The way that they're supposed to be used.  The key to building strength with novelist machines is pushing to exhaustion and beyond. The athlete is forced to go on until his body fails. And this discipline, muscles are made of paint. Resist, resist. Let's go. Jones is now a renewable television. I'm convinced that the video disc and exercise instructions can be marketed successfully. Jones will spend about $70 million. Transforming his Lake Helen plant into a series of ultra-modern TV studios. There'll be other programs too. One called Younger Women, Faster Airplanes, and Bigger Crocodiles. The enduring passions of Arthur Jones' life. Despite these bursts of apparent whimsy, Jones is a stern man, and enlightened self-interest is his motto. I am not involved in this business for the purpose of aiding or abetting mankind. I do things as all people do for self-interest because I'm curious, because I want to find out for my own information, I am not a humanitarian.  Self-interest. Passionate self-interest. He did that by himself. He didn't need a government bureaucracy to help him out. He did it himself. As one of my close friends that work with Nautilus told me about Arthur Jones, he was the last free man in America. Very interested. Now why would I be interested in this guy? My personal self-interest. Nobody has to tell me how to do it and which way to do it. I did research. I've done research on exercise. I've done research on history. And I've done research on, one of you guys is his favorite, Iran. And I've done research on Arthur Jones. And what a fascinating guy that is, and I guess it must be in my DNA because I once had a lecture that I had way back when on cassette tape with Arthur Jones at West Point, lecturing at West Point. When he did the West Point study, you can look that up on Google. And my father was in the car listening to it for a while. He tolerated it. And when we shut the car down, he turned to me and he says, I love that guy, Arthur Jones. So I guess it's in my DNA because he's a very fascinating study. And if you ever have any self-interest in any individuals, do your own investigation on them. I mean, there's plenty of your mentors out there. Don't go to Google. Go to libraries. And there's a lot of misinformation about Jones out there. Matter of fact, I had to defend Jones recently on a website called Iron History because they said he ruined the exercise industry. No, he actually was one of the most valuable people that ever existed in the exercise industry. And the guy was talking about how power cleans are supposedly good for sports training and football, but modelless equipment and the like aren't. Well, I'll explain it to you later why, but what I told him is power clean, you know what power cleans are good for? Anybody? Yes. They're good for power cleans. They make you better at power cleans. They have no transfer whatsoever as far as exercise effect. And where did I learn it from? Him, go ahead. We'll find this for human beings.  At great cost, Jones supports scores of chicken gorging crocodiles imported from all over the world. Why? Why do I like crocodiles? I don't know why I like crocodiles. I just do. Some people like art. Some people like women. Some people like crocodiles. I like crocodiles and women. And snakes. Poisonous snakes. Jones also nurtures a large menagerie of venomous snakes of the world. In a current project, he is scientifically manipulating growth variables to produce hyper-rattle snakes. Artificial 18-hour days push them into supersize. The snakes are fed mice and rats. But a concern of Jones and his helpers for the reptiles is so great that the rodents are killed or rendered unconscious before they're offered to the snakes, lest a snake be bitten by a rat fighting for its life. So in making money, inventing or adventuring gets dulled, Jones can always visit his exotic pets. Oh, think over.   They strike, blow out a charge of air, bite down with the fangs and expel the venom and they can get a standing man in the eyes of about 12 feet. But this is one of the few animals that has a weapon that works in a distance. Corvres are fun, but rattlesnakes are better. Their sound is so neat. And the rattle is like music to Jones' ears. During his lifelong love affair with snakes, Jones has been bitten 24 times. Nowadays, he substitutes a balloon for his flesh. Another fun thing for Jones is rounding up wild Florida alligators. Beneath all the fun and games, though, there's a dark underside. For Arthur Jones has a pessimistic view of this troubled and violent earth. Someone's asked me why I carry a pistol. A pistol is like a tourniquet. You don't need it very often, but when you do need one, you need it badly and you need it quickly. As we've seen, Arthur Jones has big plans for television. I do feel for him, but if he can handle alligators, not to mention snakes, he should do very well in our business. Bye bye. That's interesting, because it kind of tells me about Arthur as far as his dealings with people. He was interested in the good product, and he was interested in telling, being honest about the product. You don't see that much today. Guy was very, regardless of what people have read about him, because forget about what you read about the so-called experts. Especially the experts in exercise, because they have no clue. And they don't even know what exercise means for that matter, and I'll tell you later what it means. But he had a passionate self-interest, as you saw with many things. He started in 19... He started lifting weights in the 1930s, when he was a young guy. Back then, they lifted weights. They had barbells and dumbbells, and an occasional kettlebell, believe it or not, which actually was developed in this country in 1904, that is now touted as a way to train, which is almost ridiculous beyond your wildest understanding. And if you use a kettlebell, you should use it as a doorstop for that matter. It's got no value. It's a very limited tool. But he started weightlifting and found out that he read every book that he could possibly read on the subject of exercise. The early Leaderman books, Earl Leaderman, was a legend in the exercise industry. They had... He had programs along with the great Charles Atlas.  I'll get back and get the bully guy.  And there was another Earl Leaderman, George Jout. And they were all proponents of, believe it or not, high intensity exercise. So basically, Arthur probably got to start with reading a lot of their materials, because they were popular guys in that era, in the 1930s. So in the 1940s, right around 1940s, he said, you know what? I know the experts are out in California, and I'm gonna go find out if they know anything about exercise, because I'm fascinated. I have a self-interest in what exercise does, and I wanna find out. So he ventures out there, and he rooms with a guy I know out there, and it was in Santa Monica, California, first Vic Taney, Jim. He was the first one to institutionalize franchises of fitness centers, was Vic Taney. Or Vic Taney International, Vic Taney Health Clubs. And the first gym was like a virtual dungeon. But Arthur went there, and he used to go there, and believe it or not, you didn't have exercise outfits as you do today, as everybody has to have their fashion out. But he used to go there in cowboy boots, and wear his cowboy hat, because he was kind of like one of those Western cowboy types. And he used to go in there, and he'd do curls, and he'd do wrist curls with 135 pounds.  And he used to do, he used to, in order to do military presses, he used to take the bar, squat under it, and then press it up. So he did presses, curls, occasional squats with the barbell. And when they did exercises like the leg curl, they would lie on a massage table, and they would do manual resistance. They would pull on the resistive efforts of the person trying to curl their feet and legs back towards the buttocks. And they also used to do manual neck exercises back then. You wouldn't believe how far exercise techniques do go back, an exercise machinery. And that'll be another talk, another time. Exercise variable resistance equipment goes back to 1865. So there's nothing real new under the sun. So getting back to R3, he went out there and he asked the experts about exercise and he found out that they did not know any more than he did. And he was rumored with Ben Sorenson where the Arthur would disappear for a while from where they lived.  It was beautiful, crazy though. And I know you guys are excited, okay. But he used to disappear for days at a time and Ben used to say, what are you doing? So one day he brought back, he actually went out to buy rattlesnakes. So he could eventually sell them off. He bought them off some woman at this place and they all went, actually Ben went with them one day. Ben told me the story. He went with them down to this lady's place and he showed Ben he grabbed a rattlesnake by the throat, took a glass in his hand and emptied out the venom of the rattlesnake right into the glass. Well, any of you guys would do that? I wouldn't do that. I'd be scared to death. But he was fearless when he came to animals. So, let me get back to my, okay. Then he decided to go back to where he's originally from Tulsa, Oklahoma. Went to the Tulsa YMCA and they decided to train in that dungeon, okay. And Arthur, you know, developed his, he started to be interested in developing exercise equipment. He prototyped the first piece in 1948, which was a flop, which several were after that. In 1948, he made friends with an airline pilot by the name of Percy Cunningham. And Cunningham was a, you know, he was a major airline pilot and he made lots of money and it had lots of access to women. And Percy liked exercise and Arthur liked women and so they got along pretty well.  Throughout the 1950s, he ventured into Africa. He would work out from time to time. It'd be like a fad back and forth. He actually, in the early 1950s, grew so big they didn't even recognize him from, because it was from his techniques that he developed called, more or less called today, high intensity strength training. He said if four days a week were good, two were even better. So he found out the less he exercised, the stronger and more muscular he got. The harder he trained, the better he got. So he was doing that in the early 1950s off and on and in 1957 he decides he's gonna go back and do something that's never been done before. He's gonna go to Africa on the idea that he's gonna capture wild crocodiles and then bring them back to this country. That's never been heard of before, never been done before. But what, why did he do that? He did it out of what? Self interest, self interest. Or he was selfish, which Ayn Rand says that's a virtue, it is a virtue. Self meaning you and ish meaning relating the quality and character of you and what you enjoy, I'm gonna add a little package to it, what you enjoy doing.  You do things out of self interest and that's the greatest thing in the world as far as I'm concerned. That's why, again, I'm here because I have interest in telling you guys stories about exercise and defining exercise, which I will do later. So he goes and ventures in Africa and he tries to get a permit for it and they think he's crazy. What's this guy? He's coming in, is he looking for a lost diamond mind? What's he doing? Why, what do you want, what is this capturing crocodiles? Nobody's ever done that in the history of mankind, ever captured crocodiles and transported them back to the United States of America.  They had a guy follow him. Well, eventually the guy ran out of food and water and decided to join the pack or else he would have died of starvation. So they ventured in to uncharted areas that no man had ever been before or even wanted to go since. And that's where the crocodiles were. They ventured in there and they finally pulled out those crocodiles and he found a way to get them back to the United States. For what reason? To resell them. You know, that's what his business was. So eventually he taught himself, he was so fascinating about animals and all sorts of different animals and he taught himself how to run a film projector, how to put this on film. He never, he had a consultant book about it and he finally figured out how to do it himself. So forgive my reference, I believe it was the late 1950s. I have several bits of footage on this. He developed the program for TV called Wild Cargo where he would get on stage, he'd get on the set, I'm on stage. He'd get on the set and he would bring in live animals unrehearsed without knowing what they were gonna do. Poisonous snakes, different reptilians, sometimes he would bring in a cat of the Jaguar style, cat, young cubs and all sorts of things. These were unrehearsed, never seen before since is everybody has, everything's scripted today before it actually goes on TV and reality is not reality. That was reality. He was there with these animals, he didn't know what they were gonna do but he basically, they shot that and it was really a success, look it up, it's Wild Cargo, Arthur Jones, you look that up, you can actually Google that. You can look at my YouTube channel, I've got some, I believe I have some of that, if I don't I'll put some up for you eventually. It's under X archives and those out on the internet right now you can start doing your homework. So subsequently he changed the name of the series to Capture and it made it more mild and he made it to user-friendly, it wasn't like the unpredictable craziness he had before and it kind of waned. So he's back doing film, filming in Africa, continuing to do filming in Africa and the government decides to change on him which was the former Zimbabwe, it was Rhodesia at the time, it's called Rhodesia, they almost, they stole all what he had and he had to leave with his wife and kids at the time to come to this country and he landed in a little sleepy hollow section of Florida called Lake Helen and Lake Helen, Florida and from Lake Helen he went back to what he really liked to do, several things but this time he wanted to develop exercise equipment and he kept doing that throughout the 50s and 60s, he attends dozens of prototypes that flopped and failed because they weren't to his liking. He was a perfectionist, he wanted the best equipment possible to make and if it wasn't good enough for you or me, it wasn't good enough for him, he wanted to produce the ultimate exercise equipment. So he finally has something that he thinks is gonna be good, it's a big jungle gym with all the classic pieces of nautilus on it, the behind neck pull down apparatus, the pull over machine and he boils a station wagon from a friend and he basically has to cut a hole in the top of the station wagon in order for the machine to fit and come through the roof. And they finally, they drive all the way to Culver City, what's in Culver City that year is the 1970 Mr. America contest and he took it out there and he displayed that and people were fascinated. So on a way back from there, he stops at his old friend Red LaRille, I believe if I'm not mistaken, Red LaRille was a former 1960 Mr. America and he has still to this day a standalone, one of the biggest standalone fitness centers in the history of the United States. I know a few others, but I'm gonna talk about Reds and he stopped at Reds, you don't have to stop at Reds, but he wanted to sell Red what he built. Matter of fact, he stayed there two weeks, two weeks, why? To train people on his exercise equipment. He was fascinated by training people, imagine that. He wasn't making any money at it. It was about his interest in being a better person, himself and spreading that value to others. Philosophical, I know.  So anyways, he goes back home and he figures, wait, how do I get the word out? Well, he gets in touch with his old friend Perry Rader, who was the producer and editor of Iron Man Magazine, the real man's muscle magazine. And he actually, Arthur had tried for years to get articles published. He had a few published in some of the muscle, Bob Hoffman, York Barbell, I believe in couple of those magazines, but somehow he got Perry interested and they began to talk about the ultimate exercise equipment or the upper body squat called the pullover machine, I'm taking a look at my time. And those articles blew away anything that was in the magazine. The people that read the magazine couldn't wait to read the next installment of what Arthur had to say. His writing was so controversial, so impressive and so straightforward that people wanted to hear the real thing. So he starts to make a few dollars, whatever he can scrounge up out of what he's doing. And he proceeds to put ads in the magazine. And darn ads were as impressive as his articles, they were phenomenal. I mean, if you read the ads in the magazines, you couldn't wait to read the next ones. And subsequently, he started to make some money at this. Let me go backwards a little bit. 1970 was his first delivery, his first exercise equipment was November 1970. Subsequently, he has a guy by the name of Jack Feather that gives him a call. This was before the first delivery. And the guy wants all this equipment that Arthur was talking about, but he didn't quite yet manufacture it. Matter of fact, he had to go into the waste paper basket where he threw what he had on paper to pull it out and start to kind of make this order. But this guy was offering thousands of dollars for equipment. So subsequently the order went out and Feather got his machines, but who was Jack Feather? Feather was a guy that used to advertise the antithesis to Arthur, the sauna belt, the sauna wrap, all those things that used to, the rubberized suits you used to wear around your waistline used to strip away the fat, right? Sure, that stuff doesn't work. I'm sure they're still selling that stuff today. But Feather actually, he was quite a fellow, you can look him up too. So subsequently they started making money, started making money. And I believe from the years, I know from the years 1977 to 1984, they dominated the exercise industry and had most of the business.  And that's basically what he did. So you have to realize that reading that aspect and reading into that aspect that he had many, many things to do outside the exercise industry. And that fact, if I fast forward to 1984, he went back to Africa and he retrieved 63 baby elephants because he thought the African elephant was becoming extinct. So there he is firing up his 737 jet, 727, whatever it was. They fly out to Africa, they tore out all the seats in the plane and loaded it up with 63 baby elephants so he can bring it back to this country. So because he was interested in animals, like it said on the newscast, he had more interest in animals than he did human beings in some cases. So let me get to the gist of it. 1982 to 1986, they did research on exercise. And he found out that everybody was a separate entity as far as exercise is concerned. Everybody responds differently. There's no one size fits all. And I'll explain that in a little bit. And subsequently he sold the company and he moved on to what was called the Medics Corporation for Spinal Pathology to try and help lumbar problems and knee problems and shoulder problems and things in rehabilitation. And the rest is history as far as that's concerned. The legend will continue to, it will continue. Let me get to some points here. What I learned about exercise for Arthur Jones, he had a statement that I read for many, many, many years ago. And it's exercise produces nothing of value. It merely stimulates. Wow, nearly stimulates. It's a catalyst. It's a catalyst for positive change if you know how to induce positive change. That was brilliant then, but it's more brilliant now because I couldn't figure out. It took me almost 40 years to figure out what he meant. I'm so smart and I thought, yeah, yeah, that sounds good. I figured it out. Exercise stimulates, okay? How many of you are aware of, how many of you guys subscribe to a speed of movement in your exercise routine? Okay. Well, it's imposed on you, you have no choice. You guys heard of the super slow training methodology? Anybody have, you've heard of it? Basically, it's a protocol that goes back to 1962. If you have questions about that because we have some time later on, we will. I'll explain to you where it came from. But basically, it's 10 seconds to lift the weight and 10 seconds to lower the weight, okay? And me being part of that novelist history, was I've been a trainer for that many times since 1979. And when Arthur left, the clearinghouse of ideas left and Ken Hutchins took over the super slow guru. And I learned some, don't get me wrong, I learned some real good things from him, but 10-10 was always slightly a question but I was always loyal to the cause. I forgot I was an individual, I couldn't think for myself, I had to do it for the cause. And don't do that, don't be caught up in groups, think for yourself. So, seriously, think about this, this exercise is a catalyst for stimulus. Well, what the, are you doing worrying about your speed of movement? Take a look at my speed of movement, take a look at this, I'm gonna curl barbells, faster, slow, it's slow. It's about one second up and one second down.   The idea is to stimulate, well, what the, are you doing worrying about your speed of movement? Because everybody has a different physiology. Everybody has a different neuromuscular aspect in their body. Everybody has a different lever system. If I got long arms, 10 seconds, it's gonna be like, oh, you know, it's crazy. If you have longer, shorter levers, it doesn't take that in consideration. Just go 10, 10, it won't take care of everything else. Well, I used to see the people that did 10, 10. And they're better off laying on their couch because they weren't really getting much in a way of results. I didn't see much in a way, I actually, I saw.  You learn only when you learn from your mistakes, your successes merely reinforce superstition. Understand the foundations. If it's to stimulate, then why are we doing 10, 10? So that was a brilliant idea from Arthur. This is what I really like, because it basically represents my website. There must surely be a few people who are not idiots, but they follow the advice of idiots, listen to idiots, and train in a fashion that can only be described as idiotic. How many times have you seen that? Every day, if you join a gym, you're gonna see outside of what you do. Hopefully you guys are doing the right thing, whatever that's supposed to mean. But I just told you, it's dressing body. You're gonna see people doing things that are nonspecific to their muscles. So, how do you define exercise? What's exercise? Anybody have an adventure to what exercise is? What's it mean? Anybody want to take a stab at it? Go ahead. Speak up a bit, I'm a little hard of hearing. Okay. I'm gonna tell you the Landau definition. I took me a long time to figure this one out, too. Objective, individual. Anatomical, muscular stimulation. You need, when you exercise, and this, this cleans, this, I'm standing on the shoulders of Arthur Jones. I would have figured out, it would probably take me another 50 years, I don't have 50 years to live, so I was lucky to have a guy like Arthur Jones. Objective, anatomical, direct the resistance of the body. Anything outside of that can't be exercised, okay? Those that know grammar, a word that means anything means nothing at all. And if you watch the, some of this, this presidential stuff out there, you'll find out that there's a lot of meaningless rhetoric. But a word that means anything means nothing at all. So everybody exercises right, now they don't, because they're basically doing anything that's activity-based is no longer exercised, because it imposes a skill. You're learning an unnecessary skill. A perfect example of this is CrossFit. CrossFit, what's CrossFit? It's a sport.  You can't say, well, it's an exercise routine but it's a sport. No, it's not. Because you watch the competitions on TV, they're all doing activity-based, activity-based. Anything activity-based requires a skill. So once you impose a skill and exercise, it's no longer exercise. It's either an event, a game, or a sport. Well, why are they so muscular? You see some muscular guys out of CrossFit, don't you? You've seen them, right? Why? Because they're actually imposing a stimulus, no matter how radical it is, how ridiculous, and how grossly anatomical it is, it's still a stimulus. If you like exercise, chances you're doing it wrong, okay? You know, I actually am one of those few that like exercise. You like it, you probably do, too. You know why?  You're trying to enhance your structure.     So the idea is, chances are, most people don't train hard enough, okay? High-intensity training, you train hard. But remember, high-intensity training, I'm gonna redefine it for you. High-intensity training merely means you're training as hard and diligent as you can.  The idea is to stimulate, non-annihilate your body, okay? So those that, you ever see somebody who lays on the floor after a workout? You've probably seen the, oh, I'm laying on the floor. I work that really hard. I can do the same job with a baseball bat. I'll hit you with the baseball bat. You lay on the floor. It's the same thing. Seriously, okay? The idea is, okay, I'm gonna do an exercise. You guys all do the training. If you wanna know about the training, you can call me up any time. I'll tell you, you don't even have to. I don't charge you for calling me up because I'm very happy because I have self-interest in this. I have an interest in this. It makes me happy to help people in the aspect. But the idea of training, you're gonna do a set as well as you can. And the idea is to know when to say when. If it's no longer there, it's no longer there. Train as hard and diligent as you can. That's the intensity you want as far as exercising is concerned. That's okay. It's not necessarily a tool, but it's the set use of the tool.  I go back to CrossFit, of course, the untold thousands of people.  And you know what's inside the forum? You know what they discuss in there? You guys don't certainly discuss it because you're not part of it. They discuss injuries, like badges of honor. An injury part of the forum. They're so freaked out from their injuries they want to know how they can rehab themselves so they can go back and injure themselves more. Talk about what Arthur said about idiots. He was right. Train in a fashion idiotic, it's a sport. But you'll see muscles because they are stimulating, but unfortunately, those that were meant to withstand those forces you're seeing on TV, the rest of them are hurt and crippled. I'm gonna tell you a little bit about my theories and my applications, okay?    If you train hard enough, you'll know when to do it next. And if you train once a week, chances are that's not enough, unless you're a freak of nature, okay? So if you're interested in the proper dosage, occasionally you'll get the rare person that can get results from once a week. They're not common. Twice a week is what I do, but I don't know why I do it. I just do it because I basically have the time to do it. I might even do three times a week. You're gonna say, well, that's not what the book says. I threw the book out a long time ago when I watched people recover. I have a lady I train that's 72 years old. She recruits probably less than 10% of her muscle mass at any given moment. She can't recruit it, but she gets really good results. She looks good. She looks good. If I trained her seven days a week, she would still get results because her recovery's that quick. So those people say, well, it's linear. It's not.  It's up to you to determine how you recover. I mean, I can't wait for another workout, but by the time the weekend hits, I know I'm finished after two workouts a week. Another concept is vary the workout as much as possible because it really does get boring if you train with the same routine day and that day out. And that's been suggested in some of the books. I'll tell you one further. There is, like I said before, there is no ideal exercise speed. You know how I find out what's the ideal exercise speed? Is I put a person in a machine and I try and figure out from there because I have no clue. There's no way I'm gonna make that mistake again. I made those mistakes over the years. They weren't purposely made because that's the only way I learned is by observing people, training. So variety, like I said, this is not gonna hurt you. Look at this. Okay? You go to the books. I say, oh, you're yanking and drugging the weights. No, I'm not. So recovery, speed of movement, variation, and motivation, motivation. Okay? Key component. I'm real good at it. I could tell you things you wouldn't even know existed in exercise. They're not supposed to exist. Not even supposed to happen. According to what you've guys read, if you've read into some of these principles, I did a scientific experiment, my science, yeah, or lack of their which, which I trained a lady, five consecutive sets on a machine to find out how she'd recover in between the sets. Okay? And she trained as hard as she could. Okay? So what do you think happened after five consecutive sets? Did the numbers go down? Did they go up? Did they go precipitate this lead down? Guess what happened? They went up. I swear to God, I watched her train those. I did it as an experiment and I told her, this is building the instruction, that's not supposed to happen. That's not supposed to happen. Really? I said, yeah, that's not supposed to happen at any exercise routine. It may not happen for you and it may not happen for you, but it happened for her. Why? I don't know. It happened, but it's not supposed to happen. This happened more on occasions. Sometimes I said, Arthur Jones stated, within three seconds of doing an exercise to failure, and let me go over that point of failure before we run out of time in a second. Within three seconds, you'll recover generically 50% of your ability to produce force. Well, I decided to do an experiment on somebody and guess what? They were able to recover 90% of their ability to produce force and the previous example was 105 or 110% of their previous force.  Happened more than once. It makes you think, doesn't it? Seriously, don't limit your exercise program to one set to failure.  I didn't do well on that first set or I decide I need another set, don't discount it. If you're running a business that says you have to do slow speed, one set to failure, that's a business concept. Hey, listen, that's sometimes better than nothing, sometimes worse than nothing, I don't know. But the idea is don't limit your workouts. You can take what they do in the mainstream, you can condense it, you can structure it, you can make it great for you, but don't limit yourself to one set. If you feel that one set works, guess what? It works, okay? But don't limit the idea, give yourself an opportunity to do a second set. Matter of fact, any more than one set in the workout is what? Multiple sets, even if you're doing one set, the further you're doing several exercises, the body works as a whole, it rests as a whole, it feeds as a whole. So don't limit the concepts out there, the concepts. Once you start looking as an individual, some of these outcomes, it's like, Jesus, what did I miss? I missed something here, although, you know, you're always pursuing the best for you, and I'm always interested in what's best for you guys, as well as my trainees. I'm interested in what works, aren't you? Rose, I wouldn't be here. Training failure is a misleading concept. Let's rephrase that. Train as hard and diligent as you can. And here's another thing. Nobody knows what failure is because it defines itself differently in each individual case, so it always looks different. I've had people that train their maximum fatigue, and they basically, I'm walking to the next exercise, I'm walking to the next exercise, and they are right behind me, they're recovered so quickly. You can train one person to failure, land on the floor, they can't even get up for several minutes, it's so different. So one size fits all, you have to identify what works for you. So training the failures, training honestly and intelligently and diligently as you can. Yeah, it's a misleading term, train of failure. You know, it was great, but in hidden research that you guys either have seen or haven't seen, because I have copies of it in my film library, they discovered at Nautilus between 1982 and 1986 that each person recovers differently, everybody responds differently, it's a different aspect all together. See if I have anything else to go over for you guys. Oh, free weights versus machines. Really no difference. I like machines and I like free weights, I use them both. That's great to use a combination. No, I don't go by that, I don't care. I use the ones that I feel necessary to bring about physical change. I know you guys like better physiques and big muscles, I'm interested in that too as well.  All it is is they both provide resistance against movement powered by muscular contraction. The muscles don't know what you're using.  So the idea is your choice, choose. Choose wisely. Barbells are great tools, they're a miracle. Barbell, like Arthur said, they're a miracle tool compared to anything ever used before. Machines, we can talk about that afterwards. We're getting close to the end here. So is there any questions? Go ahead. I've been told that isolation exercises do nothing to strengthen your tendons and ligaments and cause tears in real world applications. I often hear that. I often hear real world application. You wanna see what real world application is?  Everybody's got a different lifestyle. You understand what I mean? Your lifestyle's dripping, then this guy's lifestyle. Then his lifestyle and his lifestyle. No, I'm not gonna change your lifestyle. So real world movements, I don't know what they mean. Basically your world is your world, okay. There's no such thing as isolation, okay. You can't isolate a muscle. Geez, you can attempt to isolate as much as you can. You're gonna bring in other muscles into play, okay. Yeah, if I take a barbell or a dumbbell or a machine and I just re-yank it like this, it's not gonna help my tendons. I've never torn anything. Maybe, maybe I'm not a poor example because I've never been injured in exercise. I was smart enough to know from early on to move slow. That's what Arthur said when in doubt of speed of movement. So isolation doesn't exist. I understand what you mean by it. You're saying single joint movements, moving around one joint, I've never heard that happen before. That may have been suggested to you by some sort of expert, but I've never seen anybody tear anything or tear the tendons unless they did speed of movement that was like this. I hope that helps you out because go ahead. Good question though, good question. I just wanna clarify. I don't actually mean tear muscles or joints or tendons. You said the back. In the exercise, I meant like in a sport application. When you're doing an actual sport and your muscles or tendons or joints aren't accustomed to it, you could over exert the muscle and then the tendons and ligaments can't handle it. When you over exert the muscle, where do you refer into? I'm sorry, can you say that again? The muscles exert more force than the tendons and ligaments can handle in an actual. You're talking about for sports? In sports, yeah. You're talking about sports training. Right. Okay, no, I got you now. There's nothing like the real thing. I'm gonna go out a little bit because that's a very good question. I'm gonna go out a little bit in sports today. In my opinion, okay. There's been more injuries now more, now today in sports and activities, now more than ever before in the history of mankind. In my opinion, this is my opinion, okay. I'm gonna tell you something funny afterwards. In my opinion, it's because of the faulty, one size fit saw real world training applications in sports, in sports on NFL, Major League Baseball, hockey, NBA. This, the idea, remember what I told you before, the idea is to trigger, benefit, stimulate. The idea is stimulate muscles. Strengthen them in the best way possible, okay. If you do it right. Even if you don't even do it hard, you're not gonna get, they're hurting themselves in white rooms.  But Bertrand Russell once said, since when has it been ever indicated the majority is right about anything and the majority is doing all this type of training in sports. But I've never seen so many injuries in my life. You can correct me if I'm wrong about this. I'm sure it's statistics. Sky high compared to what they used to be. So apparently somebody's wrong, either they're wrong or I'm wrong. And I don't think I'm wrong. But I was, you left me out on a tangent, but Arthur Jones once said, you know what we should do with all the football coaches that allow this to happen? Put a thousand of them in a 747, fly them right into a mountain. I agree with that. I mean, Jesus. I mean, I see cuts of the NBA athlete's training. I can remember. The war is out there. The practice of the sport itself is the conditioning for the sport. If you train the muscles directly and then apply it out to the battle out there, you're gonna be better off. Imitating any particular thing like the argument about a power claim. Like we said before, it means better power claims or what good value is it? Apparently, people have an emotional attachment to them. They tell me it's better. But the idea is to stimulate muscles, okay? And we can talk about this later. I'm an open book. I'd love to discuss it with you later because I'll stick around and watch some of the speakers. The idea is to stimulate the muscles and then apply it to the sport. There's way too much conditioning going on in sports today. You look at mixed martial arts, they do nothing. They have all different. They have wrestling, grappling, jujitsu, striking. And what are they doing? Swinging. They're doing ropes and doing things unrelated to the activity. They're beating themselves up. They're wasting their reserves in no way. Go to fight, there's nothing there. Very, very difficult. I mean, the training in sports today is a cultural disaster. What makes me an expert at that? Well, you can see it at face value. You already, you know the injuries are more today. I mean, I've seen, I can believe the number of injuries. The best example that is, if you call the sports and really not a sports game is Tiger Woods. Here's the guy that's supposed to be the most fit guy, right? Fit guy in golf. Big muscles, broad shoulder vitality. He can't even play the game now because he injured his back so darn much. He's had surgeries, he's got physical problem beyond your wildest understanding. It's not because he swings any faster than the other guys. The other guys swings just as fast. It's just his training routines were bad, okay? Well, he's somebody you know. Listen, I'm just calling the way I see it. Anybody else? We're actually out of time now, so let's give it up for David Landau.


This is an emergency broadcast from the Mantic Bunker. Containment protocols have been initiated. Hello! We said we'd be back each day with the best way to get through these difficult times. Here is our first advice. Take teapot. Put in teabag. Add hot water and leave for 2 minutes. Take cup. Oh no not that one. Take cup. Put in tea. Add a dash. Just a dash of milk. Drink. And everything will be fine. Tune in at 6pm for more from the Mantic Emergency Protocol Bunker.


It is time for another MLB the show 21 pack opening Yo, what's going on leaf nation sleeper back again and welcome back to the channel today like I said We were back with another pack opening I don't know if we're gonna be able to top last one as we pulled some unreal players But we're gonna try our hardest to do it to just beat it. So essentially I am pretty much done the AL West We literally just need Mike trout and we're done the West and then the central should be easy So we're looking for trout this up this episode or this video I don't know if we're gonna be able to do it But hey, we'll see we have 14 of the regular packs 21 of the team affinity one set One set one headliners and two of the balling as a habit pack So we will start with the team affinities like we did last episode. We pulled some really good diamonds from these ones last Video so hopefully we can continue here and pull something good, but we will see like I said, we are looking for Mike trout So I can get the angels collection done. He's going for like 350k right now. I just don't got the stubs for that obviously So we're looking for something obviously and we're hoping that San Diego studio can give us something I know it's gonna be hard to top last pack opening the last pack opening was unreal Like the players I pulled that I don't think it could be replicated But we'll try we'll try we'll try to replicate it if we can pull a couple diamonds that would be great So we'll see what happens here. Just a bunch of bronze players so far I think we haven't pulled a silver yet haven't definitely haven't pulled a gold yet So we're just waiting on something like that a couple more bronze is here getting the pregame shakes a lot here So I don't know what they're trying to tell me but anyways We will continue on Diamond silver. All right, there's our first silver pole. So not bad not bad at all Can we one up it get a gold and then a diamond? Maybe who knows who knows we get a gold equipment Bronze Clown car interesting. Okay We have what how many left the menus are so slow 14 left 14 of those one of those and two of those so We have some good packs. These are all from grinding the team affinities from doing the 42s Cuz like I said, I'm trying to get the 99 Robinson And it's gonna take us a little while obviously, but we should have it very soon. We get a diamond Equipment interesting there Phillies uniform that's pretty cool. Not too bad. Okay first diamond. It's not a player Do we obviously don't count that but yeah, let's just continue here Nothing nothing going our way so far and I don't think we're gonna have great Pac-Luk for a while after that last video I could be wrong, but hopefully I am wrong. There we go first diamond. We're going one by one if it could be my trout Man that'd be a W that'd be a W. Here we go Who's it going to be first diamond of the video? Alright, it's a purple. So it's not my trout that is okay Who is it going to be? I'm not sure who this is. Oh Trevor's story not bad. He's up to an 89. We get him for the Rockies. I do not have him yet So that's a good pull. We will definitely take Trevor's story. I like that Maybe a suit to be blue janex season who knows but we will take that Trevor story first diamond of the video We've yet to pull him this year like I said, so we'll take him Trevor story I must have Mike trail for some reason Trevor story first diamond and we still have a ton of packs left About 24 of the regular packs left. So not bad. There's our first diamond. We get Trevor story Not too bad gold equipment. No back-to-back diamonds. That's okay with me. We get a sponsorship nice nice nice Nine more of these to go these these packs the regular packs have been pretty good for me I've been pulling diamonds from them for whatever reason. I'm not complaining obviously like keep them coming, dude I'll take all the diamonds you can give me All right bronzes two bronzes Someone from the angels, so that's nice as we're trying to get that collection done eventually again Just need to pull my trout, but don't know if that will ever happen. I don't think The odds are in my favor for that like what are the odds of this pack 150 to get a diamond player, right? So I don't know it's gonna be hard to pull trout from these packs But hey if it were to happen somehow some way, I mean I wouldn't be complaining I don't think anyone would be you know what I mean So let's see gold player Glaber Torres. We'll probably just sell them as we have the The Yankees collection done But Trevor story he's up to an 89 right now So can we beat that who knows another gold player? We don't have him yet. So that's pretty good Again, we're trying to get all the collections done So any players that I don't have are helping me out obviously so gold equipment It's a bronze and a bronze another angels player that we need nice. Okay. We'll take it Two more of these left, and then we got 14 technically left. So I'm not too bad It's a gold there interesting Verdugo again, we'll probably sell them as we have the red socks collection done But not bad pulling some golds in these we won't complain about that Obviously one more diamond would be great to top off the video, but we will see about that the socks white socks we do need to get them so we'll take that and We have 14 of these left 14 of these bad boys left They're the same ones of the the other packs. We just opened just a different packaging on the front if that makes sense So let's see if we can pull one more diamond out of these maybe who knows Hopefully we're not jinxing my luck here But nothing going in these ones so far two of them and nothing 12 more to go Again, if we can top a Trevor story again, we're we're fishing for Mike trout It's as simple as that we want we're fishing for a trout Alright 11 more to go here. Come on. Give me something one more diamond at least this video and I'd be pretty content I'm content now with it obviously, but one more would definitely make me even more happy. So You know just putting it out there San Diego studio just putting it out there nine more to go And then we stole the ball in as a habit packs, too I think we have two of them. So that's not too bad. We have that and then the set one headliners I think We have a chance of pulling that Donaldson So that's not too too bad I backed out way too many times because these menus if you guys have been playing this game these menus have been brutal These last couple of since release really another diamond Equipment I don't know what's all about that. That's a pretty sick model in this jersey though won't complain about that Six more can we get a diamond player instead of two diamond equipment then that'd be nice Just saying we're pulling red socks a lot man, and we're done the collection. We don't need them no more You know what I mean? Come on bronze I have a feeling we're not going to pull anything else in these regular packs here, man A little depressing, but it's okay. It's okay three more here And then we're on to the better packs essentially Where we have a higher chance of pulling the diamond obviously. So hopefully that happens there two more to go And this is what I mean these menus man been so slow, but that's okay. That's okay. We're not complaining The game I'm having fun with the game. I've been grinding it every every day since release Been having a ton of fun with it Um, no complaints in terms of gameplay. That's for sure. So now we're on to the headliners Where we have a chance to pull in the Donaldson and the red socks jersey Um, we have a one to ten chance of pulling a diamond player out of this So this contains one silver or better player item. So let's see if we can get out of this one Uh, we opened up five of these and didn't pull one diamond in them. So let's see what's waiting in here All right, nothing in that one all good All good. We got two of these ballin as a habit packs Let's pull at least one diamond out of these two man at least one Okay, gold player josh donaldson not the diamond one, but that's okay. We'll take that And now we are on to the last one. What are what are my my odds in this 115? Okay One more diamond. Let's end off the video with a diamond here. Come on No, it's a gold all good chris brian. Hey, we can't complain. We pulled a trevor story So we did pull a diamond in this video. We really can't complain about it Anyways guys, that's going to be it for today's video. Um Unfortunately couldn't pull it out. It is what it is But we did pull a diamond. So the luck continues to be a thing for us, which is nice I'm gonna grind for some more packs. So there should be another pack opening coming soon, hopefully But uh, yeah, that is it for this video guys If you did enjoy be sure to drop a like on any figures on you around here Make sure to subscribe button down below. We're on the road to 4 000 subscribers Um, but guys, yeah, that is it for today's video. Thank you guys very much for watching and I'll talk to you guys later Peace


Hello everybody. How are you doing? Okay. So pretty much everything has been said about me thus far. So as David mentioned, I do monitoring for a living and we use Postgres. So I figured the one thing I wanted to convey or share is, you know, overall lessons learned in monitoring Postgres, and hopefully try to present some way to do it sanely without going crazy. My Twitter handle is there. If you have comments after, you can also talk to me in real life as it were. Okay. So Postgres Data Dog last year presented, it was a talk called monitoring 40,000 servers with a Postgres database this year. I didn't want to present the same, we just updated numbers. So we've grown obviously as a company and as a service, and we've grown a little bit where our database cluster is not that much, which is I think a testament to the solidity of Postgres and the viability as a solid SQL database. So who was at PGConf 2014 out of curiosity? Okay. Maybe 20, 30 percent of the room. Okay. Great. Okay. So this is what I look at every day. It sits on my desk, on my immediate right, and it's basically how I see Data Dog. So it's a lot of pretty graph and so on. What I've circled with red on the right is the key metrics that I watch that pertains to our Postgres cluster. So there's a lot more stuff on it obviously because we have a platform that runs more than just Postgres, but it's still in a fair amount of real estate. So it's a central piece to architecture. For without doing a rehash of the partition last year, basically we use Postgres and Data Dog as a catalog, if you will. So we're a monitoring company, we receive a ton of time series data. We don't store time series data in Postgres because that's a little bit unwieldy. There'd be too much storage. It's just not very optimized for that. But what we store in Postgres is all the metadata. So if you guys are, I guess, is anybody a customer or a user? Okay. So all your tags. Yeah. So you're in there, somewhere. Okay. So now there's a lot of people here, and I'm assuming you're interested in monitoring. Now you may not be very excited by monitoring. So just a quick show of hand. Who really loves monitoring? Oh wow. You guys should come and work for us.  Oh, that's good. All right.  Okay. So if I zoom in, this is what I care about. So I'd say handcrafted to our use case. So I care about whether we're getting transactions. That's this guy commits and rolled back. I care about whether we're going to run out of this space, and this is like game over here. Close. I can speak to that in more detail. I care about our apps timing out because they, for instance, the query is not returning. I care about locks on some very specific table. I care about replication lag because we use replication in a read fashion. I care a little about CPU because I'll get into that at some point, but we ran into some interesting issues with kernel, and so I want to make sure that we don't have a repetition of that, or at least I can have a shortcut to the answer if that ever happens again, but I think we're in good shape. I guess after watching this stuff all day long, that's pretty much as much as I can consume for progress. There's eight, there's probably a few too many, but three or four at most is what I care about. I don't really, I'll show you how many metrics you can collect for progress. There are many more. Most of them are not interested in sort of a high level. I'm interested in some specific situation from a sort of pure day-to-day monitoring. That's enough for me, and sort of the premise of this presentation is that that should be the case. You should not have a screen, well in this case, you can have a screen full, but not have to watch 50 metrics, because that means that you're watching stuff that you shouldn't watch. Everybody with me so far? Okay, cool.  Okay. All right, so it's not always been happy times in Postgres land at Data Dog, and that's not to say, I think in comparison with any other thing we use, any other data store we use, so we use Cassandra, we use Kafka, we use Redis. It's probably the one that's, A is given us the least amount of grief. The only exception was probably S3, but that kind of gives you, we have Elastic Search and so on.  Postgres being one, and so once in a while we'll hit a bottleneck and then I'll go through the methodology that I'm going to describe, and then we'll find the issue and then we'll tweak something and then things will be better. So for instance, so we're running on a 32 core, we don't need to be as we run of probably one of their biggest machines, not DbGus, but we've had cases where this is over the course of two hours where we basically bottom out. So this thin line is the amount of idle CPU left and then the thick line is just a moving average so it's a little bit easier to track and you see that at some point for like a good 20 minutes we're basically running out of steam. So we've had episodes like that, interestingly enough, so we've had things that look the same but actually I'll get into details that are totally different in nature. So where it's, you know, same thing where basically we're running out of CPU. And so running our CPU when you're in Postgres is not necessarily the kind of problem you're gonna have because you may be a lot more IO bound than we are and but you could potentially project the same thing for IO metrics, for IO stats. Okay, another kind of problem we've had what we call lockness when we basically there's so much convention on the box that just nothing moves. And then at some point somebody figures out, okay, I've found a way to fix it and everything goes back to normal. So we've had that. If you've been a customer or user maybe you've noticed, hopefully you haven't. If you've noticed, then we should have told you via our status page that, hey, we have a problem so it doesn't come to a surprise. We've had also this kind of stuff so this is like a CPU graph stack but just out of the interesting thing the colors are really awful. Well, so there's a, for instance, this peak has this. This is a kernel time. So percent of the time spent on CPU in the kernel and this is sort of what Postgres does. So we've had issues where there's a huge imbalance and I'll get into that as well. So sort of a bunch of battle scars with the Postgres but I imagine everybody in the room has had similar experiences. So the born of that was, I think, for me for this year I was thinking, well, what could I talk about? I don't wanna, you know, I'm trying to think of something that's useful. And I was thinking that I could basically present how we approach dealing with this kind of problem. These kinds of problems because they turn out to be very diverse. So, and I just wanna define what monitoring is because I think the title of the topic is monitoring from the ground up. So for me it's really understanding the performance of the system as it is now. And that's, to some extent you could argue and I would definitely argue that your monitoring system, whatever it is, is how you understand what's going on. The other way to do that is to ask a user maybe how it's going. There's just no other interface. You just can't go, even if you're on-prem you just can't walk to the cage, look at the servers and kind of understand what's going on. You just see the lights blinking. And so that's why for me monitoring it's sort of crucial. The crucial part here is understanding and you have to understand how it is now, not how it should be or how it will be in a month or something like that. It's really, there's a strong real-time aspect to it. Monitoring is also dealing with partial degradation or total failure and that's really the battle scars I was mentioning earlier, so that's pretty obvious. And monitoring, I think importantly enough and that's how kind of I got to these eight graphs that I watch is an iterative process. I think because monitoring is about understanding and then you don't come up with an architecture and then you understand it with all its performance from the get-go, you're sort of discovering things as they occur, as the input changes, you have more traffic maybe or less traffic or different patterns or different queries.  So now it's an iterative process which means that and because you guys have all probably somewhat different situations in terms of how you use Postgres, what size, what hosting, what kind of queries you have, you'll have different outcomes but what I want to talk about is how you get to sort of a fast or reasonably fast but a good iterative process that'll get you to having just the key metrics to watch and then the rest you can ignore for the most part. Is everybody good with this definition? Anybody vehemently disagrees? Well if so, you can do, please do. What I didn't consider monitoring which is capacity planning is, which you can use the same data that we'll see, the same metrics, you can use for capacity planning but for me it's something different. Capacity planning is obviously all about forecasts. It's basically extracting from the data some kind of model and then having assumptions baked in and then cleaning up the data so your model can be used to predict what your performance will be either in time in the future or if you double let's say the amount of traffic to your platform, what the performance will look like. So that's sort of a different class of problem and thus the method I'm gonna talk about doesn't quite apply there. Monitoring I would argue is relatively hard because the systems we use are complex. Now if anybody was sort of new to the Postgres world they would probably go through these four questions which I'll detail in just a second but the reality is if you start using some new tool, some new data store, some new application runtime, some new anything, you're gonna go through, you're gonna have these four questions pretty much in mind. So first one is well what can I measure about the performance of my system or in the Postgres we'll get into detail. Second one important is what do I wanna be alerted on? So when, what should I watch and when should I be, when do I allow myself to be woken up in the middle of the night because something's wrong with Postgres or Component X if you're talking about something else? That's a critical one, what is normal? So with all these metrics that you can collect what did they look like when it's normal and then when something bad happens what do I use? And that's I think ultimately once you have a proper monitoring system in place and once you're on the, which really means once you understand how the system works you can answer all particularly number four. So my goal is using Data Dog as an example I wanna present a sort of rational method for designing a monitoring system if you will. I wanna apply that to Postgres in particular and you know overall I wanna share what we've learned. So this is really, you know why I'm here. In particular I'm not here to sell Data Dog if you're interested in buying, they're downstairs but that's, that's, yeah that's so. There'll be some graphs from Data Dog but that's it. Okay so let's talk about some prior art. So by and large the ideas that I'm presenting are not my own, I attributed them. So there are three I think fairly recent documents or ideas that have been presented which I think apply very well and sort of are the backbone of this method. They're all luckily available online. The first one is, it's called My Philosophy in Alerting by Rob Evershoek. He's necessary at Google. I very much, if you guys are on call or if you have people on call in your company I very much recommend reading this. Who has read it? Okay, definitely go and read it. It's totally worth it. Number two is what metrics should I pay attention to by Baron Schwartz, she's actually downstairs I saw her earlier. So that's a video you can watch, takes about 20 minutes. And number three is the use method by Brandon Gregg who's now at Netflix and you can, this is his blog, his sort of website.  So Cliff Notes on my philosophy in alerting where he's philosophy in alerting as I wear. The idea is to really answer the fundamental question if you have, if you're on call first, I guess who's on call in the room? Yeah, so maybe 50, 30 to 50%. The question is what should I alert on? Or what should I get alert, you know, what can happen in the midnight by? And the idea is to answer this question in with the intent to maximize robustness so you don't miss alerts that should have opened you up and you don't get woken up for nothing and also conversely to minimize burnout. If you guys, so for those of you who carry AAPJer, as you wear, you know what burnout means. For those you don't, be glad you don't have to face that. So in this case is basically, and I've been through phases like this where every night, you're on call, let's say for a week, every night at anywhere between two and four a.m., basically in the deep phases of deep sleep, your phone rings like something's on fire and you have to react.  So what Rob said is there are basically two things. When you look at everything you can alert on and all that your monitoring system currently is collecting, there are two things. There are symptoms and there are what he calls causes. And symptoms are really things that, things like the application is timing out and causes maybe 500 or some kind of problem for the end user. So symptoms are something that really doesn't, it's like, I guess it's like a disease, right? You have symptoms that doesn't necessarily explain why you have the symptoms, but it's how you can tell something's wrong. And so the symptoms are sort of very closely end user and so on, the causes are more things like the classic monitoring stuff, like high load, low free disk space, that kind of, all the stuff that you can get very easily out of any machine, which he classify causes. So in the central message, and that's one of the key messages of first, his presentation or his document also, I guess my presentation is you wanna alert on symptoms. So the only thing you wanna alert on is symptoms. So you wanna alert if, for instance, there are no, the database is not answering any queries or if all the queries or a bunch of queries return with errors or if you look at a wider system, if you're, if no web traffic reaches, goes past your balance or something like that. You wanna be pretty high level. You don't wanna, and the causes are more like how you explain the symptoms. So maybe it's, there's no traffic to my, or the queries going to progress or return errors and the causes, well, there's no disk space left. So the thing just is basically stalled. Or it could be that there are no queries returning from my database till time out because, and the cause may be that the CPU is pegged or the machine is misconfigured. Now the, and the, sorry, I'll explain why you wanna go in this order and not in the reverse. Now the tragedy of monitoring out of the box for in most cases is that it takes the exact opposite. And the reason for that is that what's called in this case causes like CPU load, memory, blah, blah, blah, is very readily available. It's easy to collect. So by default that's what gets configured for alerting and that's bad because it's very wasteful. And I'll show you why. Okay, so Cliff Notes, I saved you maybe 10 minutes. I shouldn't have Cliff Notes for this guy because it's really important to read, but that's, so if you don't have time to read or if you don't wanna bother, that's what I extracted from it. Cliff Notes from the second document that I pointed out prior to what metrics should I pay attention to. It's a little bit, it's basically the same idea, slightly different terminology. It answers kind of the same question, what should I monitor as in what should I allow to be woken up by? So the message there is alert, so there's two things, there's work and resources and the message is alert and work and not resources. So work is, consider work is what the database is doing. So the database at the base level, it's storing data and then storing or dating and so on and then returning results of queries. That's the basic work of the database. The rest is just details. Resources are, for instance, CPU is a resource, memory is a resource. Number of queries that had shared buffer percentage is considered as a resource. It doesn't really do work. You don't buy database or you don't use a database, you don't deploy database so that your queries can hit shared buffers. It's just not terribly useful. Though you do it, you use it to store data. So one way to think about it is think of it as a factory. I mean, it's how 19th century, but kind of that's how I can see that at a very high level is, postgres is your factory, your machine. You feed it with CPU memory network IO and then it can, and data, of course, they'll come through in the network and then you'll get results out, you'll query. So the work is in the queries. The work is not in using this. Now the interesting thing with this approach of work and resources is that you can actually apply to the subcomponents of database. So you can consider that the work of the database is to return queries, return them in time and so on. Now each of the components, they also have their own factory. So the backend process it's gonna take, sort of, it's gonna use CPU and so on, it's gonna return some data. The wall sender also, it's goal is to send the walls. That's it. I mean, it's gonna need a network for that. It's gonna need to read the wall some disk. So it's gonna need some access to disk and so on, but that's all it does. So you can so recursively apply this decomposition for any metric that you're collecting. So from any of these systems, you can apply just the composition between the work and resources. Does that make sense? Who's not with me? All right, I'll take that as a yes. So again, the key message, which is very similar to the first one is, alert on work, don't explain with resources. So do not, I think the opposite is do not alert on work. Do not alert on resources, yeah, I get checked up. So why not alert on resources? So here's my graph that I showed you. So this is idle CPU, so it goes from 100, I mean the machine is either off or nobody's touching it, nobody's doing anything with it. Zero is the CPU is completely pegged. Now the problem is I could set an alert for 10% CPU left, that's great, but it could be that there's a bad query going on and it's just trashing the box. It could be that somebody's doing some, I don't know, somebody mining bitcoins on the box and then I'll get the exact same, basically the exact same profile. So setting an alert on a resource like the CPU is used, like great, okay, it's used. It's more a capacity planning problem than a monitoring problem. Because just setting an alert and say, this could use, now I'm like, okay, but I still don't know why. Maybe it's a good thing, maybe it's a bad thing. Maybe this is a bad job and weird, I don't know, let's say just crunching some report for the end of the month or something like that or maybe it's something bad and just, it's just not actionable. And so a lot of the burnout when you do your monitoring and you alert on resources like CPU, load is a classic, it's usually just wasting your time because it's not, that's not what should wake you up. What should wake you up is the database is not responding, there are no queries being returned, there's just a lot of errors, everything's timing out, that's the stuff you wanna be, you wanna alert on. Because ultimately that's the stuff that people care about, be it your boss, your customers, your fellow teammates and so on. Everybody call on why not to alert on resources? I didn't have a question, I guess. I'm waiting until something happens as opposed to catching something. Can I understand trying to look at the resource as a server but look at its own like file systems and where they fill up. So yes, yes, so I have one exception to the rule and that's, there are two exceptions to the rule in my whole presentation, they're file system and connections, the consumption of resources, it is very important if you wanna do capacity planning. So you wanna know when do I run out of this space or if we double incoming traffic, do we run out of steam in postgres or something like that. It is not though. No, go ahead, sorry, just one minute. Oh, okay. It is not though monitoring per se. I mean, because anything like I do something with a machine for five minutes, I can trip the alert because it's that easy. One thing I found is that sometimes even with resource monitoring is tricky because I don't wanna be woke up in the middle of the night because the system went to 80% plus one K and stays there all night. So a rate of change, a resource situation, a rate of change is also kind of important thing. If it goes from 80 to 90 in 10 minutes, you need to know that. If it goes from 79 to 80 over the course of the night, then you don't need to know that. You can fix it in the morning. Even though I would argue, so my two exceptions, all right, so I mean, my two exceptions are connections and disk. Look, if your CPU is busier than usual, maybe somebody pushed some bad code but the queries return in roughly the same amount of time. Yes, at some point, for more for capacity planning, I'll say, well, I wanna see kind of what the performance looks like today. Did we change phenomenally from last month to this month? I totally, you should totally look at that, but not as a wake me up because it's now 81%. So because what happens is 81%, so next morning I go, okay, I'll make it 85%. That's a classic for disk space. And then, well, just 95% and then it's game over. I've played this game, been burning a couple of times. Okay, all right, so that was another example of, I don't, something happens that this is the CPU graph. Basically, this is CPU and then the sort of the lighter blue is idle, so the horizon, the capacity we have from a CPU perspective. It kind of looked bad on the right, right? It's kind of something, it's maxing out. Turns out, and this is, it has very limited impact of it has a few more errors, but in the context of this particular application, it didn't matter. I mean, we get, anyway, these kind of error once in a while, even it's fairly decorrelated from CPU. So I wouldn't wanna set alert on 80% for this CPU because it would have triggered and nothing, frankly, there's nothing to report the services operating properly. So it's, I can look at it maybe on a monthly basis and see, well, this is where we're gonna need a little bit of extra juice if we wanna keep adding customers, but that's about it. It's not a real time, tell me something's, wake me up because something's wrong. Am I, did I sort of clarify a little bit, my point? Okay, cool, all right. So the two exceptions are, so I said in the case of Postgres, for me, it's mostly this space, it's a resource, but the problem with this space is when you have one byte left that'll work and when you have zero bytes left, it stops all of a sudden and it's fairly easy to predict. So it's kind of, you don't wanna get calls like, oh, well, yeah, we didn't monitor this space because just do it, that one, it's okay. The other one is number connections because there's some maximum in your Postgres configuration so you don't wanna run out because it'll cause back pressure and your application is gonna basically degrade fairly quickly if you'll run out of connections. So these are sort of shortcuts, but in general, I would say the resource of monitoring, alerting on resources, specifically alerting on resources, I'm not a huge fan of. Okay, the last cliff note, the use method, who's heard of that? Okay, cool, a few people. So this is more, okay, so you have the difference between work and resources, right? So that's work is what's useful and resources is what's consumed to produce that work. So you'll get alerted on work so the database is not returning any queries, for instance. Now, okay, so now you know you need to look at it. The question is how do you go about it? So the use method is a systematic review of all the resources to identify basically the bottlenecks and the errors. So use stands for utilization, saturation of errors. So it turns out you can basically map all the metrics you're getting into one of these three. And so having this taxonomy is very useful because you can do two things. So first of all, make sure you've covered everything. And also when you have a new metric, you can, it's a sort of non-categorize, okay, this is this new metric. What does it really measure? Utilization, saturation of error, or error. And so where you can apply use is for any resource, CPU, IO, memory, so classic resources, checkpoint or back-end process, you could consider them as the resources. So you can consider the back-end process, how much utilization is it, how often is it actually not in idle mode and when the connection is not idle. The wall sender is the same. You could look at errors, how many errors does the back end process generate? Check pointer has its own set of metrics and so on. The cool thing with this method is suggested interpretation because it's relatively simple is you're okay when you have low utilization, no saturation in few errors. And it's bad usually, and that's in general, when utilization is over 70% because of sampling, that means maybe you run out of steam, you hit 100% utilization where your resources is entirely consumed. Usually bad also when saturation is greater than zero, saturation as in things that your resource is waiting to process but doesn't have the bandwidth to process right now. So if things pile up for that resource, if the walls are piling up because they are all sent across the wire to your replicas. If the queries, if the PG Bouncer client connections pile up because there's no back-end process to consume, and saturation usually is bad. And errors is bad. It's more the change that matters there than a specific value. So if you have zero errors usually and something's happening, then it's usually fine. If you have a couple of errors but it's not increasing, then you're usually fine as well.  So you're gonna have some errors sometimes. But it's bad when you have a lot more errors. You have a symptom or you've been loaded on some work metric but your errors are growing. Then usually that's a good way to interpret that. Oh, this particular resource that I'm looking at has an issue. Okay, so this is sort of applying the idle CPU. So the use applied to CPU idleness, so in this case it's a measure of utilization. This idle is essentially a measure of utilization.  Load would be a measure of saturation of the CPU. So if you have a load greater than, all per core greater than one, that means there are more tasks waiting to be executed than there are cores available. Okay, so altogether, before we get into the sort of more concrete part of the stuff, so you're looked on high level symptoms of work. When you have a symptom, you just identify which resources is causing you trouble with that use method. If you see, now if you've identified the resource and it's not like just CPU but it's something at a higher level, like one of the processes here, progress, you could look at, well, what are the metrics for this particular piece of component that are called symptom and what are the metrics that are more like, sorry, what are the metrics that explain that point to work and what are the metrics that point to resources? And you have this recursive approach to finding exactly what the root cause of your problem is. Does that make sense? Okay, cool. And then you just stop when the high level symptoms disappear. And that's that. I mean, then you can do postmortem and fixings and so on. But in general, that's, you just go through this again and again and again. Okay, so I think, so what I'm basically positing is that that little recipe takes here to the last three questions of these four remains number one. So in practice, this, the source of things you can use for monitoring progress, I think come from the PG stats star, so a bunch of them, come from so slow queries if you enable the minimum statement duration whether to explain that helps come from the OS because at some point there's some resource that is measured by the OS, not by Postgres. And then you can add to it like Dtraces system tab if you wanna go like fairly, fairly deep in details. Okay, so PG stat. So I actually went through the page, I summarized so you could go there and you'll see a spreadsheet. There are 83, what we call metrics so things that you can, we'll yield basically a quantity of value and I've classified them based on my approach. So that gave me, I've classified as symptom resources of work if you will resources. And so I found 30 for symptoms at different levels. So there's like symptom level metrics for the database like there's no transactions going on, transaction zero database is either not working, not doing anything and it's in maintenance maybe or there's something bad because the transactions just don't complete. And then there are also, I guess you can measure to detect symptoms at individual components like the wall machinery, the checkpointer and so on and so forth.  So if you go there, you'll see it's just a spreadsheet. You can comment, you can add comments actually if you, so please add comments if you care. Very quickly, so what does it look like? So for me, if you look at PG stat database, the stuff that I care about is are my clients okay? And so that's the database level. This may actually may not show up in the progress PG stats database per se, but at the level of database I care about do I get errors in my queries and do my query timeouts, do my queries timeout. So that's number one. So this is a very high level, like symptomatic view of progress. Are there any transaction going on and is query activity normal? Like, and this is more am I getting queries like select hitting the box, inserts, you know updates and deletes is that thing in line with what I expect. There could be any reason why this goes out, where this is out of whack, but at least this is the stuff I wanna be woken up, not whether we use a lot of CPU or not. For, from a replication also is something that I care about because of the way we run. So the two things in this case, the two metrics that I would call symptomatic or as a way to detect symptoms of problems are just the state of replications that we're streaming or not is running or not. And then just the replay lag, how far back am I? So this is one of the queries we use for that to generate the metric.  So the exceptions to the rule, the two exceptions to the rule, I guess, the resources that I wanna be alerted on and you, I've set up data docs that will wake me up in case we add a max out number of connections because I know that immediately problems gonna hit us hard or we will run out of this space, which is even in 2015, a surprisingly common cause of crashes and problems. So another, so these are the metrics you can get. For me, the key metrics are a PG stat star. The rest is in the spreadsheet that I presented earlier, you could have, there are all the details. I will use those when I'm investigating a problem deep down, but not, I don't alert on those cause I don't care, frankly, as long as the database is working, doing its work within the bounds I care about, then I'm fine. So, slow queries and another way to look at performance, it's a little bit, at least in my experience, this is a little bit different. So there's clearly, when I look at slow queries, when things are okay, what I do usually is we'll have a baseline of PG stat statement to see kind of what the execution of queries look like, how many they are, how long they take overall. It's almost another measure of utilization of my database. And I'll set the minute, I want to log the slow queries, I'll set it to either the SLO that I have for my database, or some kind of timeout. So I want to know, we're just under the timeout. So I want to know when, basically I'm not meeting my SLO, or when the client is going to timeout. I don't really like to set it much lower because there's always going to be something bad. You know, if you set it to one second, you're going to have a query that's going to take two seconds and it's going to start popping up. And it makes the log very full of stuff that may actually have no impact on the quality of service. So that's why I don't want to set it too high. And then I just set it to explain so I don't have to explain. So that's, everybody knows how to explain.  There's just somebody rewrote a query and it's just somebody forgot to get an index or something like that. I still find it usable, but only to a point. If it's just one bad query, sometimes PG statement will flag it because they have the baseline, say, oh, this guy's total time is just shot up, even though the count of number of calls, you know, say an hour hasn't moved. So there's something different there. And maybe in the slow query, I'll see that particular query pop up. So this is a case where we have, sort of the big queries. And this is the queries that we run are very, sometimes a little bit pathological. So yeah, this is a very, very spiky pattern. And once in a while, there'll be sort of an error which may or may not be caused by this. But it's kind of under the threshold that I care about. And so, you know, that's fine. So the efficient slow query, I can use a slow query log to catch. When things get bad, I find, I personally find the slow query logs of PG stats statement to be completely useless, or not useless. The reason is when there's a bad, something sort of systemically, not systematically, but systemically broken with the database, everything will slow. So it will slow down so all the metrics will appear slow.  So the only way of being able to work around that is to really focus on the minute I get alerted is just look at what were the queries that finish just after that. Because anything after, you know, if the slowdown lasts for 15 minutes, usually after a couple of minutes, everything starts to be recorded as slow. And so it's just not, I have all the queries of my system that show up in my log, and I'm just wasting my time. So if you had cases like that where everything slows down for 10 minutes, between 10.35 and 10.45, and just slow query log is exploding. It's like everything is trying to be logged, basically. The harder case is, and overall with the monitoring is when the symptoms are not obvious. So we've had cases where the throughput is still fairly reasonable. But just to, it's a little bit slow.  But then you look at utilization. So it's more like you have that on the screen. It's like, huh, it's weird. Things are fine, but something doesn't feel right. So it's the case though, where I don't wanna, I'm not gonna look at that in the middle of the night. I'm not waking me up in the middle of the night to tell me that something is wrong, and even though everything looks fine, but the resource utilization is high, is not something that my wife's gonna like, because in the middle of the night, I'm gonna wake up, and if I sit on the earth on that, I'm gonna wake up and I go, well, yes, something is not right, but I'm tired, I'm not 100%. So even in this case, I found that the resources, I don't wanna get alerted on resources, I will try to understand why the system is behaving like that with the resources, but I don't wake me up. So what I've done there is the use method, look at everything, so resource by resource, utilization, saturation error, and just pull the thread. So one of the resource that I saw was locks, so the locks were way out of whack, and so, okay, well, that's not great, because that explains why the queries are slower, even though the quality of the service is still fine, but it's an indication, so I keep going, all right, what uses locks, and then you sort of pull the thread, and then you end up with a CPU, at some point, everything ends up there, memory, CPU, IO, network. And then I have these, what I saw in this case, I have these huge spikes in kernel, and really off the chart. So on a 32 core machine, we'd have 30% of the CPU cycles and running pushgres code, and then 70% running the kernel, the Linux kernel, and so, okay, you're there, and so now the kernel is my resource, and so what can I measure to, what are the key components? You sort of start diving into the kernel layers, and finally, we found two things which were kernel related, not surprisingly, so I just put them there because if you have a situation, we have high kernel timings, so person CPU spend kernel is high, high locks, but nothing else is showing up weird, maybe that's that, and I can save you time, we spend, I think, three weeks chasing after those, so in case it's helpful, there it is. Received size scaling, which distributes the amount of network, the network stack in the kernel by default executes on core zero, this lets you distribute across multiple cores, and scheduler migration cost is just this thing where the reason why the kernel is so busy is because it spends time moving a piece of a pushgres thread from one core to another core to another core to another core, the more core you have, the worse it gets, so that's sort of an interesting piece. So with that, I almost want to say that monitoring is not hard after all, but at least it's still hard, but I think we've covered most of the, we know which metrics are available, that's in spreadsheet, we know which ones are to alert on, we have a sense of what's normal, or at least what's normal is what it looks like when there are no problems, and then when something bad happens, how do you get, how do you go at it? And that's the use method, so the takeaways that I have for you guys is, monitoring is hard, but you can be rational about it, it doesn't have to be, it's not an art, I think, I hate that when people say it's an art, no, it's not an art, it should be more science than an art. You want to learn on symptoms, and maybe you have to learn on symptoms that are outside of pushgres, and you don't want to learn on resource except disk space and connections, you want to use the use method to find which resources that power pushgres are causing the issues, and then your last is you want A to A, D to A, D to A, and after, when you go through this, who is go through this, then you build a solid understanding of what your pushgres is doing, what the application at large is doing, and then that leads you to decrease the number of problems, and so then you can love monitoring. And that's it? Maybe that's a six take away, yeah. Any questions? Okay, so go ahead. I was wondering if you could say parsing a log file. Yeah, so it ends up basically parsing a log file, so you can use, what we end up doing is we load that into another database. I don't want to log my slow query log in the database itself, but you know, and then run some SQL against it, and basically trigger around the window when the proverbial feces hit the fan. Just look at what the queries were running, and which one were running slow, and then that's how I would...  Which is, that doesn't tell you if the slave disappears if you're not. No, there's a, that's why the other one is a state, and so, yeah, but true, absolutely. Yeah. Yes? Yeah, my kind of earlier question about rate of change. Yeah. You know, for resources like disk, where I want to alert when it gets to 80% but only if it gets there quickly. Yeah. Otherwise, I can do it in the morning. Do you have anything to kind of handle that, how do you handle that scenario? So, that's, then it's more like the forecasting aspect, because essentially you want to know, do I have until the morning to fix the problem, or is it, or don't? I, so in data per se, we don't have at this point the way to do it, otherwise it do just a linear regression on what it looks like, or the one sort of hack we have is, we sent, so if you set threshold 80%, I'll send you a little snapshot of, this is what the consumption looks like, or actually you can alert on rate of change if you want, the thing is you have to alert on rate of change, and also the fact that it's 80%, so you want to know, because if it increases fast, but it's just somebody's dump, I don't know, doing a PG dump, and there's still four terabytes of disk left, and your database is 500 gigs, then you're probably okay. I would want both rate of change and pressure. Yeah, yeah. If it goes through from 20 to 40, I don't care. Yeah. Quickly, but if it goes down to 80 or 90, then I care. Yeah, no, that makes total sense. I mean, yeah. Yeah, okay, great, well that's all the time we have, so, thank you very much.