could build his own house, farm his own acres, raise his children in freedom. They carved from the wilderness an empire of agriculture and industry. They set for themselves new and higher standards of living. And yet, in one of the great river valleys of America, something went wrong. Three centuries later, the descendants of the pioneers were neglected people, living in a ruined land. For these children, the hope and the promise were dead. For them, the only future was poverty, ignorance, drudgery. The struggle to scratch a bare living from the reluctant soil, once been bright with promise and with hope. Horace Higgins was one of the many who had given up the fight. Halley's in the first rain washes it away. It's the same with all the land that was in good land once, but it's bad land now. The eating away of the soil, the destruction which began innocently when the early settlers cut down the forests. When the farmers, out of ignorance, plowed straight furrows down the hillside. Destruction from the sky, hundreds drowned, thousands made homeless as if the river running to waste. The energies of the people, too. Henry Clark's trouble was the trouble of three million Americans in the Tennessee Valley. It became the direct concern of 130 million Americans in the 48 states. A challenge to democracy and its ability to care for its own. The valley of the Tennessee River lies in the southeastern United States. It covers an area of 40,000 square miles. Nearly as large as England. It was a problem of reconstruction. Reconstruction of land, reconstruction of people. Democracy met the test. It found the men to supervise the job. James P. Pope, United States Senator from the West. Court Morgan, President of the University of Tennessee, who had worked out an agricultural program for the whole area. David Lillianthal, Administrator and Champion of Legislation for Cooperative Electric Power. George Norris, the great American statesman. He deemed a regional plan of setting up a national experiment in one region which could serve as a yardstick for every region. This was the plan. To chain the river through a series of giant dams, checking the floods, to open it to navigation from its mouth to its headwaters. To give the farmers the benefit of modern science and research. To help them control the water on their land and restore the fertility of the soil. To reforest millions of acres on the ravaged hillsides. To exploit the mineral resources of the area. To use the electric power generated by the dams to develop and rehabilitate industry in the cities. To electrify the farms through rural cooperatives. Above all, to prove that human problems can be solved by reason, science and education. The Tennessee Valley was to be pioneered again. This time to be developed, not plundered. This time not for the benefit of a few, but for the many who lived in it. These were the new pioneers. The architects. The research chemists. The agricultural experts. The powermen. The designers of hydroelectric dams. Their method was to control nature not by defying her as in the wasteful past, but by understanding her and harnessing her in the service of humanity. In 1933, a new chapter was written in American public policy when the plan was brought before the representatives of the people in Congress. An act was passed creating the TVA, the Tennessee Valley Authority. President Roosevelt told the nation that the project would set an example of planning not for this generation alone, but for all the generations to come. Then things began to happen in the valley. Let the government men do things with machines over the mountain. None alive, but used, was more inquisitive, was still alive. Over the mountain was something new, big, exciting. They started coming to the dams to see for themselves just what it all had to business of John Warden, the TVA agricultural representative, to answer that. It has everything to do with the farmers, he said. The dams are just a beginning. Without the fullest cooperation of the people on the land, they are worth it, farmers. Ten inches of topsoil supports all the life on earth. Every drop of rain that falls on your fields carries away a bit of this soil. Every time it rains, the gullies, the scars on your fields, grow deeper and wider. The precious topsoil goes faster. The undersoil is hard and sterile. The water which should have soaked into your land runs off, uselessly. Your topsoil travels with it, uselessly, down to the delta of the river. Millions of tons of good farming soil, lost forever, going away with the wind. All this waste isn't necessary. It can be stopped. Your land can be saved if we work together. You've seen the dams, but they're only part of the plan. The rest of it is up to you. If we're to succeed, all of us, you must learn to stop the erosion of your own land. The TVA was created for you to teach you new methods, to provide you with fertilizers to restore your soil. The land is yours, the dams are yours. The whole TVA is yours. We want you to use it. This is how you can become a part of the plan. We need volunteers to try out the new methods, to prove that they are the right ones. Thousands of farmers have already volunteered. How about you? Once at the isolated back valleys, he had talked to hundreds of groups like this, and he knew they needed time. In signing with the TVA to serve as laboratories, whether new methods could be tested and observed by all. Using a terracer made available by the TVA, John Warden showed Henry the principles of contour farming, held it on the land. Through the exhausted soil, Henry used a new phosphate fertilizer, developed by the chemists from ore found in the region. The first season, Henry's chief crop, this was to provide the soil with nitrogen, and revitalize the land. But it wasn't a cash crop. Henry had been warned when he made up his accounts at the end of the year. He couldn't help but feel, well, he sort of set his heart on that new track. This was the crucial period, the first season. The TVA man had known it would be. They were sure of their equipment and methods. The human minds and emotions were another thing. It was a hard decision for Henry Clark, whether he should go on. But he was the descendant of pioneers. Of men who had taken a chance, and who had known that their salvation lay in cooperation. The old spirit of the pioneers was reawakened. The dam builders, farmers, the machines, began to work as one. Substantial quantities of food to market, 30,000 other demonstration farmers became teachers too. For the next year's harvest, Henry and his neighbors had a threshing machine, especially designed by the TVA experts for this valley. Farmers had a machine. Each farmer had an equal right to its use. No longer was it one man alone against the drought and the flood. For the first time they were acting together, cooperatively, for a common purpose. And even more important, a change was beginning to come into their thinking. For the first time they were thinking in terms of each other. What they could accomplish together by working together. With new machinery, new methods, with a definite plan to follow, a plan that embraced them all, the farmers worked and the land responded. He began to understand the real meaning of TVA. That the individual, through cooperation with his fellows, becomes a more important individual. John Warden says, when you develop people, you have something permanent to concern. The children learn how to use the things that were built. Up in the mountains on the tributary streams, high dams backed up reservoirs against the time of drought. Releasing the water when it is needed, holding it in check when the rains come. Down the Tennessee River itself, wide dams control the water step by step. Instead of the alternate floods and drought, water can now be dispatched, as trains are dispatched on a railroad system, the nation's constructive energy, Douglas, Gontesville, Cherokee, Wilson, Pickwick Landing, Chickamaug, the people of the United States. To run a million machines, running out aircraft, engine, shoes, cheap and abundant power to light the cities and villages, to a hundred homely uses, power working tirelessly, endlessly, raising standards, reducing drudgery, hands of the people, a measure of what men can build in peace, a measure of the stature of a new and better world, a world with dignity, and walk into it.


Mother, why did daddy switch to Postum? Your father says there's no caffeine in Postum. Nothing to spoil your sleep. And your father knows best. See, it's Father Knows Best transcribed in Hollywood starring Robert Young as Father. A half-hour visit with your neighbors, the Anderson. Brought to you by Instant Postum, the good-tasting drink that's entirely caffeine-free and by Post's 40% brand flakes, America's largest-selling brand flakes. Every dog, according to Jonathan Swift, must have his day. But not many dogs get to have their day with the Anderson's in the white framehouse on Maple Street. However, a certain French poodle is about to come into the lives of the Anderson's. Like this. God, what, Betty? Look, Father, isn't it cute? Yes, it's cute. But what is it? It's a dog. That's a dog? It's a toy French poodle. Oh, boy, infant, don't start mauling him. Well, don't even touch him. He belongs to Mr. Fawcett and he's worth over $300. I just want to put my hand on him. I'm not going to break him. I don't know. From the look of him, he might break at that. Get away, Kathy. Oh, for Pete's sake. Why couldn't we get some little ol' 50-cent dog? He's not ours. Betty, you know who always winds up taking care of the pets you children bring home. Mother, please, may I tell you why this dog is here? Yes, I'd like an explanation, too. And an introduction. This is the first opportunity I've had to meet a $300 dog. Well, his name is... I don't suppose I'm properly dressed for the occasion. Maybe I should go upstairs and change.   What did you say his name was? Pierre. Bonsoir, Pierre. He doesn't think much of your accent. Well, now that the formalities are over, to what do we owe the presence of this imitation hound? He's not imitation. He belongs to my typing and shorthand teacher, Mr. Fawcett. And Mr. Fawcett is in Chicago attending a teacher's convention. You mean we don't get to keep him? No. How can a typing teacher afford a $300 dog? I know I couldn't. He was given to Mr. Fawcett by a dear friend, and Mr. Fawcett prizes Pierre very highly. How long do we get to have him? Just till tomorrow morning. Aw, heck. Mr. Fawcett gets back tomorrow. Sounds like a short convention. Oh, well, he's been gone all week. You see, some of us girls in his class have been taking care of Pierre. I see. Pierre is making quite a week of it. Well, we figured it would sort of put us in solid with Mr. Fawcett. Oh, so this is a slightly political move, huh? Yes, and listen to this, Father. There were four of us, and Mr. Fawcett was going to be gone seven days. So, in dividing up the days, one of us could only have Pierre one day.  No, I say you showed great foresight there. Well, I did. I'm the one who gets to return Pierre to Mr. Fawcett, and he'll thank me and sort of remember me as the one who took care of his dog, which he prizes more than life itself. Now, if you just put that much thought to your schoolwork, you wouldn't have to resort to such maneuvers. Let me take him out to the kitchen and feed him some bones and stuff. He looks skinny. He's supposed to look that way, and don't feed him. Pierre is on a strict diet. Where's the dog going to sleep tonight? Can he sleep on the foot of my bed? He'll probably want the guest room. Oh, Father. It's too bad we don't have one. Hey, Mom. Let me feed him, please. No, I'm going to do it. Mr. Fawcett would simply die if anything happened to Pierre. Say, Mom. And so would I. Hey, Mom, guess what I got? I hope it's not a dog. No, it's a carrier pigeon. What? What are we running here, a pet shop? I got it from... Hey, what's this thing? This happens to be a dog. DOG, dog. Ever hear of one? Holy cow. What happened to him? Oh, keep quiet. Looks like somebody ran over him with a lawnmower. He's just been flipped. I'm keeping him for Mr. Fawcett. Has he got a name? It's Pierre. Pierre? Yipe! All right. Come on, Pierre. We don't have to associate with such utter plebillia. What did she call me? There's nothing to worry about, Bud. Just leave her alone. Where's your pigeon, Bud? Can I see it? No, you stay away from it. You're liable to let it get loose. Aw, heck, I can't touch anything around here. Bud, just what do you plan to do with a carrier pigeon? I got it from Ori Pugh. Well, that ought to explain something. But who's Ori Pugh? He's a guy. Oh, I see. Will he really carry messages? Well, sure he will. We've been trying him out this evening. Oh, boy! Can I send a message to Patty Davis? No. Why not? The only place he'll carry a message is back over to Ori's house. But I can't think of any messages I want to send to Ori. I can see how that would foul up your communication system. The main reason I got the pigeon is so I could print up some messages with that printing set of mine. But that's not working out so good, either. What seems to be the drawback there? Well, to get the pigeon, I had to trade the printing set to Ori. Well, that was good clear thinking. I think I'll tell him I want to trade back. Hey, there's a message I could send him. Are you going to send it right now? Yep. Can I watch? Okay, but don't give me any trouble. Oh, I won't, Anna.  Well, at least that one-way carrier pigeon was acquired legally. But I'm a little surprised that Betty would stoop to taking care of a silly dog, just a red apple of teacher. No, don't start worrying about it, dear. I'm not worrying. I just said I was surprised. Doesn't it seem a little underhanded to you? Well, it all depends on how you look at it. No matter how you look at it, she's trying to butter up this Mr. Fosdick. Fawcett. Fawcett. All right, whatever his name is. Skeaming to be the last one to keep the dog so she can return him. What an operation. Now, don't start building this up in your mind, dear. Who's building anything up? Here, Pierre. Hello, baby. What a dog. Looks like a marshmallow sundae. With fur. Here, doggie.  Well, how do you like that? He passed me up. He's going right to you, dear. Well, hello there, poochie.  He's in here, Betty.  Okay, up you go. What's he doing? He's talking to your father.  You know these dogs have intelligent faces at that. There's something very honest about a dog. Where is he? Oh, we'll be careful of him, father. If anything happened to Pierre, it would be the end of my college career. Simply the end. You've made that point very clear, Princess. I believe he likes me. I wonder if it's true that dogs have an instinct for judging character. Oh, let me have him, father. It's time for his dinner. Come on, Snooki-Poo. Snooki-Poo? You know something, Margaret? What, dear? It'd serve Betty right if something did happen to Pierre. What? Oh, nothing permanent or painful, but... Mother, where did you put that box of special dog food I brought home? Down in the cupboard, beside the sink. Uh, where are you going to feed him, Princess? Out on the back porch. Better be sure he can't push the back door open. Oh, he can. Dear, what are you... Never mind, honey. I was just thinking that... I didn't let him out, bud. I didn't say you did. Where do you think the pigeon's gone? I know where he's gone. He's gone back to Ori's. I gotta call him right away. What's the matter, pigeon problems? Yeah, he's gone. Hello, Ori. This is Bud. Say, will you look outside and see if Vern's over there? Yeah. Who's Vern? That's Bud's carrier pigeon. My name for a pigeon. Ori named him after his uncle. The pigeon's uncle? No, Ori. The one that used to be a Western Union messenger boy. Well, I can see the connection there. Oh, mother, you ought to go out and watch Pierre eat. He actually has manners. Take your word for it, dear. Eating with a fork, is he? I have to find something to make a bed for Pierre.  There's a push it open. Jim, why do you keep talking about the back door? Oh, nothing. What happened to Ori, Bud? He lost two? He's coming. Yeah, Ori. Oh, he is, huh?   This thing gets more complicated all the time. Well, okay then. Here's the message. I want to trade the pigeon back to you for my printing set. You will? Okay. Goodbye. What did you tell him the message for? He's going to print it up on the printing set and bring it over when he brings the pigeon. Then I'll send it back to him. Did he say he'd trade back? How could he tell that? He hasn't got the message yet. Well, now I've heard everything. If the pigeon ever finds out what's going on, he'll go south for the winner. Excuse me, I think I'll wander out and see if Pierre is ready for his crepe sousette. Mother! Didn't you take it up to the lake last summer? I think I'll go out in front and watch for Ori. I'll go with... I'll go somewhere.  What? Yes, why? I didn't want her to see me come in. What have you been up to? I'm going to throw a little scare into Betty. When she goes out to the back porch for Pierre, he won't be there. What? I bedded him down in the garage. But let her think he's lost for a while. Teach her a good lesson not to play school politics. Oh, Jim. Let her think she's going to have to replace that $300 dog. Let her wonder what she's going to tell Mr. Fawcett tomorrow. It'll serve her just right. Are you sure he's all right out there? Sure. The door's closed tight. He can't get out. I can hardly wait to see the expression on Betty's face. Is that you, bud? It's me, Mother. Oh, did Bud know where the blanket was? No. Uh, Princess, and you better see if Pierre's all right. He hasn't made a sound out there. I'll be down in a minute. Dear, do you think the garage is a very good place for it? Someone could come along. No. He's perfectly safe. Say, Mom, Ari hasn't called, has he? No. We've been right here. He hasn't showed up for the pigeon yet. By the way, Bud, kind of steer clear of the garage. I mean, don't open the door. Don't open the door. I just closed it. What? Yeah, it was open. Oh, no. Where was the dog? I don't know. No dog out there. No dog? What's the matter, Dad? Out of the way! You better take a good look at your father, Bud. You may not see him for a long time. $300 for some information he badly needs about a missing dog. And friends, here is some information from Ed Prentice, which may be worth a lot more than $300 to you. You know, when you're not sleeping nights, when you're always worn out, cross, and nervous, you'd give almost anything to clear up the trouble, wouldn't you? Well, your trouble could be coffee nerves. Yes, coffee nerves brought on by the caffeine in coffee or tea.  That's a scientific fact. Yes, caffeine may leave you too nervous and upset to sleep properly. And while many can handle coffee okay, others, and this could mean you, suffer sleepless nights, jittery nerves. What to do about it? Well, you don't have to give up enjoying a good hot drink. Well, see, just make it postum instead. Delicious instant postum. There's a good sound reason. Postum is totally without caffeine. Not a single bit of it. No wonder so many folks have switched to postum and sleep. Why don't you make this simple test? Drink postum for 30 days. Postum exclusively. Then check up. See if you aren't sleeping better. Feeling better too. Try postum and see. You'll like it. Delicious instant postum, made instantly in the cup. Well, there used to be a game called Button Button Who's Got the Button. This evening in the White Frame House on Maple Street, a variation on that game is being played. You might call it Strudel Strudel Who's Got the Poodle. Now, the one who is most frantically searching for a certain toy poodle named Pierre is Jim Anderson. Meanwhile in the kitchen, Bud is trying to catch up on the current crisis. Like this. Holy cow, Mom. What's wrong with Dad? Well, don't say anything about it, Bud. But your father put Pierre out in the garage. Now he's missing. Oh, that's why Dad went tearing out the back door. But what did he put him in the garage for? It's a long story. The point is, Pierre is gone, but his father has to find him before Betty finds out. Maybe a butterfly chased him. Father! He's outside, Betty. I'm going to go out and see if I can help him. Poodles.  I'm waiting for Ory. He's bringing the pigeon over. Where'd Mother go? Outside. Where's Father? Outside. Oh, Betty's looking for Pierre, isn't he? Is he? Don't you tell anybody, but I have Pierre up in my room. Huh? I heard Father talking out here. He hid him in the garage just to scare me. So I went out and brought Pierre in the front way. You better go out and tell him. Oh, no. Let him stew for a while. It'll teach him a lesson.  Stop. You're breaking my heart. Oh, I didn't know you were in here, Betty. I've been here all the time. Where's Father? Oh, um, he's outside, dear. Uh, weren't you planning to go out tonight? Me? Oh, no. I have to stay home and take care of Pierre. Where is he, Mother? Pierre? Oh, he's outside somewhere. Outside? Oh, your Father's out there, too. He's all right. I'm sure. I think I'll go outside, too. I gotta watch for Ory. Mother, what's Father up to? Well, uh, you see, dear, he... Margaret, have you... Oh. Hello, Princess. Father, where is Mr. Fawcett's $300 French poodle? Uh, Mr. French's three poodle... I mean... Oh, he's all right. You don't think I'd let anything happen to him, do you? Oh, I'm not worried. I'm sure you know exactly where he is. And I'm perfectly confident that you're taking good care of him. Oh, sure.  I'm going to put him to bed in a few minutes. Uh, all right, Princess. I think you'd better sit down, dear.  Margaret, where do you suppose that darn dog went to? There's not a sign of him anywhere. Don't look at me.  I've got to find him. But where? I'll talk about looking for a needle in a haystack. That four-legged powder puff could hide under a peanut shell. And all the silly ideas, putting him in the garage in the first place. Who's that? Maybe it's Pierre. He seems to open and close doors pretty readily. Oh, he showed up for the pigeon, finally. Yeah. But sent the message back, Dory. Look, Kathy, but... I don't know if Fawcett will be there to get the message, though. As fast as he could, but the pigeon was way ahead of him. Why didn't you hold the pigeon? Wait till Dory got home. The message was urgent. Oh, dear. Look, kids, I have a very serious problem, and Bud, you and Kathy can help me. What's the matter, Daddy? Well, don't say anything to Betty, but Pierre, he got away somehow. I want you both to go out and comb the neighborhood. I'll go, Daddy. But, Dad, there's no use going out. We've got to find him, Bud. You kids know all the nooks and crannies around the neighborhood. He can't be very far away. I'll even give it a reward for him. Dead or alive? I'm serious, kitten. Five dollars in cash. Five dollars? Holy cow! Oh, Bud, get your flashlight. Yeah, but he's not a... Well, I mean... Okay, you can stand around if you want to. I'm going out and look for him. Anything to eat in the refrigerator, Mom? I'm hungry. Find cooperation from my son. There's only one place to look for anything, as far as he's concerned, in the refrigerator. Well, dear, you brought this on yourself. Why did Betty have to bring the fool dog home in the first place and a $300 dog? He ought to be in a safe deposit box. I'm going to phone some of the neighbors. The Davises have a dog. Maybe he went over there. Where's the peanut butter, Mom? Right where it always is. I can't understand you, Bud. When your father asked you to go out... But the dog isn't outside anywhere. I mean, what's the use of... How do you know he isn't outside? Well, I'm not supposed to say. Who said you're not? Well... Bud, there's nothing that concerns the family that you can't tell me. But gee, Mom, I... Bud, do you know where the dog is? Well, sort of. What do you mean, sort of? Is he in the house? Yeah. Kinda. He can't be kinda in the house. Pierre is either in or out. Now where is he? Well, I wasn't supposed to say anything, but you see, Betty's trying to teach Dad a lesson. She got the pooch out of the garage and he's up in her room. Oh, I see. Well... Lines are all busy. What in tarnation am I gonna tell Betty? Bud, you might take just a little interest in all this. $300 doesn't grow on trees, you know. I know, Dad. Any jam around, Mom? I wonder if it would do any good to call the police. Oh, that's probably for me, dear. I'm expecting a call from Mrs. Rodney. Ask her if she's seen a silly-looking French poodle. Hello? Oh, Mr. Fawcett. This is Betty's mother. She didn't expect you back until tomorrow morning. Oh, I see. Yes, yes, we have Pierre. He's fine. Oh, it's perfectly all right. That'll be fine. Goodbye. Who was it, Mother? Did I hear you say that was... Yes, that was Mr. Fawcett. He just arrived home. He's coming over to pick up Pierre. Coming over? What's the matter, Father? Coming over here tonight? In the dark? I imagine he has lights on his car. Yes, but... Excuse me, I'll be right back. Well... Father, you haven't let anything happen to Pierre, have you? Princess, believe me, I... I didn't intend to... I have to tell you, the dog's gone. Oh, Father, how could you? But I'll buy Mr. Fawcett another one. Or I'll find him. Or I'll... All right, Father, you've suffered enough. I took Pierre out of the garage. What? I was just teaching you a little lesson, not to be so quick to jump to conclusions. Oh, Princess, that was a cruel thing to do. Not anymore cruel than what you were doing. Thank goodness he isn't lost. Where is he? Up in my room. If Mr. Fawcett's coming over after him, I want to brush him and put a ribbon on his collar. What's going on? She's still chasing that cream puff? I can't find Pierre. Not me. I hate to say this, but if you'd left him in the garage where I put him, this would never have happened. If you'd left him on the back porch where I put him, it wouldn't have happened either. This family's gone to the dogs. Nobody. Here I am, facing disgrace, utter disgrace. I'm sorry, Princess, but this shows what happens when you try to take things into your own hands. Well, what's all the wailing about now? Mr. Ambassador has gone underground again. Pierre? He's dropped out of sight. Mother vanished. I had him in my room just a few minutes ago, and now he's gone. How could that be? What was he doing in your room? Well, I was getting even with Father a little for hiding him in the garage, so I had him in my room. Now, suddenly, he's nowhere in the house. What am I going to tell Mr. Fawcett? Let's see, $300 and your allowance is $2.50 a week. That'll be about... Oh, Father, don't think about it. Pierre has to be in this house somewhere. Well, I hope you both learned your lesson about trying to teach other people lessons. I have Pierre. Huh? You have? He's sound asleep in the broom closet. Oh, thank goodness. Well, let's get him out before Mr. Fawcett comes. I want him to look his best. Oh, I forgot to mention, too, Mr. Fawcett is coming to pick up Pierre, but not until tomorrow morning. Mother... Margaret, that's not very fair. I just wanted to teach you two a little lesson. Now, I think it worked out very well. Well, let's get Pierre out of the broom closet. Well, he's right here. He's right in this... where'd he go? Oh, no, not again! But... but he was right here. I put him here. Well, let's see. Your household budget is... Oh, but Jim... He was here. I know he was here. I wonder who's teaching you a little lesson, dear. If your family races off mornings without a good breakfast, chances are they're not getting proper nourishment to carry them through the busy morning hours. Now, actually, a good breakfast, one that supplies one quarter to one third of your daily nutritional requirements, is inexpensive and easy to fix. Many authorities agree breakfast should include fruit or fruit juice, bread and a spread, milk and cereal. And which cereal? Well, for flavor and for health. Just try post 40% brand flakes. Here you get the famous keep regular benefits of brand, plus a wonderful new magic oven flavor. Yes, start serving better breakfasts and be sure to include post 40% brand flakes. For goodness sakes, eat post brand flakes. So good and so good for you. Well, it's a few moments later in the White Frame House on Maple Street, and still no clue to the missing Pierre. It's a rather tense moment, as the Andersons face the awful truth that the precious poodle this time has dropped out of sight once and for all. Like this. Well, I've looked everywhere. Not a trace of him. He's not in the basement. Oh, if only you and mother left him alone. If you hadn't brought him home in the first place. Father, if you had left him... I know, I know. Reward? All right, kiddin', we give up. Where did you find him? We're just selling brand flakes. An instant postum, the drink that's entirely caffeine free. In our cast, where Ted Donaldson is bud, Gene Vanderpile, Rhoda Williams, and Norma Jean Nielsen. If your family likes to pick and choose, give them the chance at breakfast tomorrow. Yes, let them choose their cereal for the day, the handy post tens way. Post tens gives you ten individual packages with seven famous post cereals. It's the only assortment with two leading sweet coated cereals, crinkles and sugar crisp. Post tens make a mighty thrifty breakfast treat, too. Each package holds just one serving, so there's never any waste. So when you're shopping, remember for the best choice, make your first choice, post tens. Father Knows Best was transcribed in Hollywood and written by Paul West and Roswell Rogers. This is Bill Forman speaking. This is NBC, the national broadcasting company. Thanks for watching.


Good morning. Good afternoon. Good evening. Welcome to another ask an open shift admin office hours. I am Joe I am Chris short's executive producer of open shift TV. I am joined by the one and only Andrew Sullivan You know, what do we want to call you like vert open shift? Star to the gods what I mean the cold guy. Yeah, you know that That guy that says this is documented just do it like that Where's Chris Christian calls me the cuddly curmudgeon, right the cuddly curmudgeon exactly right like very approachable, but like don't Don't ask him dumb questions. No on this show.  So no, it's uh, I'm I think Christian calls me a curmudgeon because I tend to be very conservative in my approach to things that's a It's a Admittedly a learned behavior, right? You know, I was a an administrator for a long time so, you know, the the first big-boy job if you will that I had was as a an Administrator for a production v-sphere environments and our our storage admin decided one day that nope I'm just I'm done with this. I don't want to do this anymore. So I was also the storage admin and You know learned the hard way and learned through trial and error things like You know, don't accidentally delete the NFS data store from underneath, you know v-sphere. That's bad You know, don't don't trust the network guys when they say, oh, we'll just reboot the router It won't affect anything because no no all of that IP based storage you're using is traversing the router God So yeah, it it's one of those learned behaviors of trust but verify, right? Like you've just reminded me of a story about a reboot the router story where I had to like literally stop somebody's hand from touching the power switch before right like No, don't do that Yeah, it's Yeah, and my favorite one was I was working with the government and we had a junior enlisted person who was responsible for vacuuming the data center and they Were traversing through doors and the vacuum hose caught the lid for the EPO right the emergency power off button and of course when the lid lift ups lifts up it sets off an alarm right and The immediate reaction was well if I mash this button the alarm will stop Which was true It's not the alarm along with the rest of the data center. So wow that was that was a fun evening Just wow. Yeah, so anyways cuddly curmudgeon You know, it's less about you know asking questions or you know I try to never take the approach of you should already know this right? Why don't you know right? No, because you know, what's what's that xkcd? Let me dig up the xkc. Well, it's not yeah, it's the it's the you know, don't I use Julia Evans like don't feign surprise right like treated as an opportunity to learn I don't have a printed out, but I'm pretty sure it's still hanging up on my basement My basement office that is kind of a construction zone right now But it's you know treated as an opportunity to teach somebody new and exciting to them not you know Like oh, why don't you know this already right like a man page? No, we don't want people like that in the industry right like we want people that are enthusiastic about teaching Yeah, and it's one of those like When I was an administrator when I you know I eventually became an architect and all that other stuff and it became one of those the more you know the less You have to ask me right want to be a single point of failure. I got other stuff to do I don't need to do your job and my job. So it very much becomes one of those Education teaching helping rates bringing everybody Along and up and that's one of the things I love about red hat right red hat has that culture just ingrained inside of red hat I've never sent an email and you know asking a what is potentially a dumb question and gotten a response other than Absolutely, let me show you how that works, right? JP Dade that's just about the same. Yeah, I Haven't seen an update recently as far as I know they are still blocked Let's check and see Here, I'll show I'll share my share I want to get the link real quick. So what are you sharing the Cincinnati graph? Yes, so Get hub.com slash open shifts and if we go to Cincinnati, I can never remember how to spell Cincinnati I just remember this. I only know it because my daughter went to school there If we go to blocked edges and we check way down here on four dot seven four It's still got Blocked here and with a link out to the same BZ that we talked about last week This is only impacting these fear Yeah, so this is one of those Releases are done holistically across all platforms. So there isn't a an upgrade or there isn't a It's not I can't target a release that says you can upgrade all platforms except these fear It's either all or none in that case. Yeah So in this instance, yes all upgrades from or to open shift four dot seven are blocked as a result of a bug That we found in vSphere So for those customers who aren't using vSphere, right? Yeah. Yes. I I know and even for those who are using via vSphere, right? It's frustrating. Yeah, we're two months past now We feel your pain trust. Yeah Trust me We feel it both on a personal level of I'm constantly trying to deploy and troubleshoot and work with these things and and you know help you all our audience our customers You know with these problems, so I feel it as well as I know there's a lot of pressure You know coming into you know red hats into our organization around figuring and fixing that so Keep an eye on this BZ. I'll post the link to the BZ and I got it right here. Okay. Thank you No problem, but yeah, unfortunately Yeah, there's there's still blocked and I know that they are working diligently to get that figured out and again I see those emails between you know our team and the VMware team and there's kernel developers involved and all kinds of other stuff There's yeah, there's a lot going on behind the scenes. Yeah, absolutely So I know we got a little distracted there with storytelling But just to reiterate or just to remind everybody this is the OpenShift administrator office hour Or excuse me. This is the ask an OpenShift admin Man, I reverted back to the wow We've had this title for how long well I was I was doing work yesterday to go, you know to document and retitle the original episode Making the list for me. Yeah, sorry about that. Hmm. So what that means is that our goal here our Our purpose the reason why we have the stream is for you to ask us questions doesn't matter what it's about doesn't matter You know whether or not it's something that we can help with and what I mean by that is if we don't know the answer We'll find the answer right and we'll get back to you Yeah, it's one of my favorite things as I was just saying is just finding the right people inside a red hat and Getting those answers, you know a couple weeks ago. We had Mark Curry on with the we were talking about Moltis and and CNI and all that other stuff and You know Doug and and Oh right now told me Yes, both in engineering on you know and working in Moltis I have learned so much from them and seeing the email chains that have been generated out of that So you all our audience There was a couple people who sent me emails and basically said hey How does this work or how can I do this and I've you know connected the dots there or connected the people there And I've already learned just a tremendous amount about how all that stuff works So to me it's it's really exciting to see all that stuff Nice, so yeah at any point in time, please feel free ask questions Whatever it is that is at the top of your mind something that's bothering you some problem or issue that you're having Architecture questions functionality questions all that stuff. Please. Please. Please. Please don't hesitate to ask those questions Yeah, we encourage it and if for whatever reason we can't get to your question We can't answer it here on the stream. I always follow up each one of our episodes with a blog post So if we switch over to this tab here, this is the blog post from last week I'll post that into the chat here And you can see that we essentially do a synopsis So coming down here remember last week. I said, hey, you know We we don't have enough time to cover all of these items all of these different parts and components But I'll provide links to all of those inside of the blog post. Well, here's the Yeah So you can absolutely find all that we try to get those published every Friday so far. We've been pretty consistent with that So check OpenShift comm slash blog on Friday mornings Usually and you can see this week's blog post including links to we come all the way down here to the bottom You can see we have links directly to in the Stream where we answer those questions, right? So any questions that you happen to bring up? So with that being said or I guess the next Parts or the next thing that we tend to do here and I'm sorry I keep looking over this way that your little window is was way over here. So when I sorry when I go to look at you I'm actually looking over there. So remember. I'm the camera look at the camera. That's what I do I put look at me look at me So the producer now Not just now just always So one of the things I like to do here is Talk about the things that have come to my attention the things that are tough of mind for me that I think are important to relevant for you all So this week, there's a couple of things that I want to to bring up in that respect So first I've had quite a few conversations over the last Six or eight or nine months around the load balancer that's used with IPI So if you're not familiar with this effectively with on-premises IPI deployments So these fear redhead virtualization open stack and bare metal IPI We deploy a built-in load balancer So what that means is when you go through the installation process, right? It asks you for things like what is the apps virtual IP address? So star dot apps the ingress IP address and what is the API so the API dot cluster name dot domain name IP address and those two IP addresses are Managed by keep alive D. So keep alive D puts that IP address onto an eligible host DNS directs all of that traffic under the host and You get your fully functional cluster So the problem with that is particularly for the apps for the in the star dot apps ingress point It's limited to a single node in the cluster It's one IP address. It's that IP address is hosted on a node that has a router instance right the HA proxy OpenShift router instance And it doesn't do any real load balancing behind the scenes So if you have you know, if you scale your router instances from two to three to five to 12 or however many you have All of that traffic still goes into one those additional instances are just there for high availability Or failover capability So the question is Can I use an external load balancer with on-prem IPI and The answer here there's really kind of three answers that or three possibilities that you have here so the first one is more or less to Replace the ingress load balancer. So what do I mean by that you could for example? And let me see if I have the case. Yes handy here Let's see I Have to I have to search in one window where I'm actually logged in and oh, yeah, I feel that yeah Here we go. Well, I said that and now I clicked the wrong thing Anyway, so for example usually the example I use here is f5 So the f5 folks have a certified operators if I go to catalog redhead.com and I come up here and look for OpenShift operators and I search for f5 We get back this f5 controller for OpenShift and Kubernetes So for you if you were to use this operator and they aren't the only ones Citrix has one Cisco ACI has one right. There's a number of those Effectively what it does is it deploys the f5 ingress controller So instead of using the default red hats OpenShift provided ingress controller that is based off of HA proxy instead You use the f5 ingress controller which coordinates with an external f5 appliance Same thing for Citrix and their ADC controller so on and so forth So that's one option right just don't use the default one right instead use the one You know use a different ingress controller with your cluster and then you can absolutely take advantage of kind of whatever You know, whichever of the load balancers you want to Great. So the other option and this is something that as far as I know is only documented by OpenStack So let's go to docs So if I go to the documentation and scroll down to networking and scroll down to load balancing on OpenStack They have this configuring an external load balancer section, and I'll copy and paste this into the chat and effectively what this is documenting is for on-prem IPI with OpenStack Here's how you can configure an external load balancer and then effectively swing that ingress and API endpoints over to them So unfortunately, this isn't tested or documented with any of the other deployment methods So therefore we can't authoritatively say yes, it will work and therefore it falls into kind of a support gray area So I haven't been able to get an answer out of the support folks as to yes It's documented for OpenStack, but does that mean it's also supported with vSphere for example Even though the infrastructure isn't really material in that instance So just FYI this is here if you happen to be using OpenStack and you want to migrate off of that Default built-in load balancer and you're not using, you know, the courier or the Octavia integration This is one option that you have available So the third option is kind of Effectively replacing that but using something, you know unofficial and that could be as simple as You know following the documentation here and basically I'm not going to talk about it because it's it's definitely not supported And I don't want to encourage that at all So really the two options are use a difference ingress controller or Use an external load balancer following this OpenStack Configuration here, so there is some stuff going on here, you know behind the scenes around how can we make this better? How can we configure this? How can we make it more configurable? So just to keep an eye out I do have somewhere in my multitude of tabs. I do have an RFE that I can dig up for this that way we can Take it or that way we can track that progress and If it's something you any of our any of our audience right if you raise that with your account team with them No, they can just reach out to me and then we can kind of associate account information with that which helps product management with prioritization Basically, they they they use your input as customers as a huge contributor to what's important and what should be prioritized from an engineering perspective. So So JP data asks can't you spread the ingress controller over a set of info nodes, right? Yes, yeah, yeah, so with UPI absolutely Or even the non-integrated install, right? I I can it can be info nodes. It can be, you know Router nodes you can tag them whatever you want, but essentially you can scale that ingress controller to you know 25810 however many instances and Normally with IPI excuse me with UPI or a non-integrated install You have an external load balancer that is pointed at each one of your Nodes that's eligible or that's hosting a router instance, right? A HA proxy ingress instance So incoming traffic comes in it hits that external load balancer The external load balancer looks at all of the nodes that it has available to it so all the ones that are running that HA proxy ingress controller instance and then it routes that you know external session over to one of those and You know assuming and I think the documentation by default says, you know configure all worker nodes, you know inside of that external load balancer So if I have 12 ingress instances 12 ingress HA proxy instances, then it would be load balanced across all 12 the problem with IPI on-prem IPI is That external load balancer basically doesn't exist So it's just a single virtual IP address right the start out apps the one associated with the start out apps DNS name And it just moves from node to node You know with and only on nodes that have an HA proxy ingress instance on it So it would be the same effect as if you had an external load balancer that only had one node configured on the back end Yeah, not so smart. Yeah, but anyway I'm sorry meaning multiple you don't want a single point of failure spread it out. Yeah Yeah, and you know the keep alive the solution rate the the on-prem IPI solution does that rate keep alive D Maintains that virtual IP address. So if the node that's hosting it goes down for any reason it moves to a different node So you only have a slight blip Anecdotally what I hear is somewhere between one and five seconds So you have a slight blip in that ingress traffic as the IP moves and then a reverse ARP goes out to let the infrastructure Know that it moved and then things returned to normal. So Yeah, that's where we stand today. If you have any questions about that if it's something that is an issue for you Just let me know Andrew dot Solomon at redhead calm and then we can kind of work through that and look at what those different options are as well As work with product management engineering So, yeah Yeah, you're gonna love this one DHCP reservations DHCP it's it's a reoccurring theme. I think we have talked about DHCP in more episodes than we haven't talked about DHCP That's probably accurate now so a reoccurring question that we get is if you look in the documentation and I'll see if I can find it here real quick So let's go to installing. I'll pick on vSphere this time And we'll just go with this we'll go with this one installing a cluster So if I do a quick search for DHCP and Reservation no reservation. No Somewhere in here we but we more or less tell you, you know, yes with IP I DHCP is required But requirements, yeah So We tell you Blah blah blah Anyways, the documentation says DHCP is required for IP I because it is right. We require you to The nodes to be able to dynamically get their IP addresses but then we also say static DHCP reservations are encouraged for worker nodes and should be used for Control play notes So the rationale here is and if I come back over here, let me see if I can search the KCS this time Let's just go here. Well, thank you. That was not what I wanted Come on, so if I search for Open shift DHCP Reservation I should have had this link prepared right probably Here we go. So let me post this link in here and Effectively what this says is I guess I could so effectively what this is going to say is that When you use DHCP You want to make sure that your particularly control nodes always get the same IP address Very important. Yeah.  Basically The node goes down, right? Basically it does a release and then Something else immediately is requesting an IP address and there's so few available that it immediately starts reusing the ones that are there That stuff So we make this recommendation of use static, you know, you know, DHCP leases DHCP reservations for your control plane nodes the rationale being if The control plane node IPs change at CD can't reform itself Right and if at CD can't come up then the rest of the cluster can't come up Normally if only one node at a time changes the at CD operator cluster at CD operator, we'll be able to recover But if all three change then it can't reach its own quorum and that's bad So For that reason we encourage you to use DHCP reservations. The question is but it's IPI I don't know the MAC addresses beforehand. The installer creates the control plane nodes And machine API creates worker nodes. So how can I create those? And the answer is both simple and frustrating and and that is more or less After provisioning use the tools of your DHCP server to convert it to a reservation I Have reached out to some VMware peers I've reached out to some others and to figure out things like, you know VMware has this concept of a MAC address pool Can I create a pool of known addresses and then associate that with my open shift VMs so that they only, you know Whenever it provisions a new node, it only comes from this pool and stuff like that More or less what I got told is well, it doesn't work that way So I'm hoping we can make this easier in the future But honestly, I don't I don't have any and I have not seen any information with that regard So just FYI When you see that we Suggest and encourage DHCP reservations, especially for control plane nodes, you just have to do that day two, right? So after the after the node is deployed change that over to a DHCP reservation I will note that if you If you're deploying physical IPI bare metal IPI and your DHCP lease time is set to infinite Then it will automatically convert it will automatically apply a A bit of automation and convert that to a static IP assignment or a static IP configuration On the physical node only so remember this is bare metal physical IPI not the other installation methods Let's see Suresh has a question if you have a second. Yep, and and Suresh We're going to probably need some more details about your configuration specifically, but Um, is it mandatory to have DNS records of kates nodes an external DNS server? No, it's not mandatory, but it depends on how you set things up to like Yeah, so exposing the control plane to the public Yeah, so with IPI No with IPI it will deploy an mdns instance And all of the nodes will register themselves with mdns. So it will do internal resolution So effectively control plane nodes can find worker nodes and vice versa The only time that would be an issue is if you want to connect directly to a node So if you're trying to for example ssh from your workstation into you know worker 17 dot cluster name dot domain name That dns name wouldn't be externally visible Unless you're doing like a DHCP dynamic dns update right in which case it would be visible Uh with UPI I don't believe they're required No, I don't think they are either brings in my mind. It brings into question whether or so host names are set a couple of different ways and and the most prominent and and the way that will take precedence is Reverse lookups So if there is no reverse entry I don't know how it will react to that because normally it would want to use the node name Like if you do an oc get Node, right? It'll show the node name inside of there if it can't look that up If it doesn't understand how to talk to that I don't know how that would work if it would just return back an IP address or something like that Um, honestly, it's not something I've tested. So I'm not sure. I know in the documentation We say that you know here if we look at the So this is ipi Switch over to Installing a cluster on vSphere with user provision information or infrastructure Words are hard If we come down here to networking requirements I believe it says that dns We we basically say dns is required and I know yeah here user provision dns requirements um So I I know that we say that it is required here So yeah your control plane nodes worker nodes um, and and that's just To avoid I if nothing else that is to avoid a lot of potential issues So I would say follow the docs. They are required with Uh, upi with ipi It's not on the docs. It'll take care of itself Keep alive ds poor man's f5. I wouldn't even go that far. I wouldn't even go that far It's uh, it's just to keep the ip address accessible Right, it doesn't do any kind of load balancing any kind of you know anything like that. So um So the static ip question I read somewhere haven't tested that if we offer a dhcp ip address with an infinite least Red Hat core oest will configure the ip address as a static ip address Yes, and that's what I was talking about with It's only with bare metal Right, so we can see if we go to github Type in the window here So if I go back to open shift github and this time I went to go to the machine config operator And I'm going to switch the branch here to I'll go to release four seven And in here we're looking for the templates directory and then we go into I think it's common And then Bare metal and files Yeah, so we have this network manager static dhcp.yaml And effectively what this is doing is exactly what we just described Right, it's it says is this an infinite lease if so i'm going to apply the ip address as static Um, but you'll note it's only with bare metal Right and and specifically bare metal ipi So if you're using you know any of the other ipis on prem, it doesn't take effect if you're just doing a A bare metal upi or a non integrated install. It doesn't take effect, right? It's only bare metal ipi So yeah, you can see exactly where that logic lives and what triggers it inside of here because I think if we go up a window or up One of these here is the one that basically says If it has the template for the if statements where it detects the installation type An installation method if infrastructure installation infrastructure Um jp did on my upi installed to configure dns and dhcp Um Yep, a whole lot of settings and dhcp to set hostname and ntp. Yes Yeah, yeah, um, so the ntp if your dhcp server is serving ntp. That's a nice Convenient way to get everything configured correctly. If not, we talked about this last week In the day two stuff of figuring No, I don't know where it went inside of here, but somewhere inside of here is configuring crony Yes Now so The link directly to the documentation inside of here for configuring the time synchronization service using machine config in this instance, right? I mean is everything folks Especially in kubernetes because kubernetes makes great use of certificates and certificates are very finicky about Is this valid yes or no? Wait why is this time offset different from this box in this box? It will pick stuff up like that and you'll see some weird errors as a result So yeah, please keep the questions coming in. Um, so don't don't hesitate You know with great power comes more settings. Yes. Yes. Yes. Absolutely true. Uh, so The last thing I wanted to talk about is if you missed this blog post on openshift.com August ciminelli who is one of our peers? From down under he lives out in Australia He released an update to this tested Solution for open shift on open stack. So essentially he Went through it's it's not a reference architecture But it's a tested architecture tested and validated architecture for deploying Open shift on open stack. So Great resource if you happen to be doing open shift on open stack Be sure to check that out. I know he put a lot of time efforts Blood sweat and tears into that and it's a a hugely hugely beneficial. Yeah Thing if you happen to be using that architecture So what is the rationale behind having this behavior? Infinite lease to static ip limited to bare metal ipi only why not for other setups too? So I will say andrew's yeah So andrew's opinion is that or my my hope maybe is that we will expand that to other infrastructures as well Yeah, but I don't know that for certain. I would have to dig through the rfe's And see if we can find where that's documented at or yeah, which ones goes first kind of deal And everything else. Yeah So, you know the logic with bare metal with physical servers is pretty straightforward You know one it is the only time with an ipi where we're going to know the mac addresses ahead of time There's that you know because you can Mac physically look at it, you know, it's it's not actually, you know creating machines on the fly. It's just Configuring them on the fly. So I think that's one of the things that leads to it Um, I I hope that we do add it for the others simply because I know dhcp is a point of contention And this is something that would actually be really interesting to me for anybody who's listening in Why is dhcp so contentious right and and we've heard this basically since IPI on prem was a thing. Yeah, which is what open shifts 4.2 was when it was released with open stack You know of you know, hey, I can't do dhcp in my data center You know, can I do ipi without dhcp and a lot of times we get back this kind of hand wavy Well, the securities team won't let us And right while I can make a lot of assumptions there it would be interesting to know like what's the details there What's what's going on behind the scenes? Why won't they let us do it? if you will And you know in my experience it and please audience why won't your people you know Run dhcp in the data center. Please tell us why in my experience. It's because there is Anything that's not using dhcp that you would then have to configure and that anything could be of varying sizes between, you know four to four hundred, you know instances of something or Entire ranges of ip classes like slash 16s and so off So, yeah Maybe cisco proprietary os well if you're using your switches and routers as gateways and dhcp servers and everything else Yeah, there's some limitations there potentially Like hard to configure and limitations I mean right like not ubiquitous right like i'm not going to let my You know windows dns admin jump into the cisco router and start configuring dhcp. There's there's a learning curve there right so Yeah, I know You know when I was an administrator our security team wouldn't let us do dhcp and it was largely because their fear particularly in the land of virtualization was Right anybody could stand up something and you know, we don't know what it is and it'll just be an ip address You know kind of the old saying about you know, will anybody can walk through the data center and plug their laptop into a network port? Well, no, I mean technically yes, but that's why we have things like port security right So yeah, the jpd points out plugging in a foreign device under the data center is frowned on dhcp is good for workstations only Uh to which apostola says it's old school network security thinking to which I agree It is kind of an old school methodology, right like dhcp has been around a really long time and lots of people are using it in their You know data centers. Yeah I'll also I think some of it comes from um pre I'm gonna use the term cloud native here but yeah Kind of the the current and even really the last generation of applications You know when we think back to like the 2000s, you know 2010 and earlier Applications didn't change a lot right, especially when everything you know before virtualization really took holds You know servers were put into a rack. They were given a static ip address and then All of the things that needed to connect to it would a lot of times just use the ip address Right, they wouldn't use a dns name. Yeah, because no one wants to muck with dns because dns is dns What happens if dns goes down? What happens if you know, so, you know, my database server is that you know x.y.z.a and you know all of the Applications that connect to that database connect on x.y.z.a So yeah, that's an interesting one. Um, I've been Wondering that for a while as to why we don't see dhcp. Um, you know more widespread in data centers Cluster dhcp servers. Yes. Um, that is something that's um, absolutely if you were going to be using, you know dhcp in your data center, you know Something like open shifts which are ipi which requires dhcp make sure it is an enterprise grade service That was one of the things I learned from our field early on is that a lot of customers Even if they have dhcp in the data center, it's running off of like one VM and it's not You know, it's not treated with the same level of resiliency and availability that we see with other data center services So yeah, gpdade very much to your point to make sure that it is Treated and and managed appropriately for the level of importance that it holds Yeah, and I can't say I mean, I I'm sorry if I butcher your screen name my bad But in my experience most of the enterprise network admins setups don't like any much addressing because a conventional data center is everything For the VMs are predetermined with firewall rules and IPs and everything else. So yeah Um, so I know we're uh, just like last week We're pretty far in at this point and we've we've kind of brushed against several things that I wanted to talk about with regard today's topic, which is As the title suggests day two operations. So effectively, you know, hey, we just finished deploying our cluster Now what you know, what else do I need to do to get ready to actually use that cluster? So last week as you saw, you know, if you look at these headings here We talked about kind of node level or machine level operations that need to happen Or could happen Cluster level operations and then kind of some other things just to be aware of You know, many of these could probably be lumped up into the cluster side of things, but So this week I want to focus on network storage and Kind of preparing for users So network this almost always comes down to How do I do or how do I configure or how do I manage? The network configuration after deployment and especially and we're seeing this more and more now for people who want to have a secondary or tertiary or however many interfaces for additional network traffic or additional traffic types, I should say the most common one by far is A isolated non-routable storage network right I need to connect to nfs pvcs or iSCSI pvcs or whatever that happens to be And the storage team has a dedicated, you know, ip network specifically for that You know go back to my comments about routing and all of that other stuff and we'll just reboot the router. It's fine. It's good so What's the best way to do that? And I will say that and do I have it up here? Let's come up here to I want networking and we're going to Where is it in m state? um, so i'll say that there's really Two so there's three answers of which one of them. I don't consider to be a valid option. Uh, so The first one is you should configure those interfaces at install time So with upi or with a non-integrated installation That means either passing ip equals configuration parameters You're gonna multiple ip equals. So maybe if you're first interface, right? So i'm gonna, you know, eth0 is the one that is used for Uh, sdn, right? It is the primary interface. Maybe it uses dhcp. Great ip equals e0 colon dhcp And then your secondary interface needs a static ip address You can absolutely do that and you can use Uh, the standards right a second ip equals and then follow the normal nomenclature for doing that So i'll just to grab an example of that i'll come down here to installing on any platform quick search for ip equals And we can see an example here right basically exactly like this so Right here in the documentation We can see we can just append as many of those as we have interfaces And that includes things like creating bonds. So if we scroll down a little bit further here So we have this bond equals and then we can create bond zero that has two interfaces We provide the bonding mode and then I can then assign an ip address to that bond We do the same thing with vlands Uh, somewhere in here is the vlan documentation on how to create a vlan on top of that So this is one way that you can do that. It becomes persistent It is then controlled right throughout the normal open shift management processes You're going to do effectively the exact same thing using coro s installer and the live iso method so Boot to the iso it drops you into a command line prompt you can use nmcl i And then with nmcl i configure the network however it needs to be configured And then use the copy network. Here we go the copy network option When you execute coro s installer And that will take the currently configured network config in the live iso or the live environment And copy it to the Sorry for bumping the microphone copy it into the uh installed os on the host So that is with upi that is the suggested method of doing it Right with ipi it's going to be a little bit different In particular and i'm going to pick on vmware again So if we look at vmware with network customizations Actually, I don't want that I want But where is it machine sets So sarash points out. Yeah, I wanted to evaluate the nm state operator, but I'm waiting on the ocp 47 release for vsphere Yeah, yeah Sorry, buddy Yeah, um, uh, we all are kind of right so you you can So if you look at nm states, uh, and if you saw the tab that I have here You see this big red it's in technology preview only So if you deploy nm state or utilize nm state standalone It is tech preview if you use it with open ship virtualization. It is fully supported So if you happen to be running on physical servers, if you have bare metal available You can deploy the open ship virtualization operator Even if you don't actually deploy any virtual machines, you'll get the nm state operator It'll be fully supported and that goes back to 4.5. I believe I so yeah, you can you can use it there So to complete out my other thought around, uh, ipi and networking So if you're deploying for example with vsphere ipi you want to create a machine sets that will have multiple network device definitions here Right. So for each network that you need to connect to you create a new network name device here You add that in When the machine set goes to deploy that node it will ensure that each one of those is there So the first device presumably the one that is on the primary network that has dhcp enabled remember ipi requires dhcp It'll boot it'll get its ip address. So it'll be fine the second device You'll probably see some errors right. It'll probably complain and then eventually take a link local ip address That's fine Might delay the boot process by a few seconds that type of stuff Once once it's deployed once it's joined to the cluster then you can use What I would suggest is nm state So this brings me to the the third option that I don't really consider an option So yes, it is technically possible to use machine config Right, you can go in you can create a machine config Let's see here user provision deleting a machine We'll find it eventually Even if I have to search for it Somewhere in here Machine config pool machine config. Yeah, we'll go with this one So you can use a machine config And you can use the machine config to push that yeah here here's an example of one You can use that to push an ignition config that configures that network interface The problem is now you have to have a machine config per node Because that node right each node has a different static IP address and that machine config You can't just say, you know, here's I have 10 nodes in my cluster that all need static IPs I'm going to create 10 machine configs And then I'm going to put them all in the same machine config pool Because that'll push out the same config to all machines in the pool and right Well, they don't all have the same static IP address So now you have to have 10 machine config pools one for each one of the nodes in the cluster that inherits that chain Right, so it would be for example The worker and then the node specific and then you end up with a machine Machine config specific to that pool to configure the worker's static IP address Technically possible nothing wrong with that. Absolutely fully supported a management nightmare Like and and I mean to put it mildly in many different ways because It bleeds over into things. Well now I have a one node machine config pool When we think about things like doing updates in a cluster It does those updates on a machine, right? It learns things or it inherits things like how many nodes can be down at a time from the machine config pool Great, so it when it needs to do something like apply updates It says well machine config pool how many nodes can be down at a time. Oh only one. Okay Well, I will sequentially go through and reboot, you know, apply updates and reboot the nodes one at a time Or you could say maybe it's 25 so in a pool with You know 10 nodes I can take down up to 25 of them at any one time to apply updates or do whatever machine config needs to do So now with a single node machine config pool you effectively have to disable those so now The cluster can get into the state where You're relying fully on things like pod disruption budget to control how many nodes can be down at any point in time So it's definitely A management overhead Issue as well as introduces a lot of other kind of unforeseen risk that's inside of there And that's just the one that comes to off the top of my head so yes It seems like machine config is a great option for applying static ip It's really really not The other option which you can kind of squint at And maybe be okay would be to do something like writes a bash script. I'll I'll say bash And have it Be so write a bash script that detects which nodes it in which node it's on and then applies the static ip configuration that way Right, so then you have one machine config For all of the nodes in the machine config pool and that script just says hey, I'm on node 12 I need to give it this ip address I don't know how Should work, you know should be fine You know assuming everything goes according to plan right nothing goes haywire or anything like that That is an option. So Maybe consider that with the others, but again, it can you know doing updates or changes, you know Hey, I added a new node to the pool. I went from 12 nodes to 14 nodes When I have to recreate that script when you recreate that script and update the machine config It's going to trigger a reconfigure of all the nodes in the machine config pool So they'll all go through get the new file and then have to go through a reboot and all that other stuff So still not a panacea if you will Um, do we have any questions chris? I haven't seen no there's another court on my mind. Uh, come in Okay So we've only got just a few minutes left. Maybe six or seven minutes left So I want to make sure if you have any questions, please don't hesitate to ask those Um, just as with last week, uh, I I've got I don't know 20 or 30 different links here that I'll put all of these into the blog post Look for that on monday morning Or excuse me friday morning friday morning. Um, so you can find all of the stuff that we Would talk about you can of course always reach out at any point in time streaming or not through social media So on twitter, I am practical andrew So if you saw me chatting inside of the twitch chat or any of the places where it gets rebroadcast Um, same username practical andrew, um, you know all one word on twitter You can also reach out to me directly through redhead email andrew.sullivan at redhead.com You can reach out anytime happy to field those questions make the connections and get those questions answered for you Um, so I think the other thing that I want to talk about here While we are, um, you know During the stream while I have your attention Is something that has Come up that is kind of floating in the back of my head is one of those top of mind things But it hasn't quite reached that point yet, but I think I am going to talk about it and that is The concept of and and I'm combining a few things here and that is How do I do resource management inside of the cluster? And ultimately and I think most of us probably know You know clusters aren't static Especially if we're thinking like ipi and we're doing machine sets and auto scaling and all this other stuff Yeah, you know the nodes are constantly changing but I also want to remind us that We can change the size of the nodes for example, and we can balance the resources So cpu memory storage, etc according to what the applications are actually using And additionally, you know the metric service gives us things like Well, the pods are making requests remember a quest is Effectively a reservation, but they're only using you know, maybe 10% of that So working with our applications peers to basically say hey, why are you requesting 10 gigabytes of ram? And you know only using one that's nine gigabytes of memory that I can't use for other applications in the cluster And maybe their answer is Well, you know, we do have a batch job that runs, you know twice a month that does require that much memory, you know Okay, well, let's see if we can find A better You know a better way of meeting those needs And well, I know the you know the open shift metric service has a lot of data a lot of information in it um our partners, you know, so partners like datadog and Turbonomic and Gosh, there's like fire so many. Yeah On and on and on of, you know in all of these metrics in the metric space They take the same information, right? Sometimes they take it directly from metrics. Sometimes they collect their own And they have tools inside of their portfolio to be able to automatically take action as well I recently saw the Turbonomic folks give a demo where They were able to predictively move pods to free up resources for things that were happening and it was pretty cool stuff. So Definitely always be cognizant of, you know, real resource utilization versus reserved resource utilization How can I best optimize the cluster? You know, to take advantage of those resources if you're deploying in a hyperscaler, right? You're paying for those instances If they're, you know deployed whether or not you're actually using those resources So making sure that you're actually pushing that cpu utilization to real usage of, you know, 80 90% Is a cost-saving measure Instead of, you know Years ago They were I remember having conversations around our virtualization deployment You know was only at like 30 percent utilization or something like that And it's one of those like the higher you drive that utilization the better value you get out of it Which is definitely true with the hyperscalers So we got a question From Don Weeks, are there instructions for installing open source calico as the sdn and open shift? Can that only be done by upi install? So the second part I believe is yes. Yeah, so so the sdn can only be deployed at install time So it doesn't matter if it's ipi or upi or even non-integrated It'll absolutely work and you can if you haven't seen calico's docs Um, you can look at their docs here about how to deploy inside of open shift More or less the the principle is the same regardless of the installation method So you see this create manifest and then you drop all of their manifests into the correct place And then continue with the rest of the install. Uh, they don't make any differentiation here as to whether it's ipi or upi Yeah Yep, so, uh, how often do we need to update the rel? core oba file on vmware so My suggestion would be at least once every major or y release So if yeah, so when you go 4.4, yeah, exactly so 4.6 to 4.7 update the oba Within the same release Uh, it will and even with it between the major releases so long as it's an earlier version It should update coro s to the right version as a part of the installation process Right, so essentially, you know, it'll clone that ova for the new worker node It'll reach out to the control plane to the machine config operator And it'll say give me my ignition and part of that is hey pull down this new rpmos tree image and switch over so That should work You might be able to slightly speed up the node provisioning node joining process if you're on the same release that it wants um, but yeah So Question those manif like I have no idea on the answer to this one those manifests are for enterprise calico, right? And I would I don't know um I don't think so. This is the project calico website. Yeah, I mean, this isn't the project site. Yeah. Yeah, exactly. So Yeah, it's talking about tigera status Tigera operator. See that. Yeah, here's the tigera operator. Market place. I mean I have always assumed that this is the open source version. Yeah, not the enterprise version That'd be and I've deployed this a couple of times. So Yeah, I would be interested. Yeah, you haven't swept your credit card to deploy it. Have you? Um, no, I'm aware of right Um Yeah, that's interesting. Yeah, so, um, don't please feel free to reach out. Um, and yeah, I'll I'll also poke some of our calico folks. Yeah, go ahead. You poke So, yeah, please feel free to reach out andrew.sullivan at redhat.com. Um, yeah, okay. So That's really all I think we've got time to get through today. Um, as I said, I have a whole list of Um, of stuff that I will include in the blog post. Um, we can talk about that during the next show as well Um, so which brings me to my next point, which is we won't be back for three weeks Right. There's stuff happening. Yes. Uh, so next week is red hat summit If you aren't registered for red hat summit, I definitely encourage you to do that. It's free It is free So yeah, absolutely registered for red hat summits. Uh, it is a two part event So the first one is next week. The second part will be in june. Um, you'll get all the Announcements and alerts and emails and everything about both of those events Um, so definitely encourage you to do that If you haven't already the following week. So the first week of may is kubcon So we will be off air. Um, so the this show will be off air for both of those weeks But chris will definitely be on air doing Tons and tons and tons of streaming with all the stuff that we got going on there a lot of office hours a lot of Interesting fun things are going to be happening. So yeah, please tune in So we will be back. I believe it is may 12th The currently scheduled topic will be red hat enterprise linux coro s or as we lovingly know it our costs With mark russell who's a product product manager for coro s But yeah, we can absolutely bleed over some of these topics again Please don't hesitate to reach out if you have any questions social media practical andrew at twitter Or andrew dot sullivan at redhead.com and thank you so much for joining us today Thank you all have a good one and coming up next we'll be talking about Uh three critical areas of container resource management with the folks at densify. So please stay tuned. I'll be queuing that up now


I ordered a few clothes. E.B. headband, so that's one new t-shirt, a long shirt, a cropped hoodie, a fun new bra, I know what this one is and I must afford this one. It feels so soft, full of hair, a bit of tops, they look like this. They're all the same, they're just as simple as a jumper, another jumper. I always buy jumpers because I was too big, so this is a large, I'm a medium. I've got another lovely thing that I look like at all. It's a funny hoodie, another t-shirt, so you can read it. I like this t-shirt, so it's just a black t-shirt. And then on the back it has, it's a froggy crochet top. I need to take it, some new jeans and a jumper. It's a jumper to put a vest top under. I've got a cat ear hoodie, lovely hoodies, especially ones I use. So if you guys want me to do a try on haul, hello. Off the road though, yeah. Okay, yeah I'll come down now. As I was saying before, my food got delivered. If you want me to do a try on haul, let me know in the comments down below. Thanks for watching and I will see you guys in my next video. Peace.


Fifteen years ago, a Haitian priest asked me to come to Haiti to help him build a bank the poor could call their own. I asked him at that time, well what kind of services are you planning to provide? He said all the normal things, savings, loans, money transfers.  You have to accompany her as she struggles to make her way out of poverty. I realized later that what that meant is we were going to have to teach her how to read and write, we were going to have to give her business skills, and then before we knew it we were into health care too. So gradually we built this staircase out of poverty. Our staircase out of poverty has five different steps to it. At every step, there's a different mix of financial, educational and health care services that will allow the woman to make the next step. Our job is to get them on this ladder and be there for them when they fall off. We're talking about very, very, very poor people. But over the years, as our strategy expanded, I came to two realizations. The first one is that accumulation of assets doesn't mean much of anything unless you can protect those assets. I have seen women who made their way to the top and then got knocked down because they had to pay for a funeral or because there was a bad hurricane season. In fact, I've seen the same woman, the same woman fall three times in the last six years. I simply can't continue to do my business this way. I can't encourage her to save and reinvest her profits back into her business and use credit wisely and then watch her fail because of a hurricane. And the second one was that not everyone is a micro-entrepreneur. Many of these people need jobs. They can make their way out of poverty, but not by opening their own businesses. And job creation, most experts would agree, is really best done by the small and medium enterprise sector. But in Haiti, where there's 70% unemployment, almost everybody has to be an entrepreneur. Clearly, we have to find a way to create jobs for these people. So here we are dealing with the very poorest people, trying to eliminate extreme poverty. And yet we're having to put our attention on job creation by assisting the small and medium enterprises instead of just focusing on the poorest people. That's a tricky thing when you're an institution with a mission to focus on the very poorest. People see that as you're drifting away from your mission. You're going upmarket. But in order to help them, we also have to be able to create jobs for them. The point is that eliminating extreme poverty is not rocket science. We don't have any reason that we have to have the degree of poverty in that country that we do have today.


You can download QGIS from QGIS.org. Here you see in the banner that the newest version is 3.14 pi, which has just been released. However, if we go to the upper left corner of the site, we can see two version numbers. So the latest is 3.14.0, but there's another one, 3.10.7, which is called LTR, the long-term release. And in the upper right, we see the times and the dates of the upcoming releases. Now we need to download and install the LTR version, because that is a long-term release, which means that it is maintained for a longer time than bug fixes, in this case 3.10.7. For the course you can download the latest LTR version in the 3.10 series. Newer versions have new features, and it's fun to try those features. However, new bugs can also be introduced. Therefore, it's not recommended to use the newest versions for operational use or for courses. So now we click download now to go to the download page. For this course, we assume that you use the Windows operating system. There are different installers here. There is the OSGO 4W installer, which is for more advanced use. And we have here the standalone installers, which have the 3.14, the newest version, and the long-term release. And we are going to install here the long-term release. Now you have to determine if you have a 64-bit or a 32-bit computer, because it has different installers. To find out, go to your file explorer and click write on this PC and click properties. Now you can see what kind of operating system you have, and in my case I have the 64-bit. You could also see that I have 16 gigabytes of RAM, which is recommended for GIS. So now you can install the 64-bit installer and simply go through the wizard and complete it. After installing, you can search for it by doing a search in your operating system. And each time we start QGIS in this course, we use the QGIS desktop 3.10. Whatever version you have after the dot. And then the splash screen will show and QGIS is starting. Once QGIS is started, we can see this screen. It has some news messages and the project templates. This place will be later occupied by the map canvas where the layers are visualized. We also see a layers panel and a browser panel. In the layers panel, our GIS layers will be listed. In the browser panel, we can navigate to the files on our file system and it will recognize GIS formats. We can also link there to online layers and to databases. On top, we see the main menu with different options. And we see a toolbar with shortcut buttons to the different operations that we can do. And then at the bottom, we have information about the projection and the scale. You can easily move these panels. Here, for example, I drag the layers panel over the browser panel so they are visible in different tabs. Which saves a lot of space, especially if you have a small laptop screen. By clicking the blank project button, I can clear the map canvas so our layers will be visible there. Let's explore the main menu. In the project menu, you can load and save project files. You can also access your print layouts. In the view menu, you can activate or deactivate panels and toolbars. If you might have accidentally closed a panel, you can find it back easily there. In the layers menu, you can create or add a layer. And in the settings menu, you can change the settings of QGIS. For example, the language. By default, QGIS detects the language of your computer. In this course, we use English. After changing to another language, you need to restart QGIS. We frequently will use the plugins menu to look for plugins. Core plugins are already installed. For other plugins, you need an internet connection. In the vector menu, you find the vector tools. In the raster, the raster tools. There's database tools. There's web tools. And there is the processing toolbox that we will use. In the help menu, you can find assistance and check your QGIS version. And when the new releases are coming out. Let's further explore the browser panel. The browser panel gives access to your files. You can create favorites, for example. You can browse your disk and you can add a folder to your favorites. When you expand the folder and the subfolders, it will detect the file type. So it will indicate if it's a raster or a vector or a database. In the toolbar, you can find a lot of icons for saving, for zooming in and out, for zooming to the full extent. To create a new map view and other options such as identify features, open new data sources and all the editing that you can do with vectors. We're going to add a first layer of the whole world by typing world in the coordinate field. And then I can show how to use the zoom button. And you can use the scroll button to more easily scroll in and out. Here you can change the projection of the project. You can choose it from a list or you can type their keywords or EPSG codes that you will learn more about. This is the context menu of a layer. You can zoom to the layer extent or you can open the attribute table. The attribute table shows the attributes in the layer and it has a table view and it also has a form view and you can toggle between the two buttons. So we most often need the table view to do the editing. This was just a quick overview to get you started. In the course you will learn all the features in detail.