Hello everybody and welcome to the Bruce Wagner Show. Today we have a most amazing guest. She is a best-selling author, speaker, lecturer, entrepreneur, so many things. She's even produced games, all sorts of things. So her name is Sharon Lecter. Welcome Sharon. Thank you. Delighted to be with you Bruce. Yes.  And you've been involved in many other books and projects since then. So how many books have you published? Seventeen adult books. Before Rich Dad I had actually started and built a company, the talking book industry for kids, the books with the sound strips down the side. Before Rich Dad Poor Dad? Yes. That was back in the late eighties, the inventor of the talking book and I were friends and I joined his company and helped him grow that to the international powerhouse. It was a time when bookstores were very quiet places with little ladies behind.  Yes. We've disrupted bookstores. Boy, now bookstores are disappearing. It's becoming e-books and things. It's kind of sad. Whether they're talking more, I guess that's a good thing. They're interactive and multimedia and they're on iPads. Yes. The whole world's changing. They're closing Barnes and Noble at Columbus Circle. It's already closed, actually. It's kind of a sad thing to see. The record stores and the bookstores are next. Who knows what's... I think movie theaters are going to become some other kind of virtual reality centers and stuff. So you've always had an affinity for education and especially kids education. Absolutely. And part of it is because my own children and my grandchildren now. I've had the blessing to be able to combine my passion as a parent with my profession in publishing. So with talking books and then we moved to Arizona back in 91 and in 92, my oldest son went off to college and he got into credit card debt. And I was pretty upset at him but also at myself because I thought I had taught him about money. So it was in 92 when I really dedicated my professional career to financial education and financial literacy for young people, for families, and really focusing on entrepreneurship and helping people get their companies on the right path. Okay. And you've written so many different books in the Rich Dad Pordead series and then even after that. How many books after that? Well, we did 10... I've done 15 books in the Rich Dad series. So we started with Rich Dad Pordead which actually was written as a brochure for the board game. You may not know that. The book was written as a brochure. Yeah, we never expected it. We never expected it to take off and become a franchise. It's a huge thing, right? And we thought our brand was going to be cash flow because of the game and our company name was cash flow technologies. But the world told us, no, no, no, your brand is Rich Dad. So we started and literally 15 books later. I left the organization in 2007 and that's when the Napoleon Hill Foundation reached out to me and said, we need your help because with the downturn in the market at the end of 2007, the words of Napoleon Hill and Think and Grow Rich were really instrumental in helping us get out of the Great Depression so many years ago. Many successful companies today were started during the Great Depression. And so they said, we really need to get this more front and center. And that's when they came to me and asked me to step into a project called Three Feet from Gold, which we released in 2009. Okay, this is Three Feet from Gold. Yeah, that's the first one. And it really takes the principles of Think and Grow Rich of perseverance and staying the course. And we talked to 35 of today's top leaders about not just their success stories, but about how they got through the difficult times. You know, we've all we've all been in those valleys and they persevered and were able to keep going. There was last few, few steps to turn potential failure into great success because you were telling me earlier in the in the green room that the the title tell us where the title comes from. Yes, Three Feet from Gold actually comes from a story in the first chapter of Think and Grow Rich. And it's about a young man named R. U. Darby who went out West with his uncle to discover gold. And they were all excited. They got their mind. They bought their equipment. They dug and they got some top quality gold. So excited they went and they got more money, more equipment, kept digging all of a sudden they ran out. And they kept digging more dirt, more dirt, no gold. And so they got frustrated through their hands up and went back. He sold their mine and the equipment to the town junk man. Now the junk man had been living there 20 years waiting for his opportunity studying gold. He knew all the right people. He knew the gold engineers. He goes out to the to the mine. And the engineer says, really quite simple. R. U. Darby and his uncle just went right through the the vein of gold. You go back to where they start and go back three other way go three feet from that and you'll discover it. And so the junk man discovered the largest discovery of gold of his age of that time. R. U. Darby was back east and was he upset? Yes. Yes. But he also said I will never allow this to happen to me again. He went on to build a huge insurance company. And he started teaching the principle of perseverance and never giving up. And that element is so so important today with our economy. So many of us have seen the ups and downs and that roller coaster continues today. And so that was the essence of working with the foundation to bring these words back out to the forefront today. Because there are many people that are under 4045 that don't had never heard of Napoleon Hill. Right. Yeah, and it's timeless ideas. The the perseverance and there's lessons in that for everybody. The Darby went back and didn't have the perseverance and also for the the town junk man because he just waited perseverantly for his for the right time for the opportunity. Boom. The opportunity to present it itself. And he had the right team. He had the right mastermind. Yeah. You know so many of us we've been taught through school that we have to do everything on our own. You have to work alone. Yes. And entrepreneurs become very lonely. Right. And if you want to be successful, you need to expand your horizon, have a mastermind, have advisors who can help speed your way to success. Absolutely. And have the right team around you. This person may be strong where you're weak. And that's what you want for nobody makes it alone. Everything is a team sport. Right. Absolutely. You have to have your experts. And really, one of the one of the most important things is to know what you don't know, right? To know what you're, you know, what you're, because everybody has their area of brilliance, but very few people have more than one. And most people feel like they can't ask for help because it's a sign of weakness. And they're afraid of criticism, which brings me to the latest book because right as we're about to release three feet from gold, the foundation called me and said, Sharon, we've just discovered a manuscript that Napoleon Hill wrote the year after publishing Thinking Grow Rich. He intended it to be the sequel and it was hidden away for 73 years. That's the, that's this, your new one, Outwitting the Devil, right? And this is, so this is a transcript that was written as the sequel by Napoleon Hill to Thinking Grow Rich and it was buried away by his family? Yes. Why is that? His wife was working for the Methodist Church at the time. And the, and the title Outwitting the Devil in the book itself, the last half of it is an interrogation of the devil and Napoleon Hill says you can decide, you the reader can decide if you think I was talking to the real devil or an imaginary devil. Yeah. Doesn't matter if you get value from what the content says, but his wife was terrified. And in fact, in the book when you read it, there's a section, of course, this was typed on a manual typewriter during the time of Hitler and Mussolini, but in this Q and A with the devil, he says, you know, Mr. Devil, aren't you afraid? I'm going to give away all your secrets and you're going to lose control of all these souls. And the devil says, and he wrote this in 1938. No, I'm not worried at all because you're going to be too afraid to publish a manuscript. It's pretty wild when you say, in essence, that's exactly what happened. It was put away in a closet and it didn't come out, but it's, there was no accident here because what's happened today, so many people are crippled in fear and the essence of the book is that the devil uses fear. Fear paralyzes us. It makes us sit on the sideline. And so instead of having fear paralyze us, we need to allow it to motivate us. And it talks about the fear of poverty, which of course is very prevalent today, but also the fear of criticism. And what's happened today, so many people are afraid to look bad, just look silly, afraid to ask for help. Right. And yet, Napoleon Hill himself said there's, you know, the mastermind principle comes from Napoleon Hill. And so you need to ask for help. Find those advisors that can help speed you on your way. Find those people that can, you add to your team that have strengths where your weaknesses are. Right. And that's truly the pathway to success is business is a team sport. Absolutely. And I've heard, I've heard it said, people have said that there are only two real emotions and that's love and fear. And they're, they're, they're sort of opposites or mutually exclusive. You can't be having both at the same time in the same matter anyway. So it's interesting. So yeah, you can, you can take it literally, or you can take it figuratively, or that the devil is the fear that we have, however you want to perceive it. But it's, I noticed that you have testimonials here from Harvey McKay, who wrote Swim with the Sharks without being eaten alive, and Steve Forbes from Editor-in-Chief of Forbes magazine, of course, and Michael Beckwith, who is an amazing, he was in The Secret and is an amazing spiritual leader. So you've got, it sounds like you've got fans from all different sectors, right? Business, self-help, finance, and spiritual community as well. And it's so important because today we're all being battled by fear from every direction, whether it be financial or spiritual or terrorism. You know, the fear is pervasive. And what this book does, it gives you seven steps to try and get rid of the fear in your life and ways to outwith the devil.  And when we're fearful, we kind of close down. And this gives you that opportunity to give you that hand up to get you out of that fearful situation and things that you can do to really set a goal for yourself and achieve and get past the fear in your life. Great. And so, so exciting. I haven't read it, but I can't wait to read it. So this seems like a good time. We need to take a break really quick to thank our sponsors. Obviously, we wouldn't be here without our sponsors. We are most grateful to Mt. Gox. Mt. Gox, and this is something that I was explaining to you earlier, Sharon, about Bitcoin, which is a whole interesting phenomenon. It's electronic digital currency. It's kind of, they're calling it the money of the future. They're calling it the most exciting technology since the invention of the internet itself. Well, there's a company called Mt. Gox, m-t-g-o-x.com, which happens to be the number one online exchange site. So it's a really convenient place and way to buy Bitcoin or sell Bitcoin online without leaving, you know, comfort of your own home. So if you own a, you know, a diner in Brooklyn and you accept Bitcoin, you need to convert that into dollars for your produce order for the week. You will need to use an online exchange like Mt. Gox to convert your Bitcoins to US dollars, for example. If somebody wants to invest in Bitcoin, they would be doing the opposite. They'll be buying Bitcoin for dollars. So this is kind of like a, almost like a matching dating service. It's a market online. It's automated 24-7 and actually Mt. Gox is the 90% market share at least. So they are sort of the standard. So when we talk about the value of Bitcoin, we just look at the Mt. Gox last price. So Mt. Gox is a sponsor and phenomenal company and we thank them very, very much for sponsoring our show. So check out Mt. Gox, m-t-g-o-x.com and you set up an account. It's free and brilliant service. And the Thank You Economy, which is a book by Gary Vaynerchuk. And it is a book actually about what we're talking about. It's about businesses large and small actually using social media. He talks about how to, he's a best-selling author, New York Times best-selling author as well. This is his new book. And he talks about how to use social media in this new age of the internet effectively. Because I always say the vast, 99% of people doing, you know, social media marketing are doing it wrong. Basically it's just spam and it's annoying to people. But he talks about how to really bring an old fashion feeling of direct customer service and hand-holding to your customer. Even if you're a huge corporation, you can actually engage customers one by one and give them personalized customer service using these tools of social media. And that's what it's about. Like he calls it Thank You Economy, which is really about thanking the customer for being a customer. So it's Thank You Economy Book.com. So check that out. Thank You Economy and Gary Vaynerchuk. Okay, so I have a question now. The three feet from gold, which is Thinking Grow Rich. Now is that an actual, is it actually Thinking Grow Rich reprinted or is it commentary about Thinking Grow Rich? It's actually a whole new book and it's a parable in itself where the Thinking Grow Rich is the brand name of a new series of books. So for instance I'm going to do a Thinking Grow Rich for women by women talking about women and their thought processes and how they follow the principles of Thinking Grow Rich. I'm also going to be doing a Thinking Grow Rich for children. So that's actually the brand name of a new series of books. And three feet from gold is actually a unique written novel basically that we did talking to many of today's thought leaders. And we set it in a parable format so that you follow a young man through who has gotten from the entitlement age and he's kind of a, you know, he likes to spend money that he doesn't have and he goes through this journey of having a mentor that helps him get back on the right track. And the mentor introduces him to some of his friends. Much like what Hill did when Andrew Carnegie introduced him to all the rich men of his time back in the early 1900s. And that's really where the power of Thinking Grow Rich comes from because it wasn't one man's philosophy. It's not Napoleon Hill's philosophy. He interviewed hundreds and thousands of people that were successful and thousands of people that considered themselves failures. And so it was a life study. He studied for over 20 years. He was really a researcher. Yes. And so Thinking Grow Rich is actually a synthesis of success of all of these people. That's why it's so powerful. Yeah. Timeless. Just as out waiting the devil is the same synthesis of what helped people back. Because that's the power. All of both of these books were written as a result of his lifetime of study. Okay. And the Three Feet from Gold, so it's an original novel. Right. It's based on the principles of Thinking Grow Rich. Right. But it's a completely new original novel. Right. And what we did was we took the principles of perseverance and never giving up. And we went out and talked to many of today's leaders. Okay. We talked to the founder, you know, the greater Velcro, FedEx, Chick-fil-A restaurants. So you might not recognize the names of the people we talked to, but you would recognize their brands in the world today. Sure. About how they are still utilizing these principles in today's market. Wow. So we wanted to show the relevance of the of the information to what's happening in the world today. How did you find those people? Like were they, I mean, did you already know that they were using these principles or did they contact you or how did that work? In a myriad of ways. We looked to those folks that communicate with the foundation who have created success and say, I love Napoleon Hill. Oh, they're already there. So, you know, there was already knew that they were, there was already a relationship. Yeah. And we wanted to make sure the people we highlighted really exhibited the elements that we want to people to see as a result of Think and Grow Rich. And so some of them we actually reached out to to say, you know, did you have influence from Think and Grow Rich? So, right. Various different ways. But outwitting the devil is is actually the original transcript that I mean, what do you call it? The manuscript was the original by Napoleon Hill to Think and Grow Rich. And you annotated it? Yes. What I did was I didn't change the manuscript. I did do some precision editing because he can sometimes be a little wordy. But it's his actual writing and then I add sections and it's in a different font. So you can so you can ignore my comments if you choose. If you're a purist you can just read his his manuscript and ignore my comments or as you go through the book you can read. And what I did was I would compare what was happening in 1938 when he wrote it to what's happening today because there are so many similarities. Yeah. We had high unemployment back then, high unemployment today. There we were at war back then or certainly at war today. And the economy of course was in the toilet then and the economy is certainly there today. So there's so many parallels and the thought process of moving from an entitlement mentality to an empowerment mentality was a very big issue for him. He also talked about the need to not have dictatorial education but to have experiential education instead of pounding facts into our kids through memorization. Now this was 1938. I was going to say so ahead of his time. It could be 2038. So yes, so relevant today. Yeah. He also at the time talked about the bad side of cigarettes. Now this was at a time before we knew cigarettes were bad for you 1938. We were so giving them away free to our troops everybody's and he talked about the devil says that he uses cigarettes to derail young people at a young age to get them addicted to nicotine so that they would drift and that was one of the big concepts in the book that I love. He talks about drifters versus non-drifters. And a drifter is someone who's easily swayed. They go with the flow. They don't have a mind of their own. They don't have an ultimate goal. Like sheep we kind of think of them as sheep that just easily manipulated by marketing and trends. Right and the devil loves them because he can mess their lives up and then he compares the non-drifter and the non-drifter is somebody who knows where they want to go. They have their goal and they yes we have setbacks along the way but when you know where you want to go you can recalibrate your path.  And so it's that's part of it and we think about different times in our life when maybe we were kind of going with the flow. Oh yeah. All of us. Last Tuesday now. Yes exactly and there's one section of the book about procrastination and it's like darn busted because he said you know procrastination is drifting and I said yeah but that's my excuse. I said I always work better under pressure but yes oh no. You say that too. Yeah laziness laziness plus indifference equals procrastination. I guess if the shoe fits you know. Yes we're all guilty. And the the term the devil I can see where that's kind of a loaded term but it doesn't matter whether you like personally I don't believe in the devil but you can still look at it even metaphysically it's still the same because if you like metaphysically I would think of the devil as being your your ego your you know fear. That inner voice. Yeah that exactly even so however you look at it whether you look at it literally or or metaphysically or whatever it's still equally applicable. Well when I tell people I say you know we've all seen the cartoons with the angel on one shoulder and the devil on the other.  Right. You know if you think about going to a funeral you're down you feel you know how depressed the room is versus going to a party where they're dancing and that's it's so important because so many of us the cocktail conversation is about bad health there is the or about the economy or somebody losing their house or losing their job you know it's like I came out for this exactly versus what's positive. That's so so important people don't pay enough attention to that the people that you hang around the the energy that people are giving off you to be really aware of that and it just it's going to either bring you down or bring you up and it influences your your thoughts for the next 24 hours 48 hours or whatever of what you do and it's so important to keep a to keep that positive energy and stay away from that negative stuff that's just drawing you down people complaining no matter what it is I mean there there are obviously they're legitimate complaints and things like that there's a time and place you take it you process it you file it away it's okay now I'm going to do what I have to do about that and file it in the drawer you move on and stay positive right and you turn those negatives into opportunities you see a way you know the most successful businesses do one of two things they solve a problem or serve a need and so we've got a lot of problems today we've got a lot of needs I mean last night I was speaking to a group of under 30 CEOs and one of the most common questions I had was what happens when your family isn't supportive and that's that's more often than not the case yeah because your family knows you so well and you start doing something different in your life you start wanting to go out instead of being going to the corporate job you want to be an entrepreneur the first thing that happens is mom and dad goes oh no no you don't your family says oh that's crazy you know you shouldn't do that and so you instantly feel pulled back you know the analogy of there's a one crab in a box crawls right out two crabs in the box neither one gets out and so people are naturally resistance to change and you can't change your family but you have to recognize that that's their attitude and I'm not going to allow myself to be sucked in and so you limit the time you spend with your family when you're in this build-up mode and you increase the time that you spend with your friends that are on the same path and you go to the networking groups and you find that mastermind group that supports you and where you want to go so that you're adding that positive energy and encouragement because once you start seeing some success all of a sudden your family is going to be yeah they're right more friends than ever I don't know how that would work exactly it's true there's a saying I don't a paraphrase I get it completely wrong but something about you can't be a prophet in your own village and that's kind of what it is the people that are closest to you are the most critical and one of the often and one of the things that I always think of it as a curse word it's like the worst thing you someone can say to me is these words be realistic because what they're really saying it's a polite way of saying oh you're not good enough you're never going to succeed and it's just it's it's that pulling you down you're a dreamer yeah that's good realistic you're just little brucee from the block or whatever you know I know you from way back and you you've never you've never been a super success before so you're not going to be now you know that kind of thinking that be realistic and it's their own fear that's speaking it's their own fear of criticism their own fear of change their own fear of failure they're exactly they are projecting it on you and you have a choice nope receiving that exactly or rejecting that talk to the hand right yes yes this yeah yeah yeah nope brush it off okay bye bye we'll talk later once I'm successful that's right exactly prove them wrong some people actually take that energy and and fuel this like I want to like I'll show you I call it the I'll show you feeling motivation yeah and that's what fear can either paralyze you or it can motivate you yeah and when we talk about passion one of the things in three three feet from goal we came up with a personal success equation which was common among all the people that we talked to and that's taking your passion and when I talk to people about passion we've all heard do what you love love what you do right well passion can also come from anger and so for instance my life's work is because I'm angry that we don't teach money in school we don't teach people about entrepreneurship so my passion is from anger and so you combine your passion with your talent so my talent I'm a CPA and I have a background in publishing and so I combine my passion with my talent but that's not enough you have to find that right association and that's the people that you surround yourself with or you collaborate with another company that can help speed you to market right and so that association is so important and then of course taking the right action you know we've all seen the movie the secret in the book and it's great talks about the law pop you know positive attraction well that actually was originated by Napoleon Hill in 1919 he really originated the law of attraction but he also said you have to do something you have to take action right you know we've all been given a great mind and two arms and two legs we're supposed to actually do something go the as he said go the extra mile but then and we were almost ready to go to print with that formula power you know that your passion plus your talent times the right association taking the right action and why I said no there's still something missing and the piece was faith and that's what helps get you through the negativity faith in yourself right and faith in what you're doing and the faith that you are going to succeed right and that's a common formula that every one of the people that we highlighted in three feet from gold and and the basic underpinning formula in all of Napoleon Hill's work is the fact that you have the ability to be successful it doesn't take money to make money it takes the passion and the talent the right association taking the right action and having faith in yourself right and if you have a if there's a spiritual component to it one of the things that to me I think of the easiest way to have that faith you have to have a couple things when you have to feel worthy of success because if you feel that you don't deserve the success then you will never get it no matter what that's just a deal breaker you have to feel like you're worthy that you have to love yourself effective first of all love yourself enough to feel that you are worthy of it right and then you also one of the things that I don't I don't know where I got this idea but I think about is if you if you feel like what you're doing it's kind of sort of has a moral or ethical or spiritual aspect to it if you feel like what you're doing is good basically it's ordained by a higher power by God or whatever that what you're doing is actually good then you I feel like I can't fail if what I'm doing is actually good is really helping people like our mission statement at only one TV is to help as many people as possible and we mean masses of people billions of people all over the planet in the most profound ways possible that's our mission corporate mission statement right and so I really believe that if I really feel that way then spiritually I feel I can't fail I can't fail it's impossible so if you think that way that's that's your faith you're talking about yes and it's mission driven as opposed to profit driven right and that's where so much of it is you know when you see successful companies you know are they they can be successful for short term if they're profit driven right but that's called what I call are you selling or are you serving and the issue is if you set just sell as a transaction right I can I can make a quick I can convince you to buy my book as a transaction right but I want to serve you so that I want a relationship right so you instead of having a transaction with your customers create a relationship with your customers and the money will come yeah but if you're if you're if your vision and your mission is aligned in service then you have a long-term success you may continue to come if you have a relationship right if you make a fast buck then yeah you can you can cheat people you can just do anything unethical and make you can have a lot of sick financial success but very short lived right wow so how have have you I know that you have developed lots of products because I remember the cash flow game and then cash flow for kids we actually and I actually used to have a this thing we started at unity called prosperity groups that we did this we created this whole thing and this was part of our little curriculum in our groups that we did was playing we had a game night where we play cash flow and all that now I understand you have some new products also that are targeted for teenagers yes I'm so excited about this and because not so much because we won one awards which I'll share with you but because the teens love them I left the Richard organization in 2007 and my passion was to create products that really would give young people the chance to become their own boss and become masters of money instead of slaves to money and it was really almost a download of the stuff that was in my brain and we started with an entrepreneurial kit and it's this biz kit we just within the last two weeks learn that we are we received the 2011 mom's gold choice award for this product and so it's a real it makes me very happy but what makes me even happier is when a young person actually uses it and generates and makes a business for themselves and it's really a fun yes a really fun four color product and it shows the various aspects of building a business and then at the end of each section they fill out a little piece of this one little question they answer the questions and when they're done going through the process they you pull out those pages and they have their own business plan and so even adults get a little intimidated when you say oh me too I've never I've never written a business plan I think so I've had adults use this product yeah I've had that actually I'll pretend I'm a kid yeah and so it walks you through the process and it allows them to become powerful because when a young person knows that they can make their own money that's the gift of a lifetime yeah that's exciting instead of thinking you have to get a job and be taken care of by an employer or taken care of by the government you learn that you can take care of yourself create a job yeah exactly and the self-esteem goes up yeah and the confidence goes up that little you're not worthy goes and starts going away because they realize that they can take care of themselves and that's what's so important but when I did this product I also started thinking that you know young people like to they like games of course they like the internet but when I started I actually created this product which is called Thrive Time Proteins and it's a board game and the sweet spot is ages 10 to 20 because it talks to young people about where they are at that stage of their life and it's done with humor and the beauty of it is we teach them that the choices they make will either drive them to success to thriving or dive them into debt we don't talk down to them it's a lot of fun a lot of humor in it but they they learn that it's just as important how they spend their time as how they spend their money and we've kept it very cost affordable it's $29 so my my goal is to make a difference not to make a profit and the kids that play it absolutely love it we won the same mom golds choice award on this product a year ago 2010 wow but the better award is when the kids play it and they just they they get engaged they like it the conversations they have around the table they go why aren't they teaching me this in school I need this information this relates to who I am and what I am and it talks about getting good grades so you can get a decent college scholarship so many people today are just heavy burdened with large school loans and school debt they they're getting out of college with $100,000 in school loans and no job and that's this going to cripple this generation unless we teach them how to be independent and again how to be a master of their money instead of a slave to their money how does the game handle that if you're it deals with student loans right is it is it detrimental in the game if you if you take student loans every every every decision you make you either get a positive thrive score or a negative thrive score and so in here there are certain elements of the fact that you qualify for a scholarship well that's a positive okay and then you may have an opportunity for your parents to help support you through school that's a positive and taking a school loan can be a positive if it's for the right reason and it can be a negative if it's for the wrong reason and so we deal with that we talk about your first car a car can be a negative if you go get the hot hot hot sports car it's a negative if your parents support you and you can get a decrease in your car insurance because you have good grades there's also cards that say you've done too many text messages extra hundred bucks well I think that's probably happened to a few kids this age and talks about one one card is your your boss realizes you've been giving away free food to your friends for a year he should fire you but instead he reduces your pay and makes you clean the toilets so again it's humorous there's you know and then the kids they have a great time talking about things that truly have happened in their lives there's one my personal favorite is you go to a great party at a friend's house their parents are out of town cops come you get a four hundred dollar ticket well I don't have to say on the card that there was underage drinking the kids fill all that in around the table but again it makes them experience things I often say we need to allow our teenagers to stub their toe while they're still at home yeah so they don't break their leg when they go out into the real world and so by doing this they have those experiences and they have the opportunity to think about the consequences without actually realizing the consequences and that's the beauty of what Thrive Time for Teens does it really allows them to start thinking about life choices in a safe environment with play money instead of real money it's a game yeah but the lessons they learn they can take easily take to the real world right have you had any interest from schools or educators and and using some of some of these games and books and things yeah I'm very excited we are going to be doing we're starting a plan in Arizona which we intend to take nationwide where we're doing a Thrive Time challenge throughout high schools in the state of Arizona starting in January and it will culminate in April which is financial literacy month so at the end of April we'll be doing a final competition of kids around the state for Thrive Time and the winner will get a scholarship so oh great so important and it's so important for the parents too this is a great way what happens is parents are afraid to talk to their kids about money because they're afraid their kids are going to realize that they themselves don't know everything you know that's the case they may have their own financial problems and so by by using the Thrive Time for teens game it's a safe tool it's a buffer parents got to learn too it takes the it takes the emotion out of it and so this the parent does learn along with the kid but it's also a safe way to talk to your young person about money to give them the tools they need without making it so personal right I mean blood and money sometimes is a very difficult thing to to mix because the emotional side of both money is emotional family is emotional put them together sometimes you have explosions yeah exactly and so I can imagine it's for parents it's like uh like like helping your kid with your with their homework and you have no idea what they're doing like um immigrant parents trying to help their kid with their homework and they don't even English isn't even their first language or whatever new math all those things that parents have experienced so this is an excellent tool to teach both at the same time and make it a game make it fun well part of the problem and what that I talk about this as well is I mean if you imagine the old science fair project where you can go in and you can see which projects mom and dad helped with the most right right and so mom and dad think aren't we great parents we helped our child get an A right well and you think the child is going to say my have great parents they help me well the child may actually be thinking they don't think I can do it on my own right exactly they don't learn self-confidence and then next time the kid's going to say what the you know what let mom and dad do it and so by helping you're actually disabling your child right I've done it with my own kids I've seen it I see it all over the place and in today's environment so many of us want to put our kids in bubbles so that they bounce they don't realize that they don't understand they never face failure treat them like they're helpless and then they become helpless they grow up helpless right couldn't have said it better exactly that is that's the issue we have so many with so many young people today I was thinking about you know giving giving too much to anybody when you know they're they're not going to use it wisely or whatever the case may be you giving a gift or money or something to people and depending on the circumstance can be I always thought of as enabling but you're right it's actually a better way to say it is disabling because it really is it's disabling them from doing what they really need to do themselves and you know really sometimes you just have to feel a pain or the desire or whatever to change your what your your behavior your choice your laziness or whatever it is and a lot of parents say well how do I get started and I said just you know that you can start really with a little goal and I talk about little wins when you know where your big goal is that's fine but set little goals and allow your child to achieve a little goal and celebrate because then all of a sudden the self-esteem goes up then they can set another little goal and achieve it and so it's like okay you start and all of a sudden you wake up and you're really close to your big goal you're so wise you know that I just saw this documentary on Ted I don't know if you've ever heard of Ted Talks Ted.com it's my it's like the only television I watch it's so cool I just saw this documentary on Ted.com about video games and that there's a science to these video game development companies that they use and that's exactly what they talk about they have a whole structure of this the idea of little goals and little achievements and whole all sorts of things that they've been able to actually match up to our the evolution of our brain of how we work and how we grow and how we learn really interesting stuff maybe you could explore getting some of these products made into video games that would be cool Yes, yes I would love that that would be super cool we're looking if you're a video game developer contact Sharon SharonLichter.com is your website, right? Yes, it is you can see it there now where's where's the best place that they can they can pick up these these books and these products the books are available on Amazon.com or Barnes and Noble.com you can also go we have a special website called OutwittingTheDevil.com Oh, okay and OutwittingTheDevil.com there's actually a secret chapter that was in the original manuscript that didn't make it into the final book and it's Napoleon Hill having a conversation with his invisible counselors his mastermind and these are great people from history that he brought together and it's a very it's like it's own little short story and so I made that available for free that's OutwittingTheDevil.com OutwittingTheDevil.com and you can also order the book through that website it takes you to Amazon and Barnes and Noble so great great well thank you so much Sharon it's been such a pleasure finally to meet you I'm just like you know one of your biggest fans it's my honor thank you so much and I appreciate the opportunity sure thank you see you guys tomorrow take care ciao


Yesterday, it was windy and rainy. Not the kind of day you want to make a trip to town from a place like this.  Oh, my legs are tired today. Oh, everything hurts. Oh, yeah, there's gonna be lots of groaning today. However, I went because there was an important package I wanted to get and I figured I'd make some good muscles. Here's the important package. Well, it's open now. Solar panels. There are 12 of these and they're lightweight. High efficiency. And I got them from these guys and I'm happy to plug them because they gave me a huge discount on solar panels I wouldn't normally be able to buy. So thanks to you guys.  So why are these particular solar panels so important for this project? On a boat, obviously, weight is an important issue. Now on the previous version of that boat, I had seven normal glass solar panels. They're a lot heavier. Let me just write down some numbers here and we'll go through it real quick. Alright, let's start with the glass solar panels. Originally, I had seven 40-pound, they may have been a little bit heavier than that, but roughly 40-pound solar panels. That's 280 pounds. They were 250 watts each, so 1,750 watts for 280 pounds. And then I also had, let's throw the batteries in here too. I had four lead acid batteries, 100 pounds each, 400 pounds. So that whole system was 680 pounds and I could get 1,750 watts out of it. Now the new solar panels, I have 12 of them, they're 6.6 pounds each. Look at that difference, 40 pounds to 6.6 pounds. Now these ones are a little bit lower power, but not by that much. So 12 of those is about 80 pounds, 79.2 or something. And 12 of those will give me 2,600 watts of power, much more than this, at a much lighter weight. And then the batteries, I'm going to be using some lithium iron phosphate batteries. I don't know exactly what I'm putting in, but it's going to be under 100 pounds probably. But I'll just say 100 pounds. So that brings this up to 180-pound power system versus a 680-pound power system. So I'm saving 500 pounds, that is potential cargo space, that's like three people or two big people. Or 500 pounds of stuff, like more than a tank of resin. And I'm getting more power. Now you might say, okay, if you want more cargo, just make a bigger boat. The thing is, a bigger boat's going to be even heavier and the heavier your boat is, the more of it's in the water. And pushing through the water is the difficult part of going. So the lighter you can make a boat, the faster you can go with the same amount of power. And now I've got more power and less weight. So this is going to be a huge improvement over what the boat used to be. I haven't used it in a few years, but like five years ago when I first made the boat, I had the glass solar panels and the lead-acid batteries. And it was pretty good then. I used it for a lot of things. I only retired it when I made a different cargo boat. But now I'm going to have a fast boat that can hold a pretty good amount of cargo. Like, I'll be able to put 500 pounds of stuff in it. And it will still only equal the weight it used to be. Oh, that's so good. And it's going to have more power. This is excellent. Another huge advantage to the lightweight is I want to mount the solar panels like this. This will actually be longer. So one, two, three, four solar panels, four here, four there for a total of 12. And then it will be hinged here and there so I can close them up. So when I'm not using them, all the solar panel surfaces are protected. And then when I want to use them, it's easy to pull them out. And also when they're closed, I'll have all this standing room on the side of the boat. So it just makes things a lot easier in a lot of different ways. So with the old solar panels, one of these things here would have weighed not counting the frame because the frame is non-negotiable. I have to build that either way. It'll be lightweight and fiberglass. But just counting the solar panels, the old ones, the glass ones, would be like 150 pounds moving up and down here. But with the new ones, it's more like what, 6.6 times 4, like 30-ish pounds, less than 30 pounds. Yeah, like 25 pounds. I mean, that's easy to move. So that's going to make opening and closing them so much easier. It's not going to be like some huge ordeal to get the boat ready, pull out these 150 pound things. Plus the frame would even have to be stronger, so it would be heavier too to accommodate all that extra weight. Yeah, this is going to be a piece of cake. I'll be able to do it with one hand, no problem. So another cool thing about these solar panels is the thickness. They're so thin, they'll be easy to fold over each other and they won't take up a lot of space. The glass ones are much thicker. It depends on which ones they are, but they're definitely thicker. These stick up, so I'll have to figure that out. But even with these sticking out, they're still much lower profile than the other glass ones. I also supposedly have a motor coming for this boat. I haven't gotten confirmation of delivery yet, so we'll see.  It's been raining so much the last year, although the year before that there was a drought. I guess I'd probably take too much rain versus a drought. Well, I'm not going to get anything done on it today. I'm too tired and sore to go out and do any major deeing stuff. I guess I could, but... I've got kind of a mess in my house. I think this is the perfect day to go clean it up. The thing is I don't really live in here.  And this place is basically storage. There's solar panels, here's bolts of cloth, stuff, parts. I've got batteries for the boat in here, stuff that's what's it called, inverter that's going to go in my house. Everything in this place is just storage of either boat parts or stuff that's going to go into my house once I finish my house. So it's just kind of a disaster, but I think I should clean it up a little. Well, there is another reason I need to clean up. I kind of put myself in a position where I have to. Because yesterday when I went to town and I went to the store, I got some fiberglass stuff so I can build these boat parts. I did not get myself a sheet of plywood, which is something I will need. I decided instead I'm going to steal one of the floor panels from inside this boat that's not being used anymore. So I need to clean up the area so I can take the floor panel out. Because like DeShana's whole side of the boat just isn't used at all anymore. There's some junk down there that the kids left and stuff. So I'm going to clean up and steal one of the floor panels and use that as the mold for the boat thing. And theoretically I could put it back once I don't trash it or anything. And I didn't get any video of yesterday because I drove there in the dark and then the whole day it was just rainy and windy. Well, let me give you a quick reenactment so you can feel like you were there. There's still a rainy and windy, but I got the solar panels. And I put the solar panels on the wheelbarrow and they're wrapped up in a tarp from so they're safe. I'm soaking wet, but I gotta go get them.


A new Azerbaijani checkpoint blocking access to an Armenian-held breakaway region is located right next to a base of Russian peacekeepers who are meant to be keeping the road open, footage filmed by Radio Free Europe slash Radio Liberty on April 23 showed. Azerbaijan said on Sunday it had installed a checkpoint on the Lakhinkorador, the only road connecting Armenia with Karaba, where 120,000 mostly Armenian inhabitants broke away from Azerbaijan in the 1990s. The corridor has been mostly closed since December, when Azerbaijani civilians identifying themselves as environmental activists blockaded the road. Russian peacekeepers who under the ceasefire agreement which ended a 2020 war for the territory are required to keep the corridor open have not intervened, angering Armenians. Azerbaijan said on Sunday it had taken appropriate measures to establish control at the starting point of the road.


We all rely daily on the structural integrity of man-made objects. From the supporting structure of the building you are in now, to the mode of transportation that brought you there in the first place. Your safety constantly relies on something not breaking. However, no structure is indestructible. Time, repeated use, the environment and even the occasional mishap all take their toll on a structure bringing it closer and closer to failure. Not the most comforting thought to have when flying on an airplane, particularly when you consider the amount of effort spent to optimize its design in terms of weight. So how do we deal with this sobering reality in practice? In the simplest terms, we design aircraft structures to fail, but in a progressive and manageable way. A friendlier wording to use is to say that we design structures as damage tolerant. We accept that damages can occur, whether it's causes predictable or not, and design the structure to tolerate that damage and its potential growth over time so that we can find it and repair it before it becomes catastrophic. Applying the damage tolerant methodology relies heavily on our understanding of progressive failure modes. The time between a damage exceeding a level that we can detect until it reaches a critical level defines the window of opportunity to detect problems through inspections. Ideally, we would like to have as large a window as possible to detect damages and make potential damages as easy to inspect for as we can. Both of these can be achieved through careful design. This approach to safety has been very successful in the aerospace industry, but we face an ever-increasing challenge. The structures of today are relatively simple, but the structures of tomorrow are becoming increasingly complex due to new technologies such as additive manufacturing, also known as 3D printing. These technologies enable structures that are more bio-inspired, more environmentally conscious in their design, more intelligent with embedded sensors, and made with new, complex material systems such as self-healing materials. We need to understand the implications of each of these complexities on the progressive failure behavior of future structures. In order to maintain the level of safety and confidence we have come to expect in modern structures. Although understanding these added complexities is necessary in order to achieve the current level of safety we all enjoy today, my research vision goes a step further. I believe that not only can we make structures as safe as today, but we can enhance their safety through this understanding. For example, by understanding the complex cellular structures of citrus fruits, we can tailor complex graded 3D printed structures to absorb energy in a way that improves the survivability of a crash or impact scenario. Other safety benefits can also be envisioned. What if we could improve the damage tolerance of a 3D printed part design simply by printing it in a different orientation? What if we could design a highly redundant and damaged tolerance structure inspired by the complexity observed in nature? In my research, I strive to generate this enabling understanding to ensure that the aircraft of tomorrow exceed the unprecedented level of safety we enjoy today.


So, Ben Miro, I just made you co-host. You should be able to share your screen. So as a quick introduction, this is the first of our large-scale SIGS video meetings. We used to do IRC meetings, but we also will semi-regularly do video meetings so that we can have more of a presentation and a theme discussion around a very specific question. And today, to kick this off, we have Ben Miro Morera from CERN who will talk about the choice between regions and cells when it comes to scaling out your open stack infrastructure. And hopefully that short presentation should trigger a lively discussion between the participants. The goal is to share as much best practices and good advice and experiences and questions as possible so that everyone can learn more about how to better scale open stack infrastructure. From a general standpoint, everyone manages to scale out open stack to very large infrastructure, but at the same time, we don't have strong documentation explaining how to do it. And for newcomers to the open stack ecosystem, it can be a bit daunting not knowing exactly what path everyone has traveled. So to kick this off, we have Ben Miro to get away. Thank you, Thierry. Can you see my screen? I can. Yeah. Okay. That means that you can hear me as well. So hello, everyone. I'm Ben Miro Morera. I'm a cloud engineer at CERN. CERN is the European Organization for Nuclear Research. And today in this first meeting in these new formats, as Thierry said, so we're going to try to discuss how to scale open stack, open stack deployment using regions and cells. So this will be a very brief presentation just to set the pace for discussion. So when managing an open stack infrastructure, one of the questions that operators always have is when should I start wondering how to scale out my open stack infrastructure? Of course, there is no special formula that will tell you that you need to scale your open stack infrastructure. As almost everything else in open stack, this depends on your particular requirements and expectations toward infrastructure. There are, however, some factors that can help your decision. For example, as you add more and more compute nodes into the infrastructure, we increase also the risk that if something goes wrong with the control plane, that will affect all your infrastructure. So to mitigate this risk, usually what we do is to set up high available solutions, most of the time very complex ones, for Rabbit, MQ and databases. Usually they are complex, difficult to configure, difficult to maintain and keep running. Network partitions in Rabbit MQ are huge issues in the infrastructure. So if you reach this point, probably you should consider the option to scale out your deployments. Of course, you can have also other motivations. So in these slides, I try to enumerate some of them. So of course, you have a large infrastructure, you have a large number of compute nodes that could be a reason. This huge number of compute nodes create a huge role in your database and in your Rabbit MQ. And you are reaching the point that it's being very, very difficult to continue to scaling up the database and your Rabbit MQ cluster. Or in the other way, you just want to avoid to set up all these high available solutions for these components. You want to keep it simple. Another reason could be that some of your OpenStack components, it's being very difficult to scale them. For example, Neutron. So if you have a lot of compute nodes, Neutron agents could be very, very shatty. And that could be very problematic to scale Neutron to thousands of compute nodes in the same cluster. Another reason could be that you want to partition really your deployment physically because you have multiple data centers in different areas and you want to expose these to users. Or just logically, because you as operator want to organize your resources considering the hardware type, different features that your resources offers to users. There are different ways to do this in OpenStack. We will try to explore some of them. Or in the other way, you want to isolate resources for specific workloads or for specific users. All of these are good reasons. This is not only to do with scale, but good reasons for you to scale out your OpenStack deployment. So there are different options to scale out. Today we will consider cells and regions. Let's start with cells. So cells exist in OpenStack almost from the beginning. They were initially implemented by Rackspace to help them to manage their infrastructure. At CERN we are using cells since our initial production infrastructure in 2013. I will give CERN example a lot during this short presentation to give you a concrete example. In 2013, we had two cells, one cell per data center. And we had two data centers and two cells. Notice that we didn't have regions at that time. And of course, in 2013, these were cells V1. Cells V2 were introduced much later in, I think, bike release in 2017 or 2018. So what are cells?  This allows you to share the communication load of your compute nodes with the control plane using different write in queue servers and databases. So as a result, you can increase massively the number of compute nodes in infrastructure without having, for example, to manage complex and central write in queue cluster and complex database setup. So let's see some of the cells' characteristics. So with cells in nova, we are able to chart the databases and the write in queue because we are able to do that. And we have a different control plane per cell. We can isolate failure domains in nova, increasing the availability of the clouds in general. Cells are completely transparent users, so there are no APIs to interact with cells. You need to use the old nova manage commands for this. They allow you also to dedicate resources to workloads easily, meaning that if you have resources that were bought for a specific project or set of users, you can easily map them. So basically, these are logical partitions in your cloud infrastructure. And last but not least, they are very easy to configure. So because they are so easy to configure, at CERN we use cells a lot. Currently we have more than 80 cells in production. Basically 80 cells that are the sum of all the cells that we have in our regions. Okay, so a certain example of cells. So this is the architecture of cells that we have. So what you see here on top is basically the cell API, where we have the nova APIs and the main controller. So this is the super control conductor and also the nova schedulers. Then we have the database for the cell zero, the database for the nova API cell. And of course the write in queue and the DNC proxy. So I'm just reusing a old slide, now the DNC proxy is in the cell control plane. And this is the cell. The control plane is very simple. So the nova API for the metadata service write in queue conductor. And then you have your compute nodes. Because we have a lot of cells, so we don't have high availability in this control plane at all per cell. So this is our choice. What you see here is basically the architecture that we have per region. So that's why I'm saying here that per region we have around 30 cells currently. And we have three main regions in our infrastructure. So I would like to show you now the cells that we have in one of the main regions. As I said, because cells are completely transparent users and there is no API to interact with them. So we need to use the old nova manage commands. So to see the cells in a region basically just do at least cells. I'm doing all of this because I don't want you to see all the database details and write in queue connections that also are displayed by these commands. So as you can see here, we have in this region all these cells, the cells zero. Then we have all these shared cells. We call them like this, like a normal user authenticated in the CERN cloud. If they try to put an instance, it will end up in one of these cells. This is completely transparent to the user. Groups of these cells will be one availability zone. For example, we have three availability zones. We group them like this. And then we have all these GVA projects. So these are dedicated cells that we have for specific projects. And those projects can only create instances in the cells and all the other users cannot touch them. Another particular cell that we have here is, for example, this one. This is the cell where we have all the bare metal nodes. All the ironic nodes are managing this nova cell. And these ones are particular cells for resources that have special features. In this case, these nodes are backed up by diesel generators in case of a power failure. So this was just to give you an example of a real use case of using cells. Back to the slides. OK, but cells are the problem. This is only for nova. It's a nova feature. With this, you are not scaling any other components. For example, if your neutron is struggling with loads, cells will not solve that. So we continue to add more and more nodes into nova and just increasing the load issues into neutron. So, for example, again, at CERN for a long time, we only had one region because we didn't want to expose this concept to our users. And because the load on neutron was starting to be so high that we decided at some point to introduce regions in our infrastructure. So that leads us to a region. So what is a region? A region is basically an open stack deployment. It's a completely independent open stack deployment that typically shares the same keystone and horizon components. Regions are useful when you want to explicitly expose users to a deployment partition. So users are aware of the region. And also, this usually helps them to achieve a greater fault tolerance for their applications because they are aware of this partitioning that in case of cells, they are not. So they know that they can deploy their application in region A and B. And if something happens to A, B should be up because it's independent. From the public cloud providers, this is the image that we have from regions. But the region doesn't need to be in a different continent or we don't need to have data centers spread between different continents or different countries or even different data centers to set up regions. So let's now go through some of the region's characteristics. So as I said, they are independent open stack deployments. Users are aware of them. They can give some fault tolerance to the applications running on top of them, the open stack infrastructure that use them as a whole. They can be used to scale components that can be shards. A good example is always neutron. However, we can share more open stack components. Usually it's only Keystone and Horizon. But also you can share other components like Glance, for example, and avoid all the issues of image synchronization between regions. Again, at CERN, we are sharing Glance between our different regions to avoid this issue of image synchronization. Something that we also do, we don't only dedicate cells to users, but also we dedicate entire regions to some special users. So for example, we have the use case for the data processing from the LHC. So we have dedicated regions only for these use case. And finally, regions, because they are an entire open stack deployment, they are much more complex to deploy when comparing these with cells. So as I show you the cells that we have in one region, now I'm going to show you the regions that we have in our production environment. So this is visible to the user. So if I do open stack region list, we're going to see the regions that we have available. Because users are able to see them, it doesn't mean that they will have access to all of them. In fact, most of the users, I will say 99% of the users will only have access to this region. We have three main regions that is the batch CERN and point date. CERN is for generic use cases. This is the region that I show you all the cells in the previous commands. Batch and point date, this is regions dedicated only for batch processing data. And then we use regions to test different functionality. So instead of pulling a completely separated open stack infrastructure to test, for example, STN, we just create a new region and because we need to deploy all the projects, we can have a different configuration to test this functionality. But we use the same Keystone and Lens and that gives us the ability to allow users to easily give us feedback because we don't need to create special projects or again users in these new open stack deployments. They use exactly the same infrastructure. So we just give users access to those regions and they can give us easily a feedback of these new features. So when we have, for example, these regions here, they have an handful of nodes there. We don't need to have thousands of nodes to the region. I think these regions have five, ten nodes maximum. All right, so we are reaching the end of this presentation. So should I deploy cells or regions? Of course, this depends on the problem that you are trying to solve. If you are adding more and more compute nodes, I would recommend that you start deploying new cells because that will allow you to scale massively. But later, if you start finding problems in other components, probably then regions will be a good way to go. More challenging, but if you grow a region too much with a lot of multiple cells, you can always split the region into two or more regions. We did this in the past because the scalability issues that we eat. So this was just a very brief introduction about these two concepts. Let me know your questions. I think this can be a good start for discussion. Maybe you will not agree with me. But that is good for discussion. Thank you. Thank you very much, Ben Miro. It was very interesting. We already have one question on the group chat, which was from Seung Soo Cho. He says, I upgraded Nova from Kilo to Queens so I don't have any cells in production. In this case, is it possible to create a new cell and re-sign existing compute nodes to that new cell? Basically, a question of how do you migrate from not using cells to using cells? Well, if you are still in Pyke, it means that you have cells V1 only. And creating now a cell V1 in Pyke, if you want to move forward then bringing to other open stack releases probably is not the best choice. I don't know what others think about this. Ben Miro, do you remember when cells was everyone started using cells by default? I can't remember what release that cut over was. So they were introduced in Pyke and the next rule is I think that became mandatory, no? Maybe. Maybe it was that quick. I think it was Cata. OK. I guess that's the thing to say. I think I'm messing up with the number of releases now. Could have been Queens, but yeah. I guess the thing to say is that currently when all of your hypervisors are in a cell today, so the nova compute nodes are all in a cell. So the trickier question, I guess, is how do you move a hypervisor between cells? Have you ever done that? Without deleting all the instances, I guess. I'm not sure that's possible today. I think I saw something in one of the release notes that said that some version allowed live migration of VMs between hypervisors. So I think one thing that you might be able to do is bring up a new cell and then migrate the VMs across. And then once you've drained the first cell sufficiently, you could then just take hypervisors out and we deploy them to other cells. That might be one way. So we haven't actually done it yet, but this is a fantastic talk because we're about to try and start going multi-soldier where I work. So I want to thank Balmyra, by the way. Thanks, Chris. I think that is supported since Huzuri, the cold migration between cells. But I think it's called migration. Okay, cold, but even cold would be great. Yeah, but definitely not pike, right? So you need to be really close to upstream or to the latest releases. We have two more questions on the chat. Benjamin is asking, is there a way to scale out neutron like nova cells in a single region? I've seen some user having a dedicated rabbit MQ cluster just for neutron, but wanted to know if other solution tricks exist. Well, if I can start, I think that is the best having a dedicated rabbit for neutron, we have it. So per cell we have a region, we have around 3,000 compute nodes. That means only one neutron. And we have a big ribed MQ to handle that only neutron. Maybe you have a dedicated database as well. No. Oh, right. But okay, so I can continue. So in our case, we have a dedicated ribed MQ for everything. For everything I mean for each component. So each cell has its own ribed MQ. Complete and independent. Then the nova API cell, of course, is also independent, right? MQ seen their different component has its own dedicated ribed MQ. Ironic as a dedicated right MQ. And this is exactly the same for the databases. So we don't share my SQL instances. Between different open stack components. So right MQ as a dedicated. Sorry, a new turn as a dedicated. My SQL instance. Like each cell, like each other open stack components. And do you have in separated database. For cell. Or is it the same my SQL for all cells? No, so it's a completely separated my SQL instance for each cell. Okay. One question. I have a brief. Pause there. We have multiple completely separate open stack installations in different locations. Do you know if we could just tie those together and make them regions or is it. It's too late now. So, I think that should be possible. There's two kinds of regions. So there's regions that share a keystone and regions that don't. It sounds like you've got regions that don't.  So I think in some sense you already what you can do is really have, I think the keys. I think you put the keystone. Is it right? You put the keystone for the other regions in the other regions. I've run out of all this, you know, I mean, like all the regions can see each other's services, but they're in a separate keystone effectively. I think that's how it's done. I don't know if you've, I guess you've not done that down my room, but. Just to be clear during the presentation, when I talk about regions, I was always thinking sharing the same keystone. Because for me, if you are not sharing keystone, they are completely independent open stack deployments. Yeah. Yeah. I think that's true. There's a long question on the chat, but it's, it's like a lot of words. So I wonder could that Chris just speak it or if not, I can read through it. If you don't have a microphone. Sure. Let me see if my microphone works. It works. Perfect. So my question is about new Trump scaling. You mentioned that you had to move from cells to what regions to scale in ultra well to scale your networking capabilities. However, in your presentation, you mentioned that you have 30 cells and each cell has on average 200 200 computes. So my question is, could you describe some of your scaling issues when it comes to neutron at that scale that you have? Because in my experience, one of the first scaling bottlenecks that we hit was with actually an arms networking and our computes scaled pretty well, but a number of flows, a number of instances, a number of routers created by our users, they basically all started overwhelming the control plane of the Newton. So my question is whether are you, whether you're using one of the standard amount to drivers for neutron, for example, into bridge or yes, or are you using some sort of custom networking solution that let you sidestep those issues and scale note on higher than what we have observed now deployment. Right. So in our case, each region has more or less 3000 computer nodes. So that is this 3000 computer nodes is what's the each Newton installation handles. But we use a very simple Newton setup to reach this. We use the Linux bridge driver. So it's extremely simple. So I don't, we don't have the issues like routers, all the shattiness that you have. However, even though we're using the Linux bridge, the Linux agent is extremely chatty. And the load that we see is not really in the AP, the Newton API servers, but in the right in queue. So the scaling issues were more related with scaling the drive, the dedicated right in queue cluster that we have for Newton. Probably if you are using a different network driver, you will, you will reach these scaling issues much more early. Okay. So if I can, if I could, like ask me a follow up question, my understanding is that what you said is that you have a fairly flat network with your new talents. And those talents don't really think about open stack in a context of networks. They don't create their own. You basically have an open stack that plugs into our machines almost directly into your, into your underlying network, right? Exactly. Okay. So that's why when you saw the regions and I saw, and I told you that we had very small regions for testing new features. Those regions were called two of them, SDN one and SDN two. Because we are testing as the end functionality for infrastructure, but it's still lovely days. Sure. Thank you. There is a question in the chat around, how do you share, share glens between the regions? Does glens has its own storage backend? And someone already provided some answer. Maybe, maybe you can, like a the burger can, can already help answering that question. And then someone else can, can reach, can complete that answer. Well, I can't, I can't add anything more than I just said, because we didn't implemented it where I work. But that's mainly the idea we have when I'm asked that question. Yeah. So your answer was that using a shared keystone with replicates with backend for glance, you should be able to achieve. A, a glens between regions, right? Exactly. Okay. So how did you do it? So we do it in a very simple way. So our storage backend is stuff that it's running in our main data center. And we set up glance normal glance, it's just once a piece running pointing to that CF cluster. set up a different lens in a different region because you are concerned about all the latency and network traffic between the data centers. Even when in the past we had two different data centers, we only add one lens. Initially we deployed the lens cache in the other data center. You can use that to speed up the transmission. But at some point we just remove that and we rely completely on the one-glance setup. So to have that between multiple regions, it's just pointing the region to the same glance endpoint. So when you do open stack and point list, you get all the endpoints and each endpoint is pointing to a region. So if you just duplicate that and the point for a glance in this region is the same like the other one. And because you are using the same keystone, it's completely transparent. So they will use the same glance in the endpoint that you specify. But again, this depends in your use case. We don't feel any need to use two different lenses in different regions because actually in our case they are in the same data center. Is there any other question? I thought we could, if there is no more question, we could like do a quick round around the table to discuss like other setup because we've had lots of details on the certain setup. So you're basically combining cells and regions. So I'm kind of curious what's the state of skating out for others on the call. If, like Chris, you just said that you were in the process of migrating to multiple cells or regions. Belmiro, I'd love to get a copy of the slides because I have other people interested in what you share with us here, if that's possible. I don't know if you just published them. Yes, sure. We will also reference them on the large scale seek page, like all the video meetings with presentations, with the recording and the deck. Fantastic. I can talk about what we're doing. So our new opens deck is taking off at Bloomberg. By new, I mean it's opens like Rocky and it's Neutron. We had other stuff with Nova Network that we're retiring. So three data center sites, three open stack installations, all completely separate, but we are seeing scaling problems already with Rabbit and MySQL. And each site is going to have basically additional what we call data holes. So they're not really separate sites, but they're physically separate buildings, physically separate power. So that seems like an obvious candidate for cells so that we can just partition compute and give each data hole its own control plane, its own rabbit, its own MySQL. And I did some research on that, and that seems to be the way. But I learned something today, which is that that's not going to solve scaling Neutron. So we're going to have to put our thinking caps on and make sure we don't have like half a solution when we do that. And the scale is, you know, approximately 1000 compute per location right now, but it's growing every week. So it's a small thing compared to CERN, but it's still enough for us to get into trouble with scale. And the population is, so our model is more like, you know, more traditional stuff, like fairly heavy VMs. We do have, you know, containers and Kubernetes is going on as well, but, you know, quite a few are just traditional Linux servers. And it's in the tens of thousands of VMs already on this, on this new platform. So we have to solve scaling either with cells or with, you know, extreme measures in Rabbit and MySQL.  So I think Belmiro's insight here is fantastic. And I think we're definitely doing cells. I don't know if we can go and stitch it all together into regions. The other thing that I found was another learning thing was fantastic is that you actually have cells for your kind of preproduction resources like testing, because we just got a whole bunch of hardware for preproduction testing. And we were thinking of making them entirely separate open stack installations with their own endpoints, their own DNS entries, their own keystone. But maybe that's wrong. Maybe they should actually just be, you know, part of the open stack at that site. So that's another great thing that this meeting gave to us. So thank you to Belmiro and CERN for forging that path. I don't know what else I couldn't share really, but we're kind of following along, I think, things that CERN has already seen and solved. It's a nice feeling. Thank you, Grace. Maybe I'll know you can share a bit on how you do scaling at DoVh. Yeah, hello, everybody. At DoVh, we are not using cells at all for now, mostly because we were eating some scaling issue on Neutron before the one on Nova. So we decided to scale using regions. So yeah, that's basically what we do. On Neutron site, we are not using regular ML2 drivers. We built our own driver on top of OpenVswitch1. So we don't have this all of the flows issues that you have between computes. The most impact we have on Neutron server is about the load that can be on on HabitMQ RPC side. Because every time you restart agent on the compute, for example, if you restart agent in a bulk, you will end up in with a high load on the RabitMQ side and on Neutron side. And that's why we decided to switch to scaling using region instead of cells at the beginning. I don't know what I can say more. That's great already. It's an interesting view that you did not need to do cells actually. Yeah, because it was mostly in Neutron. And we know that we can do more because, for example, we are not using separate HabitMQs like you do at CERN Belmiro. We are using one big cluster for every OpenStack services. And that's a thing we can upgrade. I mean, we can separate RabitMQs in order to have one specific for Neutron, at least. That's something we can do. It's not planned yet. But we don't know what will be the future because we don't want to scale region indefinitely. We want to maybe roll back or to find a mechanism to hide these regions to our clients. So for now, we don't know. But rolling back from region to cells will be hard because of the way it's done. So having a mechanism on top of regions to hide everything could be nice. But as far as I know, there is nothing magic already existing on that topic. So for now, we are just getting using regions. Okay, any question for Arno or Chris? Maybe? Well, Arno touched an important point that I would like to highlight. If you restart all the computes in bulk in one of our regions, we'll see exactly what you have served. So RabitMQ will not handle this. Even with a simple ML2 driver that we use, you'll not be able to handle this. So every restart needs to be very controlled. Yeah, exactly. Exactly. So what you do at CERN is you are restarting it very smoothly, right? Yes. Yeah, just like we do. Okay. Well, we avoid to have a bulk restart. But if it's needed, it needs to be with some care. It's interesting because it's been mentioned by several people that restarting actually triggers a lot of problems that you don't really have in regular operations. If you're like, it's an interesting topic for maybe a future. How do you avoid most of the problems around that? Yeah, we also hit that problem. So we restart compute nodes by batch to reduce those on neutral. Jean, can you share more about regions and cells at line? Yeah. So we currently don't use cells because we're using the older version of open-stack, which cell is not quite implemented at that time. We do use regions, but we actually separate the regions by data centers. And yeah, for the neutral issue, everyone is hitting. So our solution we go with it's going with an L3 data center solution. So we use our custom Neutron drivers similar to Calico. So we don't need to handle all the ARP issues or other issues when using ML2 or OBS drivers. Next, regional OBS drivers. Yeah, but it's kind of a different use case because when you use Calico, or like an L3 driver, all the VMs will be reachable, so there will be no private networks in this case. Yeah, that's all I have right now. It also seems like the choice of Neutron driver is affecting how load affects the cluster in general a lot more than I expected. And the choice of driver actually has a big impact. That's another question we should probably explore at one point. I see Imtias or Amtaz. I probably butcher your name, sorry. Can you share more about your setup? Sure. Yeah, Imtias, you got it twice the first time. Perfect. So we also looked into cells and regions, and I think I discussed with Belmira a while back. But currently we're not using either. Like even though we've become quite big, we just surpassed like 11,000 compute nodes. But one of the reasons we could have used regions, but between our data centers, there's not much connectivity. They're completely isolated. It's a little complicated to set up the inter-data center connection. So that's why we started with very simple setup. So each deployment is its own like open stack cluster, and we scaled it up to 300 nodes. I know we could probably go even further, but our limitation was more the Neutron provider that we chose, in which in this case it's the Contrail. We are looking at Calico moving forward. So that also opens up, and I know talking with other operators, like it would allow us to scale even further. Like even without using cells. So that's what we are doing. But I mean, we do see the advantage of sharing a common keystone and glance. Like I think we are noticing that our number of images we have is growing rapidly as more and more services are on morning. So trying to replicate for each cluster does that up. We are using Cephas to back in, and Cephas now maturing more within our company. So it will let us like do a lot of glance replication and having a shared glance for multiple clusters. Right now each cluster has its own glance and like the images get copied everywhere. So like one of the plans moving forward is to have a shared glance service, which like you can have different open stack instances, but you can still share the same like glance back in, which is also backed by Cephas. So that's what we are going forward with. And I think you guys already talked about neutron being the bottleneck or a decision point in terms of when you choose, when you are trying to decide whether to use cell or not. And that was our case as well every time we tried to consider like whether we could use, let's say, Contrail with cell, the version of Contrail we had an open stack combination that didn't support going to cells. So that was another point. And I'm not sure with Calico, I think I hear it's possible. We haven't explored that. So has anyone here used Calico hand cells together? We're about to try that. Let us know how it goes. Calico is very lightly integrated with open stack. I think that's a nice way of putting it. So we don't do much with open stack with Calico really together. Calico is just controlling the firewalling really at the endpoints. And we put the policy in separately from open stack. We don't use security groups. So I can't see why it would not like separate cells. But of course, famous last words, right? So I definitely will report back if we find stuff. Okay. Anyone else interested in sharing if they use regions of cells and why and or questions for people that already expressed their explain their setup? Well, I'm curious if someone is using cells V2. If it has more than one cell V2. Oh, because you were still using cell V1. Oh, no, no, we are using no cells. We are using cells V2. I said cells V2 just because because in my mind, cell V1 was completely deprecated. It was completely removed already. But if you are in the old open stack version, you don't have cells V2. Okay, looks like they are not that popular. So we would be on it already if we could have wound forward on open stack version. We're still on Rocky. We want to get to a story urgently for many reasons. And as soon as we do that, we're going to start testing cells on hardware. And we think, as I mentioned, we think we're going to go larger. It's just that we got an open stack with Rocky and Neutron and Calico working.  And so we've just been just keeping our head above water. So it's not that we don't want to, in fact, I'm being asked which quarter are we delivering in this year. So I'm going to be about to be doing it any second, really. So this talk has been incredibly timely. Yeah, thank you. I was just going to say we'd have done some work with Color Ansible and getting that able to actually configure multiple cells V2. But a lot of the larger deployments have been so heavily bare metal focused that we just haven't needed that kind of Nova scale out. Like there's just three Nova computes on three controllers doing fine with, you know, 1200 nodes of Ironic, for example. It's just not got that far. But in case someone wants to start writing that in Color Ansible, it's there's bits of it there. Okay, anyone else interested in sharing their current situation and questions? I had a question for the people running multiple regions. We have a few folks doing that. We get a lot of users, well, in terms of like having their application and doing like global server load balancing within the open stack ecosystem. That's quite hard to do. Or if you've got whether you want a Kubernetes cluster that's spanning the multiple regions or doing something outside, I don't know if people have found people gravitating towards a particular solution for that kind of thing. I guess, Belmour, in your case, all of the flat network, all three, all the regions are connecting to the same flat network. So, okay. Yes, they are. Just wondering if the people have hit that. Well, in our case, we took a look last year on the how we utilized the sort of global keystone because we have a global reach and having a sort of one keystone and one horizon for the users. But that turned out to be discussions with the people that if you find out how to do it, then let me know. Kind of thing. And the other thing that we had was this net Newton issue that we really couldn't find any kind of, let's say, good solution to scaling over our about 300 nodes set up at the moment. So we are still in the drawing board with the regions and cells and trying to follow the situation that what happens on the neutron front. Because that neutron gives kind of interesting day-to-day problems also. If you, for example, in our case, because we are utilizing that they're not networking and stuff like that, then guessing the quotas for the neutron elements is kind of really guesstimate. And then you have these eager users who can create 100 networks there. And suddenly you have tens of thousands of open stack boards in the cloud and the neutron is kind of having very much issues there. But as I said, we are still on the drawing table with this one. I suppose I should also share with Color Ansible, we have used that to set up the shared keystone, shared horizon with then two regions that each region has its separate kind of Color Ansible config, actually KUV config. So that should be possible. I think there are Color Ansible docs on that. Okay. It depends what tool you're using of course. If no one else wants to share, I'll conclude this by explaining a bit more what the large-scale SIG does and our current activities in case you're interested in joining. So I'll just share a quick slide here. Hope you can see it. So the large-scale SIG is a group of operators and developers and interested in facilitating running open stack at a large scale, like answering questions that operators have as they need to scale up and scale out their input stack clusters, but also try to address some of the limitations we encounter in large open stack clusters, like try to detect the bottlenecks and then instrument them and potentially push back some of those limits that force you to look into scaling out solutions like regions and cells. We are meeting every two weeks. Usually it's on IRC, but we also do sessions like this one around a specific question and kick-started with a presentation by some large-scale SIG member to try to get the discussion going. The current schedule for the meeting is Wednesdays at 15 UTC. That's 4 p.m. Western Europe and 9 a.m. Central in the U.S. But if we have substantial participation from the west coast, especially we'll probably try to rotate between multiple times. We tried that, but nobody showed up at the other time, so we stopped doing it. Finally, this is the main product of the large-scale SIG. It's a wiki page called the Scaling Journey. Basically, as you create an open stack cluster and you add more nodes, at first you need to configure it so that it can handle some scale, then you need to monitor it to see how well or bad it runs. Then you scale it up until a certain point when you have to scale out. Then, obviously, you have to maintain the life cycle of the installation, a grade, et cetera. Those are all stages in your Scaling Journey, and we're trying to list the questions and provide answers that almost everyone will have at one point. As an example, stage 3 has this fact here where we're saying how many compute nodes can a type equal open stack cluster contain. The answer is, obviously, it depends. It's always it depends, but at least it gives a range between 100 nodes and 2,000 nodes depending on the type of usage you have and the API shown that you can have. Those kind of questions, we're trying to make sure that we provide a basic answer to them and hopefully that gets people more confidence before they start their Scaling Journey with OpenStack. The main activity of the SIG is to try to document those questions and try to extract all the knowledge of all the participants in the OpenStack community. Obviously, this session was really great because I feel like I learned a lot of things on when you use cells, when you use regions, the problems it solved, the problem it does not solve, and that's exactly the kind of content and value that we're trying to create at the large scale SIG. Thank you again for everyone participating, and thanks again Bill Miro for kickstarting this series of video meetings. I'm not sure exactly when the next one will be, but I'll make sure that everyone knows when we schedule a new one. And in between those, if you're interested in joining the large scale SIG, please join our IRC meeting. The next one is in two weeks. Anything else, anyone? I think it's important for people to give their opinion and for us to understand if they think this is useful, if we should try these new formats in the future with a different topic. What do you guys think? Should we try again? We should definitely try again. I wanted to speak for about 10 seconds. So some of you guys have been to ops meetups sessions, but just to let you know, most of the team is no longer working on open SIG or has personal issues, whatever. I'm still here in Bloomberg, so we're going to try and bring something back, but I don't know when we can do that. So I'm going to try and turn up to open to large scale SIG from now on, and maybe this is my new home for these things, but I'm still working on open SIG ops meetups when the world comes back to normality, but the rest of the team probably isn't. So that's probably why we've been so quiet. In case that's of use to anybody. That's a great point. One of the reasons why we're doing these video meetings is to provide some virtual discussions around operations of open stack. When we had the ops meetup, it was a great way to share that experience, but I feel like even in the world of tomorrow, we'll have to have some kind of mix between in-person and virtual events anyway. So I can totally see how we can do the ops meetup for like having this anchor point and have all those personal relationships building and at the same time do maybe with less frequency, but do those virtual discussions as well. So that's the best of both worlds, I think. Thank you. We're over time. Thanks again, Belmiro. Everyone else, please, if you are happy with the session, make sure that you mention it in the chat. Keep it open for a few minutes and we'll let you know when the next one is. Thank you all. Thank you. Bye-bye. Thanks. Bye.