Welcome to the Bogle Heads chapter series. This episode was hosted by the Chicago Virtual Chapter and recorded April 15, 2021. It featured Lauren Bolin, the developer of Seafire Sim. Bogle Heads are investors who follow John Bogle's investing philosophy for attaining financial independence. This recording is for informational purposes only and should not be construed as investment advice. All right. Well, thank you for inviting me, Jim, and for what else was involved in that email chain. My name is Lauren. I'm the developer and creator of this site called Seafire Sim. Let me, I'm going to start sharing here. Yeah, I'm going to stop sharing. Which screen? This one. Okay. Bear with me. All right. So I'm the developer and creator of this sort of retirement backtesting calculator called Seafire Sim, sort of a mouthful, marketing and SEO or not my thing. I developed this sort of in 2013, actually as a side project, and it's sort of in this passion project of mine. I have a very loose agenda here.  I'll try to hit at least everyone a little bit. We can go into more depth if people have questions on specific things. And always, you know, one of the most important things about my site is the first word crowdsourced. I really like to take other people's input when I was developing this thing. The whole reason I developed it was because of other people's input, and I'll get into that a little bit on the history. So at the end, if your question isn't answered, my email is at the top of the website. You can feel free to contact me anytime, plenty of people do. So here's sort of a rough agenda I'm going to go over. I'm going to sort of introduce myself a little more, talk about how I developed this project and go over a little bit of my sort of development practices, which I won't get too much into the tech details or tech nerdery. But I have a little bit of information out there if you're the kind of person that's interested. Then I'll sort of go over basic functionality on the site, some of the more advanced features. My favorite part of the site, which is the different spending plans. And then I'm going to go over some sort of fictional scenarios to show you how someone would use it given your situation, whether you're paying for two kids college at two different times or maybe you're still saving for retirement or maybe you're going to have a part-time job in retirement. All those things can be done on this tool and it's kind of useful. Then at the end I can talk about any sort of questions. So before I go anywhere, I guess I'll introduce myself again. My name is Lauren, I'm a software developer by trade. Me and my wife live in the suburbs of DC. We have two kids and I've been interested in sort of the retire early, the fire movement probably since 2008 or 2009 when your money, your life was really the only resource. I was kind of really involved in the internet when it became more of a blogging thing with Mr. Money Mustache and various other people and now that sort of movement is quite rampant out there and I'm happy to see it. At some point in my life I was a moderator on Mr. Money Mustache forums, I was a moderator on the Reddit financial independence site so I've been around. So a little bit of history about Seafire Sim. Really this project was born out of conversations I had with people on earlyretirement.org.org. So for those of you who don't know, I mean I think most of you would but earlyretirement.org is sort of the forum website that happens to be attached to a website called FireCalc. FireCalc, if you look at it today, is really a sort of retirement calculator that looks like it was written in the 90s and I believe the data is updated yearly still. But the reason it hasn't been updated and not to diss them honestly because I like their project in general, but the reason it hasn't been updated is because the person sold it. He retired and is living on a boat as far as I know so bravo to him but the folks at early retirement.org get a lot of clicks for having that site up there so it's good. But in I'd say around 2011 or 2012 I was hanging around on those forums and it was pretty clear that there were people asking for new features on this website. No one was there to make those new features so it was a lot of people sort of screaming into the void. I was not a software developer at the time. I was trying to learn it and I thought that hey this would be a great project. One of the things that was useful for me is that FireCalc outputs a CSV file at the end and it's kind of a mess a lot like Cephire Sims CSV but it's essentially a data dump of every single iteration of the simulation. So I figured if I looked at this long enough I could reverse engineer it and figure out what's going on and turns out it was right. I think it took me a month of nights and weekends of just sort of going through the steps to figure out what was going on, how they were applying inflation and doing all the calculations but I did reverse engineer it and I started putting some code out there. I took a lot of input from folks on that forum and from folks on on credit in other places during that time. There's a couple people who are either on this call or will be listening that were a big part of that. I've never actually heard his name said out loud but Jerome moistened was a bogeyhead in your group that was a big help back then and there's a username CMON that also I spoke to quite a bit about 10 years ago so if you're out there thank you for your help and anyway I put this project out there in 2013 and personal finance and fire is sort of my thing and I actually used it as a way to shift my career now I am a software developer by trade so it's great to be paid for something that you like but real quick going over sort of general development practices of how this tool work or how I work on this tool for any of you nerds out there I host this on a platform called digital ocean which I believe just had an IPO recently so go digital ocean I am not I'm not an advisor don't take my word for that I sort of viewed the the website's written in python and a framework called Django has a mysql database and I use a lot of docker containers and of course there's some javascript written in there for the form magic that you see in the past I've written it in PHP then I rewrote it completely in AngularJS then I rewrote it completely in python so it's gone through a few iterations um one of the things I know a lot of people are concerned about is uh testing how do I go about testing this when I add a new feature does it change the outputs this is kind of a tricky question um some of the time it does change the outputs and the question is whether or not that's the intended thing I talk to a lot of people when I make changes that actually affect the outputs a very a very big example of this is I would say sometime in 2014 I'll provide a link that we can attach to this video or or the thread in the bogey hats forum but sometime in 2014 or so someone came up to me in the forums and said hey um how do you calculate the bonds in on your on your site and I gave them the formula that I've been using it's based on the gs10 long interest rate that's you know publicly available data that's gone back for 150 years you know someone's printing in my office um they came to me and they offered some information and we sort of hashed out the mathematical formula well it turns out that um sorry this is really loud and bothersome how many pages um it turns out that the fire calc and a lot of different sites were calculating the bond rate incorrectly um we sort of went over it as a group and came up with a slightly different formula which actually affected some of the success um I would say that there's probably a two or three percent difference in success rates between cfire sim and um fire calc given the amount of bonds you have in it um so this is sort of like an interesting point in testing where um it's just going to be different um I do have lots of unit tests and for those who are not software developers sort of unit tests are pre-built tests that have pre-determined outcomes and every time you make a change in your software you can just run them immediately and see what happens um if those pre-determined outcomes are different it will let you know uh so I do keep an eye on that honestly the biggest thing that I rely on is user testing um almost I would say 95 percent of the bugs that happen on my site are just UI bugs um I was mentioning to Jim before everyone came on that you know JavaScript is the programming language that lets you kind of mess around with the UI I'm not that's not my day job I'm not used to doing anything with JavaScript um so sometimes little UI bugs get in there but um and lastly I've mentioned this before but I always take advice from users um I'm always looking for new features or her quality of life things I don't want to make this thing too complicated in the end so I you know reserve the right to shoot things down but um over the years I've added a lot to the site based on suggestions and it's been great um all right all right that is basically some of the back and history of this and I'm going to go through some of the basic functionality on the site and see if any questions arise so I'm going to refresh this just in case when you arrive here on the site you're presented with what I call the you know giant form um there's a lot of inputs here um you can open it up depending on what you select for even more inputs but there's sort of this basics section that I'll go over in the beginning um a little bit of the the back end data is to know is uh I go off of um Robert Schiller's data set which is pretty common um Robert Schiller's data set has like sort of the s and p 500 um equivalent uh stock index for the last 170 years and also has the bond rates and I also have a separate um data source for gold um you know one thing that I would say is lacking in this program is historical cash data I've never been able to find a good historical cash data that goes back all the way to 1871 there's quite a few that go back to 1900 um and I've never bothered to sort of jam those together so uh you know that being said I get I offer the ability for you to put in how much cash you think how much growth you think cash will have in your portfolio um maybe this is a lesser um tended to feature because I don't have a lot of cash but so the basics here are sort of I would say pretty common for a lot of retirement calculators you have a retirement year you have a retirement end year better known as your mortality um you have sort of a a check here for what kind of data you want to use I almost always use all historical data but some people like to really nitpick on the worst periods of history and see if their portfolio would have survived um which is useful but I mostly like to see everything um and then you sort of have your portfolio value as it stands now which is important and then what your yearly spending is now in today's dollars um so I know I'm not going to assume people's knowledge here but everything on this page is essentially in today's dollars which means um you don't want to inflate these numbers for the future if you're going to retire in 30 years and right now you're spending 40 000 dollars a year sure maybe in 30 years that's going to be 70 000 dollars don't put that in this calculator everything is in today's dollars I personally find it easier to wrap my brain around when everything's in today's dollars um and everything does get adjusted by inflation um as you'll see on the output um so initial spending spending plan is something I'll go over later there's quite a bit in here um quite a bit of complexity in there but right now this is inflation adjusted spending which will adjust this initial value up and down based on historical inflation which is a pretty common feature and I think some of these calculators um here's sort of a drop down for what kind of inflation type you have so this is CPI this is the historical inflation you can do your own flat rate inflation um if you really want to some people like to use this for you know sort of bad scenarios where they're like oh maybe inflation is going to be four percent for the rest of eternity or maybe it's going to be two percent I'm not sure um but I tend to use historical myself um those are sort of the basic things that you need to run this in general everything else below here is sort of adding details to make it more of a fine-grained simulation so like I mentioned before about cash here's your portfolio area important to note that this is essentially the portfolio that's going to be throughout your simulation um now you may ask you know I'm going to change my portfolio maybe I'm going to get more aggressive or maybe I'm going to get more conservative as I get closer to retirement you can do that I do have the functionality in there to provide what's called a glide path which sort of takes your allocation at one point and moves it slowly to another point at over a certain amount of years so the default here is 75 percent equities 25 percent bonds I think this is a pretty standard default for a lot of retirement calculators if you want to do those glide path type things you would uncheck this keeping the allocation constant it will reveal a new panel here and so in this particular situation this person is retiring in 2021 which is this year um there are some financial gurus that seem to think if you start off in a conservative portfolio and slowly move to a more aggressive one during retirement that that's a viable thing so I wrote in here we're starting off with a 5050 portfolio and here is the start year of the glide path and here's the end year of the glide path so over a 10-year span it's going to slowly ratchet it up to a 90-10 portfolio I would say one of my one of my personal uh to-do lists for features would be to add the ability to do more than one of these I think it's pretty common for people to change their allocation throughout their investment careers multiple times so I wish I could do this twice or three times but right now you have the ability to do it once for simplicity's sake I'm going to hide this for now but this can be pretty useful in trying to model different risk scenarios so I'm going to click this button and make that thing disappear you'll sort of notice that a lot of things on this website appear and disappear based on your inputs hopefully that that's intuitive I haven't found a lot of people have a problem with that so hopefully it's not too bad um rebalance annually is an interesting thing I don't know your own personal views but I think most folks that are interested in planning their own retirement are rebalancing annually or twice annually I mean maybe people are doing it every quarter I don't know all this means is it's rebalancing your funds back to this portfolio every year based on the simulation if you don't do that it'll just let the simulation drift it'll let your equities go up and up and up or your bonds go up and up and up during certain times in history and it'll never rebalance that's certainly an interesting thing to simulate you can try it out on your own and see the effects of that and lastly one thing I haven't mentioned here is fees this is essentially the average fees across your entire portfolio I know that that's not ideal people have different fees and different accounts because they have different funds in them for the purpose of simplicity I put this in here as just an overall average on your portfolio this is like some sort of average of vanguard funds that I found out there so I put it in there all right below portfolio we start getting into some of the inflows and outflows that you can add to your simulation and starting with the most popular at least in the United States here I do have several people from Australia and the UK that visit this site and you know make use of other adjustment types but since I'm in the US and I will be collecting social security hopefully I added this so we have two different social security fields one for yourself one for your spouse by popular request I've made it so that you can change this from a monthly to an annual number I used to have everything annually but most people think in monthly terms when it comes to social security how much am I getting each month so whichever you put in there it will do the right thing on the back end of the simulation of course you can model different start years I know lots of people try to figure out whether it would be worth it to take social security early or take it at the full of retirement age you can certainly do different simulations based on start year and of course there I used to not have an end year in here I used to just assume hey this is going to be you're collecting this for the rest of your life but lots of people want to model their spouse's deaths so I added an end year in here and you can model different people's deaths and changing of values in here so this is essentially one example of what I call an adjustment it's sort of a general term for the inflows and outflows of money in your portfolio each year in the simulation a pretty common theme and I'll show you right here this is the generic this is the general adjustment section so you can put up to I believe I've allowed up to 10 adjustments in here there's no reason I couldn't increase it I just have kept it at 10 but um this is essentially where you would put your pensions where you would put you know a different spending things that would happen during retirement maybe you're going to retire and then you're going to start paying for your kid's college so you you know my kids are going to be in college in 2030 and then I've got another kid in college 2033 I would model a couple different spending things good old in-state tuition in Virginia right now is roughly 15,000 a year so this is sort of a general thing that you can do with these adjustments I have pension availability in here any sort of income and savings which could be I mean it could be anything it could be you've got a part-time job you're renting a house out and you're a landlord so this is your rent you can also make this a one-time thing so imagine you're going to sell your house you can uncheck this recurring box which will sort of gray out the end year and this particular adjustment will only happen on 2030 so these things these kind of things are pretty useful some of the inputs in here might look familiar you know amount per year that's pretty simple this label here you know I find this useful mostly because there's an ability to come back to your simulations later so real quick I'm just going to run this and sort of ignore the output because I can click this link and it will reload the exact same things that exact same inputs that you entered before I'll talk a little bit more about that when I'm on the output page I apologize for jumping back and forth so this label is important this is also important some people have been known to mess up and put the wrong thing in here where they they meant spending and they put savings and they're like why is my why do I have so much money at retirement it's like well you you said your kids college was actually savings or income stream instead of a spending stream so that's an interesting thing that you need to look out for and one sort of option in here that might be a little different for you is this value here I wasn't really aware this is this is where the crowd sourcing comes in I wasn't really aware of this because I don't have a government pension I know that there are other my wife has a she has a private pension and actually I think this applies to it but I don't pay much attention to her pension and stuff but um this little checkbox essentially is for pensions so a lot of pensions will have this setup where they'll tell you hey you're going to get $5,000 a year you know once you start collecting your pension that value is often frozen and it's always going to be $5,000 until you start collecting and then only after that will it be adjusted for inflation so if you have a pension that's like that you'll want to click this button and have this clicked inflation adjusted and it will stay at $5,000 until you start collecting it now that may seem intuitive but anything on here that you select inflation adjusted for that's in the future for instance so I just unchecked this and this is nine years in the future what's going to happen is even though you're not taking this pension until 2030 this adjusts that value for inflation every single year so nine years down the road this might suddenly be $7,000 that wouldn't normally be a problem for a lot of things but like I said pensions have this knack for being frozen in time until you start actually taking them so this is a worthwhile thing I forgot to mention at the very beginning of this some of the things at the top of my site fairly recently I've started writing tutorials that sort of describe some of this functionality so if you miss something on here you don't want to go back through the through the video you can certainly check out this tutorial section I'm going to click here real quick right now I sort of only have information about the basic section and what the output looks like but I definitely have I've already written articles for spending plans I just haven't bothered haven't gotten them up yet and I definitely am going to talk more about some of these things I'm going to quickly scan over essentially the information that I'm going to put in these in this demo so hopefully that's helpful and go back here okay so adjustments recurrent the recurrence button I sort of mentioned does this occur over a span of years or is it just once and then whether or not this is inflation adjusted or not is pretty straightforward just like above you can choose the inflation type it can either be the historical CPI or can be a flat rate I actually know that a lot of pensions have a flat rate adjustment so that can be pretty useful and let's see I believe that that is essentially a quick scan through of all available inputs without touching spending plan so spending plan has lots of different things that show up after you click the appropriate spending plan and I'm going to go over that in just a moment firstly though I'm going to refresh this page have the default scenario and we're going to talk about the output page so this is a pretty standard situation in fact I'm pretty sure this is the exact same inputs that fire count does this is a 30 year scenario it may look like it's 29 years but it counts this year and this if you it's 30 years with a million dollars and $40,000 spending per year this is the classic four percent rule I'm going to hit enter or hit this button up here and this lovely rainbow graph is what you would see when you hit run simulation there's a lot going on on this page I've tried really hard on this site to put as much information out there as I can without making it too confusing however I do understand that because I'm the one who wrote this I know where everything is and it might look a little wild to someone who hasn't checked it out so I'm going to go over a couple of the sections here so the first thing you're going to see is this really big wild rainbow graph so what this is is essentially every single simulation of 30 year span over the course of history starting in 1871 and what I mean by that is let me see if I can I think I wrote a pretty nice nice graphic of this oh no I got rid of the graph oh wait I'll pretend I got rid of the graphic that's a bummer so what you have to imagine is is this data goes back to 1871 so it's going to take your inputs and it's going to go to 1871 and put them in as a theoretical million dollars and a theoretical $40,000 spending and it's going to run the data from 1871 to 1901 and it's going to see how your portfolio would have done then it's going to start a new simulation in 1872 and go to 1902 and so on and so forth until you get all the way to the 2020 data which if I try real hard I can find it in here but this results in sort of this cool graph and you can kind of get a sense of what's going on here there are periods of if you hover over this it sort of hovers on an entire simulation so right here it's kind of hard to see in the blues but that blue line dips down really far this was not a good period in time in fact this is the classic beginning of the stagflation period in the late 60s and early 70s so what this represents is your portfolio over time now like I mentioned before this is today's dollars so it's adjusted to make it easier on your brain to figure out what this is worth so if you started with a million dollars this is worth roughly six times as much however in nominal dollars this is probably way more I'm just going to make a speculative guess and say this is like 11 million dollars you know double that or more in nominal dollars now how do you figure that out you could go into the csv up here I guess I will open this up this is sort of stream of consciousness so if you click on this button you're going to get this long named csv file I don't even have word on here or I don't even have excel on here where my screen go all right it's going to load it up and this is not going to be very easy to show people but this is a very long file and it has every year of every 30 year period in this simulation so if you go to the top it says the cycle start year is 1871 so this this is the 30 years starting in 1871 and it has your starting portfolio it has the adjust of inflation adjusted portfolio has the spending and the inflation adjusted the spending so for this particular one which is the one I think I was highlighting if we look at the ending portfolio and this this is just a quick a quick tangent ending portfolio right here what was the ending portfolio of this one 3.2 million and the inflation adjusted was higher gotcha anyway this is a bit of a tangent but good to know that you can look at all the numbers there this here is a big ugly link that will allow you to get back to your same inputs a lot of people ask me why I don't have a login system so that you can save your data I have taken a heart stance on that because I don't want to deal with keeping the data anonymized to be to be honest right now as it stands I have a database full of people's inputs I have no idea who put them in and I'm okay with that I am not in the business of selling data I don't make any money from this site this is 100 percent passion project and I want to keep it that way so I anonymize the data and I put it into here as a link so you can get back to it rather than have a login system that's tied to your email maybe that makes you feel good maybe that makes makes you feel like this tool is clunky I'm not sure but okay back to this chart real quick as I'm scrolling through here if you look below down here I can't point to it while being on the screen but if you look below it says that the year is 1953 the cycle start year of this particular simulation was 1935 and at this point in time the portfolio is 1.3 million so as you go through here you can kind of get a good idea of what happened in history very similar to this chart is right next to it the inflation adjusted spending over all the simulations now a lot of people get confused when they first see this and they say why is it a flat line well because you chose $40,000 as a spending point and because you chose it to be just straight up inflation adjusted it's $40,000 every single year and you see that there you see it there if we choose other spending plans which are variable which we will go over in a bit they definitely go wild over here and we'll we'll talk about that okay there are a lot of numbers here and without getting too verbose I'm going to talk about the most important ones and then let you sort of figure it out after that I think the most important thing is this little section here right below the portfolio chart this is the success rate success on this tool is defined as you still having any money in your portfolio when the simulation is over or when you're gone you can have one dollar in your account and it will be a success if you look on this chart there are a couple of these this is the very last year of this portfolio it started off with a million dollars it's down to 50,000 but that person you know made it to the end that's a success so out of a possible 121 different 30 year periods in history this particular setup would have only failed five times this is essentially the definition of the 4% rule that is often cited the 4% rule is not 100 percent successful it's 95 successful which for most people is good enough you know I'm not going to tell you what success rate is is desirable you can look and read a lot about that topic one of the recent papers that sort of has blown my mind recently is Michael Kites a financial planner has written a paper that essentially says that if you do a Monte Carlo simulation which is different than this this is just historical data if you do a Monte Carlo simulation which is randomized data and you have a 50 success rate for retiring that if you did that every single year you'd be fine as long as it was 50% every single year now do I do I feel comfortable with 50% probably not but that's up to you up to you to decide some people need this to be 100% it's totally whatever however your brain works this number here is sort of what your portfolio was at retirement for this particular scenario because I retired right now on this on the scenario it's just exactly what you put in as an input for your portfolio beyond that this side of the this entire side the left side is all talking about portfolios this entire right side is talking about your spending I have all sorts of statistical numbers in here I've got the average ending portfolio I've got the yearly with average yearly withdrawals I've got the average total withdrawals I've got the median the standard deviation which is sort of the volatility I've got the highest value the lowest and then by popular demand I put in the lowest 10% in the lowest 5% just to see sort of what the slices of the statistical pie are here for these particular simulations you'll sort of notice that a lot of these numbers are the same that's not a bug when you have very flat data it tends to end up like that like I said before the spending is $40,000 flat every year inflation adjusted so these numbers are all going to be the same just like over here I'm not going to go over this entire chart for spending but I will say that I've split it up into quarters and the first five years these are sort of very these are both statistics that are very often cited in sort of financial planning articles or people even on the Bogle has forums will say you know the first five years is the most dangerous part of retiring so I've included sort of statistics on spending for these different time periods now someone asked me recently like do these quarters mean only the retirement period and the answer is yes if I wrote that I was retiring in 2031 so 10 years of pre-retirement simulation this information would only be including the retirement period I'll let you look at some of these later I don't want to get too verbose on this page the last thing I'll talk about is this I kind of hide this a little bit because it's long and most people don't want to see it but if you click this show button it brings up a extra chart now these are the single years where the portfolio was the lowest out of all the simulations it tells you what the portfolio was now if you look five of these failed so of course we have five zeros but look at this guy I didn't see it before but in starting here of 1967 in year 28 it got down to four thousand dollars actually this makes sense because right here starting year of 1967 the very next year it goes to zero these are just individual data points people like to see how low it goes before failure if you happen to have a simulation that has 100 percent success it's still interesting to look at how low it has gone in the past so you know keep this little section in mind if you're the kind of person that likes to look at the very edge cases of things I don't usually look at this very often but to each their own okay I've gone over this another basic functionality to note is if you look up here I have these two little tabs when you submit the form it doesn't refresh the page it opens a little tab here and puts the information here if I click back onto the inputs tab I see my same inputs that I had before this is just the regular default data now what you can do is you can do things like this the retirement end year is 2050 in this case what if it was 2055 I'm going to leave everything the same but what if it was a longer period of time what if I think I'm going to live till I'm 95 if you notice up at the top here it loaded a new tab called sim 2 this shows the statistics for what I just put in it failed another extra couple times because it's trying to eke out another five years of retirement but most importantly what I wanted to show you is that you can click between these tabs and essentially have a bunch of different simulations at a once whichever tab you were last on when you click on the inputs tab it will load the inputs for that simulation it will tell you that it loaded it and if you look this is the input that I used for sim tab two if I go back to one and then go back to the inputs it will load it from sim tab one which was 2050 it does that for everything on this page so it's kind of useful if you're sort of iterating through and trying to find a very specific time that you know how many years you can retire or things like that so this is a very useful feature I personally like it you can also hit this little red x which I hope is intuitive and it will make it go away so let's see one last thing I guess I missed this so I haven't gotten to do a lot of these yet but if you click this button open investigation options it opens this little extra spot down here once upon a time this was a very popular thing on the site and I can't remember if fire calc does something similar but essentially what this does if you click this little slide thing it's going to look for the maximum initial yearly spending for this amount of success this is actually a typo I should just delete this and just say for this amount of success so if you type in 95 here it's going to run the simulation many times to figure out what is the maximum number of spending here to get 95 success I unchecked it right before I hit enter let's go back this one takes a little bit longer because it actually runs multiple scenarios in a row so I would not worry too much about it spinning okay this looks a lot like the other simulations but down here I sort of write this little note just says investigation was complete then it tells you the maximum initial spending is 40 507 dollars exactly and it results in 9504 I would say that this is accurate within 10 dollars usually just the way that I'm iterating through the thing it makes it really inefficient to find it down to the one dollar I'd rather it spit out something very close so if you were to put this amount back in the inputs tab you know it would give you the same it should give you the same amount of success boom 9504 perfect I'm really hoping I can make it through this demo without displaying a bug in real time okay um I've gone over most of the basic stuff the spending plans I've yet to go over Jim or whoever else Keith I don't know if we want to take any questions now all right we're recording all right spending plans spending plans are my favorite part of this tool um honestly this is the thing that I mess with the most in terms of my own personal situation um and I think that it's a pretty undervalued thing um yeah I am not a I'm going to say this a lot I I work for a financial firm and I am not this is not part of their work and I know all the legal disclaimers I am not a financial planner however um I know that people do not spend the exact amount of money or same amount of money every year and modeling that is kind of unrealistic um ideally people are vacillating between you know a spending high and a spending low and they know roughly what their average spending is we can do that with these different variable spending plans um VPW was just asked about in the question and answer section so I'm going to go over that one first um so VPW is a essentially a spending plan or a withdrawal plan if you will that was come up that came up with on the Boglehead forums I can't remember the username off the top of my head I want to say was was it long invest anyway um one of the users came up with this and the premise of this is pretty simple um well the premise is simple but the calculations are complicated the premise is that this user would look at sort of the standard inflation adjusted simulations and they would see that a person who had a million dollars and spent roughly 40 000 a year would end up with a portfolio of six million dollars at the end and they thought you know what you know you can't take it with you I want to spend all the money how can I do that in a safe way um and the the models basically if you look here if you go back to inflation adjusted you can't just say I'm going to spend 60 000 a year um I gotta back this up you can't just say I'm going to spend 60 000 a year because if you spend a flat 60 000 a year it fails a ton um and it's just not sustainable you can't do that so he wanted to find a safer way to get as close to zero at the end of life as possible what he did was take a combination of um sort of writing his own annuity each year based on um the end point of your simulation excuse me talking too much um so he wrote out a sort of annuity formula and then based it on whatever future value you wanted so these two values are new values when you select vpw and this is sort of a rate of return um that the the aforementioned annuity is based on and this future value is essentially what you want your portfolio to be at the very end now this number is largely ignored in the vpw I believe it has some significance to the starting point but it very quickly gets changed along the way so I'm gonna just just in case I change some other things I'm gonna refresh this real quick to the default then I'm gonna switch it to vpw and then this is the default stuff I'm just gonna run the simulation and see what happens so you'll see that it says 100 success now that means that the portfolios were above zero when the person died now if you look at this chart on the left it looks a lot different than the other ones if you look everything is spiraling down to zero but it falls just short so the math behind this allows you to spend it down so that you essentially end up with less money at the end of retirement now if you look on the right here this is the first time you've seen the spending graph um become a variable graph now if you look here this might be sort of um you know sort of unfathomable this is a million dollar portfolio and at some point spending is getting to be like $200,000 in a year how can that be well I mean it follows um the market sort of gains for a given year and every single year it's calculating based on the current portfolio and how many years you have left to live and the expected return rate what it thinks you're gonna have at the end so it keeps increasing that um well it really just varies it up and down but so if you look here it goes up and down quite a bit but if you see here if you look at the average spending for this um entire simulation it is quite a bit above 4% but if you look on the left side of this chart the average ending portfolio is $85,000 this is as close to zero as we can get without failing before your end of life so this is an interesting example of something crowdsourced you know someone on the Bogleheads forum was writing the sort of thesis for this and you can find it out there it's still a very um I think it's a very trafficked thread um but that person came to me in like 2015 or 2014 and asked if I could make this happen as part of the website and I did um so it's kind of kind of an interesting way of thinking about spending um if you're willing to spend it all down um and before I move on to the next one one thing to note is you can change the future value um you know if you're planning on spending more than normal but you want more than just zero left at the end maybe you want to leave 200,000 to your kids or something um you can change this future value and it tries real hard to make things happen at 200,000 actually it seems to not have done that I told you I was trying to make it to the entire video without a bug but here we go um I'll have to check that out so if you look um some of these portfolios they actually dipped down to zero um it looks like they're failing right before the end so that's interesting anyway we'll move on to the next one all right so um when you have inflation adjusted of course this is the not inflation adjusted um scenario um essentially it takes your spending and never changes it um that means over time your actual uh value of your money is going to go down because inflation is going to be sort of reducing the value of it so if you run this really quick um you'll see on the right side here that inflation really changes things now what's what's interesting here is in the early 1800s there's quite a bit of deflation so even though you're not adjusting your spending here um your inflation adjusted value of that spending is going up but the vast majority of sort of modern times your the value of that money is going down quite a bit if you look at sort of the averages here um really you're averaging 30 you're averaging lower than 40 000 per year if you never adjust it and um I don't know if anyone's uh I can't remember from our poll who uh frequents early retirement dot org but it's like a common trope on early retirement that you don't want to be eating cat food in retirement so you know if your spending reduces to half of what it started you might be in trouble um okay i'm gonna go back what's next percent of portfolio um okay this is a pretty simple one on its surface but there's some nuances that a lot of people miss um so percent of portfolio pops up one new field called the yearly spending which is a percentage of the portfolio for that given year so when we talk about the four percent rule if you're not familiar which I hope you are if you're here but if you're not familiar for people out there the four percent rule is based on a study where um you know uh they had portfolios of people at retirement and starting at retirement their starting spending was four percent of that portfolio value then from then on out no matter what the portfolio did you just only adjusted your spending for inflation so every year wasn't four percent it was just the very beginning this particular spending plan percent of portfolio every single year it's going to take your portfolio value and then adjust your spending based on that if you start off with a million dollar portfolio and somehow it gets to be two million dollars well four percent of two million dollars is eighty thousand dollars and that's what it's going to set your spending to um so if you run this you'll see you on the right that the spending is wildly varied um you know if you're if you would have retired in the eighties um your spending would have gotten up to 157 thousand dollars by using this method um now this says here that it's a hundred percent success rate now that is true and that is true because no matter how high or how low your portfolio gets it's gonna only take out four percent so if you look here the spending in this year was only 15 thousand dollars that's because the portfolio dropped quite a bit um by definition if you're only ever taking four percent of the portfolio it's almost never going to go to zero um so how do we simulate these on a more realistic manner if we go back here i'm going to introduce this other section of the variable spending that is very important it's a concept of a spending floor and a spending ceiling so well i'm going to go back to the tab real quick if we look here this is trying we're trying to spend four percent of our portfolio now if you're trying to stay within the realm of 40 000 dollars inflation adjusted um spending uh you know you look at this chart and you say wow it's really i mean it doubles or triples sometimes and it goes down to you know major i'm eating cat food level um how do we fight against that we do that with spending floors and spending ceilings so the idea of a spending floor and i'm going to choose a defined spending floor value is that the simulation will never allow the spending to go below the floor so in this case this is a four percent rule um i personally like to think like hey i can take a ten percent haircut on my spending and be okay so i'm going to put the floor to be thirty six thousand dollars now if we just run this as is without doing the ceiling if you look on the right you'll see some very distinct flat bottoms um it never goes below the thirty six thousand dollars that we set and that's great however it also goes out of control in these other places now maybe you wanted to go out of control maybe you want to set a floor and when your portfolio goes crazy you can go crazy too that's great some people are trying to pay more attention to their average ending portfolio to make sure that they can pass money along for some sort of generational wealth so if we go back to the inputs again what we can do is we can rein in the spending by setting a ceiling now like the floor you might have guessed the ceiling makes it so the spending will never go above a certain amount so let's say we have a forty thousand dollar target um let's go crazy and say if in good years we can spend up to sixty thousand dollars that'd be great um and we're going to run that now uh this might look a little wilder because this is sort of to scale so when when the um when the values went up to 157,000 they looked a lot more squished um but now they're within uh you know 24,000 of each other so if you look there's some very distinct flat tops and flat bottoms it's bouncing around in between the spending floor and the spending ceiling um I love this feature um I think that it's one of the more realistic things that happens in retirement um most people will have sort of this idea in their head of what they want to spend and they'll have like a sort of you know belt tightening uh amount in their head that they don't want to go below and then on on good years maybe they'll spend more um this allows you to do that I really like this feature um so that's what that looks like um one of the things you can do is set the floor to be the value of your adjustments um so if your retirement is dependent on a lot of social security and pensions and say this person has 30,000 dollars worth of pensions and social security if you can set the floor to be that then um it's a quick and easy way to uh deal with the floor um I don't have that same ability for the ceiling because I think it's kind of silly to be able to select your adjustments as the ceiling um but maybe someone can uh convince me otherwise um but they're percentileo um wow it's 930 eastern already


Welcome to another great Friday afternoon here in Honolulu, Hawaii. I'm Stan Osserman, I'm the energy man here on Think Tech Hawaii. And we're going to talk all things energy. And today we're focused on something that's come up in conversations that I've had with several industry partners that we have out here. And that is when is more of this equipment that we're talking about with hydrogen and especially with hydrogen systems going to be more available like commercial off the shelf. And so today on our show we have a company called Gencel and Alex Ossido is our guest. And he's going to be talking to us about something that is commercially available and we could bring into the state, we could bring into actually any community and start using for backup power, for primary power and as part of systems of power. And so Alex, thanks for joining us today. And I appreciate you calling in all the way from Atlanta. And why don't you tell us a little bit about yourself first and then a little bit about Gencel. Sure. Thank you very much for the opportunity, Stan. So again, my name is Alex Ossado, I'm the director of sales for the company called Gencel Energy who happens to be an Israeli company, but I happen to manage all the Americas. And one thing that you should know about Gencel is that we really kind of look at the hydrogen technology and the way that we kind of build building blocks to provide a ultimate result. Now, one thing about the height of Gencel Energy is that the engineers that are actually working for Gencel are the same engineers that were working for the Russian space program to generate power out in space. So technically our fuel cell really is rocket science. Now, one of the way we built the solution is really designed to provide you two primary components. One of them is to have an extremely reliable piece of equipment that basically uses no mechanical components to generate power. And the second, which is to me is one of the most important items of the solution is related to the bidirectional communication aspect of the solution. When you think about risk, the risk really comes from the unknown. When you have applications where you're designed to power mission critical equipment and you don't know what's happening with the power source, that in itself creates a lot of risk and liability. So one of the things that we've designed is a solution that the system may is smart enough to know what it is. How it's supposed to be functioning. And if there's any deviation in performance, the system will automatically send you alerts in a variety of different ways. It can be from emails, text messages, if anyone is still using beepers, you'll get a beeper message. But the whole premise is that we want to make sure that we provide you with the operational intelligence that's necessary for you to make informed decisions about what to do and when to do it. That's great. And you know, that speaks right to what we wanted to focus in on the show. You know, people nowadays are, the consumer is pretty demanding in terms of the products that they get. They need to be reliable. They need to be efficient. They need to be affordable and they need to function consistently and, you know, for a fairly long period of time without being replaced. And then there has to be a good way to replace them and follow on without making a huge new investment or, you know, making a big change over in technology. It seems like GenCell's kind of got that upfront in their design package and how they put some things together now. You have a couple of pieces of equipment that are basically power supplies. And if I recall right, one of them is hydrogen based purely and the other one is ammonia based. Is that correct? That is correct, Stan. So we've designed basically two primary solutions. One of them is designed primarily for backup power, which is called our G5. And, you know, that in and of itself, it has a lot of value because in today, when you're thinking about the electrification that's happening, not only nationally, but you think about internationally where, you know, companies like Tesla or Nikola or any of the other companies that are really making a big effort in trying to eliminate the fossil fuel power generation process. I mean, you even look at the Mariah from Toyota, you know. Even vehicles, you're looking at big companies like Bloom Energy that are putting very large, you know, hydrogen fuel cell technology for large stationary power where you're actually generating megawatts worth of power. The electrification process in the world now is really becoming to be mainstream. And this is where I think we have a piece in that market where we're very specific in what we can apply. But the reason why we created our solution is because we know that there was a gap in the process when you're dealing with things like diesel generators or UPSS. They have limitations in terms of what they can actually provide over what period of time. So if you think of traditional UPS, the UPS will give you, typically depending even on the size, usually between, let's just say, 35 minutes to about a little over an hour, regardless of the size, because usually they're sized based on the load. But typically, they will give you the same kind of range in terms of energy outputs. The problem with the batteries is that, you know, they're also very finicky. Even though a lot of people or at least most of the industry look at batteries as a failsafe, it really isn't because between all the various variables that go into making sure that those batteries give you the outputs necessary and give you that confidence that you're going to have the ability of time really kind of goes to a lot of things like age or significantly impacts battery output, the maintenance associated with it, temperatures, and a couple other variables. So you may design a system, you know, when you first bought the system for eight hours, but over time, if you're not maintaining that system and you're not correctly managing that process, you really may only get five or six hours. And the problem is even though the indicator may show that it has 100% charge, typically you're only measuring the certain charge. So unless you take the specific gravity of each individual cell, in many cases you won't really know how much energy you have to work with and how much time you have to work with. So that's a limitation. If we focus on the diesel generator component, you're basically talking about a mechanism.  I'm not sure if it's connected. Let me interrupt for one second or two. Sure. You know, one thing that you mentioned that I think very few people really think about because they don't do much work with batteries, but you mentioned that a battery is a collection of cells and it's almost like a chain where the weakest cell is as strong as you're going to get. And a lot of the folks don't realize that when you have the current battery technology that we have, you may have hundreds of cells in your battery. It only takes one or two to start deteriorating abnormally because of a defect or something, and it drags down the whole performance of the whole battery. That's one thing. And the other thing is the batteries that would last you for days or weeks, and you talked about a half hour to an hour as a backup system, if you were going to go for over a day or into a week or a month, you couldn't afford all the batteries it would take to give you enough storage to maintain energy for that length of time. And I think that also lends well to what your technology brings to the market. Okay. That's absolutely correct, Stan. And it's not even just the expense associated to the number of batteries that you just described, but it's also the space. The space is another limiting factor in terms of battery technology.  I'm just saying that there's limitations to what those batteries can offer. Exactly. If you switch over directly to the power generation when you're using mechanical components, that also has a certain amount of limitations, because that's why most companies that are depending on power generation through diesel, gasoline, or propane, whatever the fuel happens to be, it's all mechanical. And the process is that they have to turn these systems on at least once a week or twice a month or whatever their protocol demands. But that's just to verify that the system will work when you're relying on so many mechanical components. So it's not just the generator. You've got the switch gear. You've got the auxiliary pumps for fuel. So any one of those components fail, the whole system shuts down and nothing happens. So that's just where we kind of created a hybrid solution where we incorporate both a, basically what we call the energy, which is kind of like a UPS, and also the power generation components. In this type of configuration, you get all the great benefits of fuel cells that if you, if you would mind going to the slide that I have, that I presented to you guys. This kind of gives you all the great, the next slide please. This will kind of give you some of the great attributes of tool cell technology. So considering that there's really no mechanical components internally to generate power, it's all a chemical reaction between hydrogen and oxygen. Literally, the maintenance is very low and very long, meaning that you only have to do it once a year. The second thing also is that you're not burning anything. So there's no emissions, there's no combustion, there's no noise. We're operating at about 52 to 55% efficient. If you also include the thermal energy that's part of this chemical reaction, you're up to about 87 to 90% efficient. So literally you have a great power of those things. You know how it's supposed to be functioning, and if there's any deviation in performance, the system will automatically start sending you alerts so that you know what to do and when to do it. And one of the things that you have on that list there is the harmonics and the balancing. You know, as we get more and more into computers and complex communication systems that are really sensitive to the quality of power that you're getting, I haven't found anything that can quite replicate the clean power signal-wise that comes off of a fuel cell compared to a spinning generator or some kind. And I think that's really another point that's lost.  You know unless you're looking at everything that impacts the life cycle of that equipment, whether it's maintenance or whether it's a critical failure or meantime between failure, cost of materials, source of materials, you really have to do a critical analysis of the equipment and the system. And the more I look at hydrogen and hydrogen-based systems, I just see much, much better picture for hydrogen than anything that's currently out there. So, Sam, so you are absolutely correct. Your point about the total harmonics, the power factor, those are very critical to many systems that are depending on the electronics and making sure that you have a steady stream of power because any type of heat or stack is going to definitely devastate the process to where it can even shut down certain systems. And any type of disruption in power and significantly both operational and financially devastate the operation. So if you would mind going to the next slide. We actually have built a solution specifically designed for utilities, oddly enough. So, going back to what we were discussing earlier about battery technology, you know, when you're dealing about the transmission and distribution of power from the utility, they have distribution or substations every so often. And these substations have very critical equipment internally that has to be able to be remotely turned on and off. And when you're thinking about some of these systems, the only option at the moment would be a battery. So most of these substations will actually use a battery bank to support this application. Well, there was a company in Southern California that had a situation that in 2014, they lost power and this kind of went across state lines. So to restore power, it was very complicated because they were banking on eight hours worth of runtime in terms of time flexibility to restore power. And by the time they came around to doing it, they quickly realized that they really were only getting about five to six hours, maybe seven hours. It did not reach the eight hours. And as I mentioned earlier, one of the reasons is because of maintenance, it could be age, it could be any number of things, but the whole process is that they were not able to restore power. So that created a huge issue to where they contacted us. And now what we're doing is we're actually providing electrical energy to support a substation from a utility to be able to support the restoration of power and give them a little bit more range. If we can design a system to operate for eight hours, we can operate for 16 hours, we can operate for days or even months and years at a time. Just depending on what the customer requires is how we size the system. And when I say we size the system, it's really contingent on how much fuel we have on premise. That really, just like any other power generation, depending on how much fuel you have on hand, ultimately it's going to give you the ability to run for that expected amount of time. The difference is that we can quickly understand exactly how much power we have and our systems are very efficient. So when you compare it to traditional fuels, we actually require less fuel to operate for longer periods of time. That's a great point. And we're going to take a quick break here. And when we come back, I actually want to talk about the connection between hydrogen and ammonia and why your systems, why you have different systems that run off different fuels. We'll be back in 60 seconds. Hi, guys. I'm your host, Lillian Cumick from Lillian's Vegan World. I come to you live every second Friday from 3 p.m. And this is the show where I talk about the plant-based lifestyle and veganism. So we go through recipes, some upcoming events, information about health regarding your health, and just some ideas on how you can have a better lifestyle, eat healthier and have fun at the same time. So do join me. I look forward to seeing you and aloha. Aloha. I'm your host, Sharon Thomas Yarbrough of Sister Power here at ThinkTech of IE. And Sister Power is all about motivating, empowering, educating, and inspiring all people. And we have various subjects here. Sister Power is here at ThinkTech every other Thursday at 4 p.m. Again, my name is Sharon Thomas Yarbrough, host of Sister Power. We look forward to seeing you. If you have any questions, feel free to contact me at sistersandpowerandcavaii at gmail.com. Look forward to chatting with you soon. Aloha. Hey, welcome back to Stand Energy Man on my lunch hour, of course. We're talking to Alex Oceto from GenCell. And we left off talking a little bit about the different kind of fuels that his equipment uses, and we're focusing on hydrogen. Number one, because that's my favorite subject. But his equipment also, they have a series of equipment that runs off ammonia. And I wanted to just make a couple points about why there's a connection between ammonia and hydrogen. Well, ammonia is NH3, which is one nitrogen atom and three hydrogen atoms. And ammonia is actually one of the best ways to transport high density of hydrogen at ambient temperature and ambient pressure. And it's easier to do because it's a liquid fuel. So when you talk about liquid hydrogen, you're talking about a super cold, super energy dense liquid, but it's also a lot harder to deal with. Hydrogen, I mean, ammonia is also really important because if you have an agricultural community that needs fertilizer, ammonia is also a big part of doing fertilizer for your agricultural community around you. So when you start looking at how these things synergize together, I think it's really important to recognize that when you make hydrogen out of an electrolyzer, you also get oxygen for welding and medical purposes. And then when you're dealing with ammonia, which is another great energy carrier for hydrogen, you're dealing with fertilizers and things of agriculture community. So Alex, I'll turn it back over to you to talk about your system. Great, thank you, Sam. So yeah, so one of the key things that Gensel really kind of evaluated in terms of our solution and how we would be a little bit different from the common companies out there that are generating power-reusing fuel cells is that today either you're using straight hydrogen or you're using some sort of fossil fuel, whether it's methanol, natural gas, or something of that sort. But those systems typically are not 100% green because you're still creating CO2 and you're still creating some sort of an emission. So exactly what you mentioned, Sam, is that the alternative to that, ammonia is highly rich with the energy density of ammonia, which gives us a great way to extract out the hydrogen and be able to feed our fuel cell to generate clean electrical power. So one of the things that we did is we built a cracker that basically what we're doing is taking in hydrosimonia, which as you mentioned is NH3, we put it through our patent technology cracker, we split the molecule between the hydrogen and the nitrogen, and then that feeds our fuel cell. And the reason why this type of technology is revolutionary in the fuel cell environment is because this really kind of gives us the ability to generate power for long periods of time. So if you think about the fuel cell, all the great attributes that we just spoke about initially, and you're thinking about how we're going to use this technology and how we're going to get the fuel, because obviously you're not going to go to your neighborhood gas station and pick up some ammonia or some hydrogen, which means that you have to bring the energy to you. Now part of that is exactly what you mentioned, Stan, is that from a transportation and logistical perspective, bringing in hydrosimonia is very easy because the supply chain is already there, the safety protocol is already there. It's already used in a lot of different types of applications from you mentioned agriculture, that's one, refrigeration, cosmetics, water purification. So every single country in the world has it, whether they produce it or import it, but they have it. So this makes it an ideal solution for us to be able to generate power, sell our equipment, use all the great benefits around the world from a solution that is very easy to extract and also operate for long periods of time. So let me give you a little bit of a scale. If you think about ammonia, we bring in one tank of ammonia, which is approximately, one time is approximately 400 gallons of fuel. And that 400 gallons will give us the ability to operate for 24 hours a day, seven days a week for 30 days. That's revolutionary, nobody else is doing this. Now you compare that to diesel fuel, at least on the mainland you're talking about approximately $3 a gallon. And if you compare that to a diesel generator that's burning between one gallon and two gallons an hour depending on the size, we got this up the load as well because, you know, even if you have a very low consumption rate, your generator is still going to be operating at the same level as far as consumption because regardless of the load, that engine has to turn, you know, that RPM has to continue turning. But your consumption really doesn't change. So if you think about one gallon to two gallons an hour, trying to operate for 400 gallons, you're only going to be able to operate for about 15 to 16 days max. Now you multiply that times $3 a gallon, now you're talking about about, I would say, close to between $12 and $1500 for 15 days. Right. And a diesel engine has a sweet spot. Like you say, if you're not operating in that sweet spot, you're also becoming very inefficient and you've got to keep it running at a very inefficient level because you don't need all the power that it's generating. So you're actually wasting a lot of fuel that you don't need to and the equipment that you have can kind of load follow a lot better. Correct. Dynamically, it adjusts the consumption rate or the conversion rate of hydrogen to electrical energy depending on the load required at that specific amount of time. Going back to what I was saying in terms of the cost savings that we're talking about, so if you think about $12 to $1500 for 400 gallons or a ton, which fuels actually a little bit more, weighs a little bit more, so we're only talking about 400 gallons from the ammonia side in the U.S., at least on the mainland, you're talking about a $500 to $550 a ton. You're talking about a $500 to $550 a ton of fuel for 30 days compared to $12 to $1500 for 15 days. So there's a drastic difference in price in this woman's optics perspective compared not even factoring anything else. You know, this is where you can really significantly start saying that, you know, from a full cost of ownership, we are superior to a fuel country. Okay. You're looking at not only the fuel, but then you're also talking about the maintenance. Think about all the maintenance you have to do on mechanical components compared to a computer that basically is generating power because there's no mechanical components necessary to generate the power. Yeah, also, if you're not used to dealing with diesel engines, and I am, as backup power for our military operations, you have to sit there and actually clean the fuel every so often. You can't just leave it sitting in the tank, so that's another added maintenance cost that comes into maintaining, in addition to running them regularly so you know they work. Correct, correct. Let's go back to the slides. Yeah, I mean, so one of the other things, if I may stand. Okay. So we talked about the ammonia, but one of the other things that we're using as well, because again, as I mentioned, we're a technology that we use to building blocks to provide a result, right? Right. So not always do we say that fuel cell is the end-all, end-all. So one of the things that we also have complementary technology to where we can still connect to solar panels and batteries where necessary, whether it's applicable, where you still may want to use solar panels because it may be much cheaper, but you still need to have a fail-stage solution to generate that power, because again, we're talking about mission-critical equipment. Right. So if you use solar panels, that's great, but Mother Nature is never going to give you 100% of the energy needed all the time whenever you need it. So you have to have something on the backside as a redundant system that you can rely on, that you can actually generate power on demand. So that's one application. So we can always connect to that. The second one is that we can also utilize electrolyzers using water to work in similar to the way we were splitting the molecule between the ammonia molecule and the nitrogen molecule. We were splitting the hydrogen molecule and the oxygen molecule to generate electrical power, or I'm sorry, getting the hydrogen to feed the fuel cell to generate electrical power. So you can also use water as the means to generate clean electrical power by simply using electrolysis, which really doesn't matter where we get the electrical energy to support the application. We're coming from off-grid applications, from the utility, use saving at night, converting the water to hydrogen, storing it on tanks, and then using it today to kind of help support low-saving applications or strategies. Great. Well, actually we've got about one minute left, so I'm going to turn it all over to you because I know you've got lots to tell us. I don't want to hold you back. Well, thank you. That's pretty much what I wanted to let every of your audience know is that, you know, GenSo is a company that really kind of thinks outside of the box in terms of, you know, our main goal is to give you a result, and we wanted to make sure that we're using all available technology to make sure that we're giving you the most robust, most resilient, and more importantly, making sure that we're giving you the tools necessary to manage your operation more efficiently and cost-effectively. Great. We'll have Robert throw up that last slide on the screen there and we'll get at it. It's kind of a cool slide. It kind of says it all, you know? I mean, when there's no power around you, you're going to keep them running. So, Alex, I want to thank you very much for being on the show with us again. This is your second time on the show. I'm sure we're going to have you back again, and I'd like to actually talk more about, you know, what's going on in the future and how you folks see the future happening with your technology and your industry, which is the energy industry and using hydrogen and ammonia to create some power. But thanks again for being on the show and we'll get you back in a couple more months and get up to date with you. Thank you very much, Stena. I look forward to it. We have some very exciting things to share with you. Great. And that's going to do it for Stand Energy Man this Friday. Thank you for being a guest and thank you for being an audience here and watching Stand Energy Man. Thanks to Robert and Cindy here in the studio and we got a new production. Haley, the production, new production, you're all in the room there running everything, running all the background stuff. Haley, good to have you on board. Thanks for doing such a great job today. Until next Friday, m'Allah.


Check out how I just made this dano spicy chipotle chicken sandwich. Cut in half just like that. Put it in saran wrap and pound it out thin. Who needs a meat tenderizer when you got one right here? A little bit of pickle juice. A little bit of buttermilk. Then we're gonna add in some context of canned Kentucky Bourbon hot sauce.  Hit it with a pop, pop, pop, pop, pop, pop. Get this good and mixed up. We're gonna let this sit for about an hour. Heat up some peanut oil. Alright, it's been one hour and now you get some flour. And we're gonna get some dano's hot chipotle. Take the chicken, throw it into our flour, give it a toss, and just like that, it's ready to go into the fire. Now that our oil is at 350, we fire our chicken. It's been about three minutes. We're gonna give it a little flip and let that continue to cook. Now we got our chicken out, set to the side. Now we're gonna make us a spicy sauce. Let's start out with ranch. Big ol' squeeze of sriracha. And now time for the main ingredient, the dano's hot chipotle. Pop, pop, pop, pop, pop, pop, pop, pop. Give that a mix. Now you need your brioche bun. A bunch of the spicy ranch we just made. Two slices of dill pickle. Two slices of fried chicken. And last thing, top it with your sauce. Bun on top. And that right there, my friends. Now we just made a dano's spicy chicken sandwich. Let's get a bite. A little heavy on the sauce. But that is dantastic, my friends. Yum, yum, get you some. Sizzle sizzle


Hi everyone, my name is David Gray and I'm an associate software engineer at Red Hat working on the OpenShift Performance and Latency Sensitive Application Platform team. Hello, I'm Kevin Pouget. I'm a senior software engineer in the same team as David. In this talk, we are going to present the tool set we use to deploy parallel scientific HPC workload on OpenShift container platform. And we'll introduce the performance results we obtain in a proof-of-concept environment using these tools. First, we'll give a bit of background about OpenShift and why you'd want to use it as a platform on which to run scientific HPC applications. Then we'll go over the setup of some useful tools and OpenShift features which can be used when deploying workloads like these, including how to enable the Lester file system, CFFS, the MPI operator, and Maltes CNI. Then we'll go over some of the performance results we've collected so far from the scientific applications Gromax and SpecFam 3D Globe, and some MPI micro-benchmarks we've run as well. The first to give some context, why are we testing high-performance scientific applications on OpenShift? With the emergence of big data and machine learning, scientific computing is becoming a more integral part of enterprise computing. Organizations are running more and more HPC-like workloads to create value for their business, and this change coincides with the rise of containers and Kubernetes for deploying enterprise applications. Increasingly, enterprise applications are deployed in the cloud and in the data center as containers and microservices. In the HPC community, containers and public cloud are also catching on. Containers are really helpful for packaging complex applications along with their dependencies, so that they can be run in many different environments. For these containers to be useful, you need a platform to run them on, like Kubernetes. OpenShift is an enterprise-ready Kubernetes platform for the hybrid cloud, and this talk aims to show how HPC applications can be effectively deployed on OpenShift. Both Gromax and SpecFam are scientific workloads which support multi-threading via OpenMP and parallelization over multiple nodes with MPI. Gromax is a package for running molecular dynamics simulations. With Gromax, we benchmarked with the popular water dataset which simulates hundreds of thousands of water molecules. Typically, Gromax with MPI scales well up to about 20 nodes or so, but beyond that, high-speed networking with technologies like InfiniBand becomes a must. SpecFam, on the other hand, is a geophysics code that simulates earthquakes and seismic wave propagation in the Earth's crust. It's a reference benchmark for supercomputers thanks to its good-scaling capabilities. Although it will not be discussed much in this lightning talk, both of these workloads can be accelerated by GPUs. On OpenShift, this can be enabled using the NVIDIA GPU operator, which is based off of the special resource operator, which will be discussed later in this talk. To run Gromax and SpecFam 3D Globe, we used several operators, including the MPI operator, special resource operator, and more.  In short, an operator is an extension to Kubernetes. It's a method of packaging, deploying, and managing an application, which is deployed on Kubernetes and managed using the Kubernetes API. Operators enable the creation of new object types through custom resource definitions, and they manage those custom resources. One example in this talk would be the MPI operator, which introduces the concept of an MPI job resource and manages MPI job objects when they are created by the user. An operator works as a software control loop to set and maintain desired state of certain Kubernetes objects. For the purposes of running parallel multi-node HVC software on OpenShift, a few existing operators really simplify the process. Some of the testing we have done with Gromax used Lustre as the shared file system to store the input data for the Gromax containers. For these experiments, we used an OpenShift cluster running on Amazon Web Services, because AWS has a Lustre file system service called Amazon FSX for Lustre. FSX for Lustre allows for Lustre file systems to be created and linked to an S3 bucket. There is a container storage interface driver for FSX for Lustre, which really simplifies the use of AWS FSX Lustre file systems in Kubernetes. Container storage interface, or CSI, is a standard for exposing arbitrary block and file storage systems to containers. Although this CSI driver is specifically written to run on AWS, a similar approach could be used for an on-premise Lustre file system instance. Normally, this CSI driver would require every node to have the Lustre client kernel modules installed on the host, but we were able to deploy the kernel modules with driver containers. Driver containers are being used increasingly in cloud-native environments to deploy kernel modules or to enable special hardware drivers. Driver containers are especially useful for pure container operating systems like Red Hat Enterprise Linux Core OS, which is the default OS for OpenShift nodes. We are using the special resource operator, or SRO, to deploy the Lustre client Kmod driver container and the AWS FSX CSI driver for Lustre. We've created a recipe for SRO to manage the deployment of this whole stack. This can be found on SRO GitHub page. The benchmark results we will share at the end of this talk are from a bare-metal cluster not running on AWS, and we didn't use Lustre in this environment, but I wanted to highlight what we've done to create an SRO recipe for Lustre to show how Lustre or a similar storage system can be enabled in OpenShift. Most of the benchmarking experiments we've done with the scientific workloads was on an on-premise bare-metal OpenShift cluster. In this environment, instead of using Lustre, we used Ceph FS, an OpenShift container storage running in external mode. OpenShift container storage is a software-defined storage integrated with and optimized for OpenShift container platform. It is built on Red Hat Ceph storage and can be deployed on-premise or in the public cloud. In our tests, we ran OCS in external mode so that we could use a Ceph FS file system which was running external to the OpenShift cluster. This way, we're able to use the same file system for bare-metal baseline benchmarks and our benchmarking on OpenShift to compare apples to apples. Alternatively, OCS can run Ceph on the OpenShift cluster. I will note here that the performance of the workloads we are running, Gromax and specs of 3D globe is not dependent on the file system performance, and benchmarking the file system itself was not a part of this effort. We aim to show how Lustre can be used on OpenShift because of its popularity at HPC sites. Additionally, Ceph offers an alternative, scalable, flexible, and reliable storage solution, which is easy to use on OpenShift thanks to OCS. The message-passing interface, or MPI, is a specification for message-passing libraries, and MPI libraries are the de facto standard for writing parallel message-passing programs. Many HPC applications, including Gromax and specfm, are written so that they can be compiled to run across multiple nodes using MPI. The MPI operator enables the easy deployment of MPI programs on Kubernetes and OpenShift. The MPI operator came out of the Kubeflow project and was originally created for running distributed ML model training. The operator defines the MPI job custom resource. MPI jobs define the MPI run command, the hardware resources required for the worker pods, and the number of worker pods. Once an MPI job is created, the operator creates the worker and launcher pods and configures them to be able to communicate using the MPI library installed in the containers. One of the things we will discuss as we go through the performance results is using Maltis. Maltis is an open-source project that enables Kubernetes pods to attach to multiple networks. It is a container networking interface plugin that calls other CNI plugins. For running MPI applications, we use it to bypass the software-defined overlay network in OpenShift. For certain types of network traffic, the SDN can have some performance overhead due to encapsulation and open flow rules. In general, throughput is not diminished much, but latency can be affected. Another way to bypass the SDN is to give the worker pods access to the host's network namespace. However, this has security implications as it could be used to snoop on network activity of other pods on the same node if the pod were somehow compromised. Because of this, using Maltis is preferred. And in our tests, it gave us the same performance improvement as using the host network. Now I will hand it over to Kevin for the next part of the talk. Thank you, David. So for this second part of the talk, I will present the results of the benchmarking that we performed on our local cluster. First of all, let me introduce our test environment. The OpenShift cluster was composed of 37 nodes, three masters, two for infrastructure management, like telemetry, routing, storage operators, and 32 worker nodes. And for getting the reference time, we converted these 32 worker nodes to run real 8.2 bare metal. Each of the nodes have two high-speed network cards, one used by OpenShift for the cluster communication, and one available for our MPI communications. And the nodes have four cores. We didn't use hyper-threading and 62 gigs of run. In this slide, we can see the result of the MPI microbenchmark, where we compare three different networks of OpenShift. The default software network, software-defined network in blue, multis, in red, and the host network in green. These last two options use a dedicated network for MPI communications. And as David mentioned, the host network isn't a secure option, so it's just here for reference. And last, we show in black the bare metal benchmark, again for reference. And the graph shows the point-to-point latency, band and bandwidth, and MPI all-to-all latency with 20 nodes. And the main thing to note here is that multis in red has performance right on par with bare metal, not clearly visible in the plot because all the bare metal lines overlap. And OpenShift software-defined network has degraded performance and more variability in the measurements, which was expected because it's not a dedicated network. Now let me present the result of specfam benchmarking. In this slide, we see the performance of the cluster for a fixed-program size. This program took 55 minutes to execute on four machines and only eight minutes on 32 machines, as we can see in the time graph. For four machines, the slowdown of OpenShift was only 2%, and it grew up to 12% when OpenShift overhead took over the extra computing power of the 32 nodes. But on the strong scaling chart, we can see that the efficiency remains very good, not far below the bare metal reference and the linear time. And we can also remark that the different OpenShift network have barely no effect on specfam results. And in this second slide, we see the performance result for multiple program sizes with most of the execution time between one and three hours. We can see clearly that when we feed bigger problems to the cluster, we get back very close to the bare metal reference with less than 6% of overhead on most of the runs. And for problems big enough, they supply up to 32 nodes, including the super linear efficiency. And I will let David continue with GROMAC results. Thanks, Kevin. The graphs here show the average and standard deviation of five or more runs of GROMACs. The performance measures the hours of computation to simulate one nanosecond, so lower is better. On the left is results of running GROMACs at different node counts with multis. The green line here reflects the performance while running the GROMACs pods on worker nodes, which are also running on some cluster infrastructure pods. This includes pods such as monitoring the cluster ingress controller and logging. The rest of the results we gathered are with these pods isolated to two specific infrastructure nodes to prevent these pods from using any CPU cycles during the benchmark run, which improves performance and reduces variability. The graph on the right shows all of the OpenShift results compared with bare metal. As we can see, the result using the SDN for all of the network communications is the outlier here. To get good performance on more than two nodes, bypassing the SDN appears to be crucial, and using multis to do this gives performance just about on par with bare metal. This graph shows the parallel efficiency of scaling up GROMACs to higher node counts. As the previous graph showed, we can improve throughput by adding more nodes, but the main thing to show here is that we can get similarly strong scaling with OpenShift as with bare metal. In conclusion, these results show that OpenShift container platform can be effectively used to deploy scientific containerized applications. By using multis to bypass the software defined networking layers on a secondary network, we can get performance that is within a few percent of bare metal. Additionally, isolating some of the OpenShift infrastructure pods to specific infrastructure nodes can also help to reduce overhead. We have more results of running these workloads on OpenShift than we were able to fit into this lightning talk, but we have a couple of blog posts for the OpenShift blog on this topic, so keep your eye out for these on blog.openshift.com for a deeper dive. Thanks for listening.