The American Trail, the American Trail, blazed in blood, defended in blood. Chapter 6, The Golden Ocean. It seems to me I can still see the covered wagons. The way they rolled along the Cumberland Road, head and waist. Only when me and Kathy crossed into Illinois, we went alone, me driving the two horses, and Kathy sitting beside me on our wagon. Just the two of us. Me and Kathy used to have a piece of land in the Virginia Valley. When we decided to move west, I saddled a horse, went around saying goodbye to my neighbors. There was Cyrus McCormick.  That was where Cyrus spent most of his time, bending pieces of iron into all kinds of square shapes.  You'll see. Sit down, Jeff. Use that barrel. I reckon I'm not staying, Cyrus. Well, there's cider in the pitch here. Help yourself. You got time for that, anyhow. Jeff, I think I just about got this machine figured out. I don't ever do a man's work, even if you do get it to run. Do the work of 20 men in a quarter of the time. Jeff, there's a need for this machine. I don't see no need for a listen. You know when my folks first came to this country, 1743, going on 90 years ago. Now, what about it? We're still plowing and reaping just the way they did when they first came here. Well, what's wrong with that? I'll tell you something else. We're still plowing and reaping the way people did 4,000 years ago. There can't be much wrong with it. Jeff, fellas, like you make me tired. Farming ought to be made much easier and quicker. A man who's worth his salt doesn't want things made easier for him. Jeff, when I've got this machine reaper working, I wouldn't sell you one if you begged me for it. Cyrus, you'd have to travel a long way to sell me one. Me and Kathy, we're leaving here. Leaving? Where are you going? Out west. Virginia's fine, Cyrus. But it ain't big enough for me. A few days later, me and Kathy set off. I can't even remember all the places we went. Ohio, Indiana, some states that only had a handful of people. No roads, hardly anything you could call a town. Just every once in a while, a little cluster of cabins on all that land. Then we moved even further west into Illinois. Nothing but planes. Planes. When are we going to see some people? I'll forget what people look like. Then one night, we drove into a big covered wagon camp, maybe a couple of hundred settlers. New England from Europe. Right nice bunch. I'm Fred Tucker, wagon master of this train. Well, howdy, Fred. I'm Jeff Baker. Hi, my wife, Kathy. Oh, man, it's a pleasure. Oh, I could cry. I'm happy to see people again. Well, I bet you could, ma'am. You people staying here? Going on. We don't have to go any further. Just beyond that ridge under. No further than that. Yeah. Yeah, you see where I'm pointing? I see. Some of us rode over there this morning. Man, you should see the soil. Jeff, we can grow enough food in that country to feed all 13 million people in the United States. Ragon, this is where me and my wife are going to stay, then. It was the middle of summer, and we moved in on the Great Plains. We staked out land, built log cabins, and shelters for the horses and cattle. But by that time, winter sat in. It was a hard winter. A long one. Snow drifting across the plains. Fourteen, sometimes 20-foot deep. Sound of cold winds sweeping down from the continental divide, whipping across the Great Plains. Neighbors living five miles away. We can't even get to see them. It's lonesome, all right. But wait'll spring. We'll see everybody then. No, Jeff. That's no. It's going to melt one of these days. It'll soak into that good soil. And then it came. The spring. The real hot weather hadn't set in yet. It was just right for hard work. The two horses had grown fast. They wanted to work. They wanted to be hitched up to the plow. They wanted to feel the soil under their feet. Hey! Pull that plow there! Hey, a fire! Man's work! This is real man's work!  Hot like a desert. The wheat came up. We thought the sun would burn it up, but it was good wheat, strong. The soil was full of moisture in that winter snow. Cathy, you ever see a sight like this before? Oh, it's beautiful, Jeff. Real beautiful. You ride all through this country, days and days, miles and miles, waving corn and wheat. It's like looking at a golden ocean. Far as you can see. It all belongs to us. But, Jeff, you'll have to let me help you with the harvest. Sure could use some help. I'll hate to see you working in the fields, Cathy. Well, you can't do it alone. You just couldn't do it, Jeff. You have to know to the day when to start reaping. It's a man's work, all right. Been an overall day, been in and swinging that side. Cathy following me, bundling up the wheat as I cut it. Oh, look at the color of this wheat, Cathy. The size of it! Jeff, I can't stand the sun. Jeff, I feel like I'm going to faint. I got a whole of you. I feel so dizzy from the sun. I got a whole of you. I'm going to carry you back to the cabin. Oh, I want to help you. I ought to help you. No, no, Cathy. Won't let you work in the field no more. Never again. It's too much for you. This is too much. When Cathy felt a little better, I went back to the field. Out there, just me alone, in the middle of all that wheat. With a job of reaping to do before something happened, I... I lost my harvest. I'll never forget that day. First sound of thunder, me looking up at the sky, seeing it getting dark, and then darker until it was near black.  I get the pit of my stomach. The rain coming. Big rain.  No. No. Give me a week. Just one week. Let me cut my harvest. Don't ruin it for me. Just give me a little more time. A mud sheet of water. Like somebody had opened up the sky. He just stood there. Seemed like my wheat wasn't there no more. All flattened out. Oh, yeah. It's all gone. A harvest. Gone. Ruined. The only way to have a little more time if only I could have had that sweet cut before it happened. Hot sun came out next morning, but I had no harvest left. Me and Kathy, we just stood outside the cabin looking. I was so happy here. I was content. Now I'm scared again. Jeff, I want to leave. We could get through winter, maybe next year. Next year. Next year it'll be something else. Hey, Kathy, we need some help, but we could only get it. Where are you going to get anybody to help you? You're right. I know you're right, Kathy. Only we're still pretty young. Well, we won't be much longer. We'll be old before our time. Jeff, let's go back. Anywhere. I don't care where. Just so we don't stay here. It's too big for us here. It's too much for one man. We started getting ready to leave. Me and Kathy, a lot more of the settlers. Then one morning, Fred Tucker rode over. Jeff, Kathy, saddle up and come with me. Come on. There's a fellow down by the river. A fellow from Virginia Valley. He's got something to show us. I'm no crazy man.  You don't have to go back and you don't need to hire more help. It was him. Cyrus McCormick. A few years older now, not so much of a dreamer as when we last saw him, but real business like standing there beside the river.  That machine was the queerest looking contraption any of us had ever set eyes on. Here it is, my friends, the McCormick Reaper. Takes only one man and one horse to run it. And it does the work of 20 men and half the time. Cyrus, you can't prove a thing around here. We ain't got no wheat to cut. I'm not here to prove anything, Jeff. I'm here to take orders. Just come from Ohio and Indiana. It was harvest time. I demonstrated this same machine to a lot of farmers back there. They saw how it worked. Did you tell your machines to them, Mr. McCormick? I took orders, promised delivery and time for next year's harvest. Cyrus, how would we pay for one even if we stayed on here? Well, you buy on credit, Kathy. You pay later. It'll be you I'll sell one to, Kathy. I told Jeff years ago I wouldn't sell him one. Well, Cyrus, that ain't no way to remember an old friend. Look, I was just telling Kathy it don't make sense for a man to break his back when he can get a machine to do his work for him. It don't make no sense at all. Cyrus, we'll buy one. Kathy, wait until next year. Wait until you see us plowing this land and waiting for harvest time. Wait until you see me sitting up there in that seat like I was a king driving my own reaper machine. Like he was a king driving his own reaper machine. As far as he could see the yellow weave waving in the sun and him, king of it all the golden ocean. Yes, there on the great plains the McCormick reaper met the need of the day brought to farmers an end to drudgery. A century and a quarter of progress has passed since then. Today, tractors replace horses. Machines plow the soil, reap the harvest, bind it, and stretch the wheat. The great plains of America have become the bread basket of the free world. Out of this very heart of America thunder freight cars laden with grain enough for our domestic needs, enough for foreign trade as American wheat flows to the four corners of the earth. The product of the golden ocean. This has been the sixth chapter in the story of the American nation brought to you by the ladies' auxiliary to the veterans of foreign wars. Next week, another story to make you proud of this great country of ours as we follow the American trail.


India and Singapore ties are rooted in strong commercial, cultural and people-to-people links. In recent years, collaboration in digital domain has created new possibilities for the two countries. In many areas of India's development priorities, Singapore is a key partner. Smart cities, urban solutions, financial sector, skills development, ports, logistics, aviation and industrial power. And now we are building new partnerships for the digital world. The Prime Minister and I also agreed that fintech can be another key pillar of our strategic partnership. India is a significant market for Singapore fintech companies while Singapore, which is a global fintech hub, can be a gateway for India to enter the Southeast Asian market. Since 2014, under the visionary leadership of Prime Minister Narendra Modi, digital payment landscape in India has been revolutionised. India now accounts for around 40% of real-time digital payments globally, which is primarily driven by UPI. UPI backed by state-of-the-art technology enables fast and secure digital transactions even for small amounts. PayNow in Singapore is a foremost payments platform that lets its users send and receive funds quickly using just the mobile number. The launch of UPI, Rupi and Bheem enabled Remiton Sab by Prime Minister Modi in Singapore in 2018, underlining the mutual spirit of breaking fresh grounds in India-Singapore partnership. Today, another milestone is being achieved as UPI and PayNow partner together for the benefit of both Indians and Singaporeans. With the help of UPI and PayNow Singapore, our people can now transact money with their loved ones in real-time at low cost. This has been possible thanks to collaboration between Reserve Bank of India and Monetary Authority of Singapore. Let us see how this works. Let's try sending 100 Singapore dollars from India to Singapore. Simply log into your bank's mobile app or net banking. Go to foreign outward Remitons and agree with the terms. Enter the beneficiary's mobile number or VPA. Upon validation, proceed for transfer. Enter the amount. Check the summary and exchange rates and confirm with an OTP or UPI PIN based on your application's requirement. And you are done. A success screen will get displayed. So quick, right? Now let's try sending 10,000 Indian rupees from Singapore to India. Simply log into your PayNow participating members' mobile app. Go to overseas transfer. Select UPI and enter UPI ID. Upon validation, enter the amount. You can see the Singapore dollar's equivalent amount and we are done with an instant confirmation of the transfer. Wow! That was fast. Presenting UPI and PayNow Singapore, making cross-border payments real-time and low-cost.


Welcome to the 2017 NCHA Lucas Oil Superstakes. I'm here with the winner of the amateur four-year-old final, Kayden Rutherford, who rode fantastic one and marked a 2-15. Congratulations to you, Kayden. Thank you very much. Thank you. How does it feel to have this win under your belt? It's pretty special. It's been a dream come true. Everyone I've looked up to has won in this pin and done well in this pin. I just wanted to go out there and have a clean run and just happened to be my day. What was your game plan? Today I just wanted to get drove out far and just be clean and accurate and get this horse through a run. Cody's going to show him on Thursday in the open and I just wanted to keep him together for Cody to show him and not do anything too crazy out there. So describe your run to us as you went through it. First cow we cut was a slick yellow cow and she was a little tough there to start. She kind of got me a little flat in the corners a little bit. It was trying a little hard. I didn't really get driven up as far as I wanted to. But we held her and it was all good win there for our second cow and I cut a black calf and she let me kind of drive her up pretty far. And she was awesome. She kind of helped make my run. And then we finished up on a little black baldy that Cody kind of pointed me out. It was right there on top. And she just helped finish the run and just helped get through it really. Overall it seemed to be a fairly low scoring final. Were you nervous with the 215 because you were in the first set. Yeah I had a good draws fourth horse in the first set and I mean the cows were really tough all morning long. And I was I mean I wasn't even in here for the second set. I didn't watch the second set. You butterflies. Yeah I was just pumped to get shown in here in the finals. I've had a lot of tough luck in here in the finals and I was just really happy to get through it in here in the finals for sure. How do you find competing in the Will Rogers. Does it just make you more nervous than other other arenas or what's your take on it. I mean it's really hard to describe what it's like showing in here. The five six year old I had a lot of trouble just getting shown and getting through runs and I don't know what it is. I mean this place is just special. There's something about this place that just makes you feel all the history. And I don't know what it is. I mean I've seen a lot of fertility finals in here and seeing this place get really wild. And there's just something about that even when it's empty it's like just surreal to be in here and competing in a go round much less the finals of a triple crown event for sure. Okay so tell us about how you got into cutting. My dad actually was given two mayors, two Doc Lena mayors by his old business partner Max Pointer from Jacksboro. And he's one of our great family friends and I started riding with Diego and a Senseon and a Don Benuelos. And those guys kind of taught me the basics of cutting and moved on, rode a Tom to Vorac. And that guy's like a brother to me. I can't say enough good things about that guy.   And I just can't say enough about him and Cody Porterfield as well. I kind of am a new part of his program and I've been riding with him for about a year. And both him and his wife Cheyenne, they're just top shelf people. And really look out for me, look out for my horses. Just want me to do good. Sounds like you've got a great cutting family behind you there. Even here you've got your own kind of private house at Tom to Vorac. Yes ma'am. Tommy D sold my dad actually part of his place, a little house kind of in the back part. And my mom's renovated and getting ready to move out there pretty much full time. So we're pretty excited about that. All right, finally tell us about Fantastic One. It's a horse you share with your brother. Explain that. Yeah, we bought this horse. My brother and I bought this horse at the NCHA Ferturity this past year after the amateur go rounds. And Clay pulled me to the side one day and said, hey, I think this would be a great horse for you to show. And I think Cody could do well on it too. And picked up the phone and called my brother and he was like, hey, let's go have some. Let's buy it. And I was like, all right, man, like that sounds good to me. But he's giving you the reins to show it essentially. He's just awesome. We're just a tight family. And if one of us is all in for something, everyone kind of jumps on board and goes with it. It's pretty cool. That's really nice. Great speaking with you today. Congratulations. Thank you very much.


There's a video out there called the dark side of new age gurus and I think the channel was like James Janney or something like that and This video is completely wrong in many in many different respects, but I'm gonna just go through them kind of one by one and Debunk the video. I guess you could say now if you haven't seen the video Obviously, this won't make much sense to you. So if you if you haven't seen the video go and check it out I'll try and remember to put a link in the description here But the chances are you have seen it because this is a pretty big channel and you know, it's it was I think it was published in March of 2021 and He goes on to talk about various new age gurus and specifically the idea of new thought which I Assume he means is like new age. Now before I start this video. This is not intended to you know offend Or you know, annoy this this particular guy. I'm sure he's a lovely guy. Nothing against the guy It's just the things mentioned in the video. I'd like to offer my perspective Because I think there's a dangerous kind of precedent that is being set by some youtubers against the Ideas and concepts are starting to come to light now Specifically the idea that we create our own reality and that our thoughts our conscious thoughts and attention Have an impact on physical tangible reality. This is not science fiction anymore This isn't like this and I really want to make this clear before we begin This has been proven many times and the whole notion of like quantum mechanics And many other different respects have have proven this and demonstrated it time and time again This is not some kind of fairy tale make belief in order to sell some kind of dream or product Yes, people use it for that and I'll get into that in a second, but that doesn't mean the concept isn't real and This is very important because a lot of these debunking videos They kind of Suggest that that idea is not true and that you can't affect your reality with your thoughts Which is just fundamentally wrong in many ways you can and do affect your reality with your thoughts and in fact you've been doing it your whole life it's kind of It's a bit, you know mind-blowing to imagine otherwise so the video starts by talking about how The new age gurus I guess make money from selling you the idea that your thoughts create things your thoughts conscious attention can manifest and into things and either attract things you want or Give you less of what you don't want and he suggests in this video that that's a ridiculous idea That's not the case. I mean you and you can demonstrate this yourself if you just go 30 days and Intend to only think positive thoughts only focus on what you want and not think about what you don't want Just your subjective reality will confirm that that is that is the case that that happens for you and This is such an easy thing to demonstrate anyone can do this and It's very hard to you know To argue against that the law of attraction and manifestation You can call it what you like with a fundamental concept that you can create and You know draw certain experiences and things to you through your conscious thoughts is completely undisputed How else do you and does anything happen? You know if you let's say at a fundamental basic level, right? Let's say You decide to go to the gym Okay, so how so obviously you could say well I walked to the gym So my walking made it happen. No because before you just before you walked to the gym You must have had the thought I'm going to go to the gym then from the thought comes the intention I intend to go now From the intention comes the action the walking and then from the walking comes the result you arriving at the gym This is like such a fundamental concept To suggest that this isn't the case is just absolutely hilarious to me because We all experience this every day of our lives now where it gets a bit more debatable is How much of an impact your thoughts have on stuff outside of your control because you might say well, you know Obviously we can decide to walk to the gym, but we can't Use our thoughts to then manifest money coming towards us or you know our perfect partner or dream job or whatever it is and Yes, you're right. That's where things get a bit more debatable because the things the the chain of events The chain of events that happen takes a bit longer not a lot longer, but it takes time, you know, obviously so So we first go back to the basics, which is that our thoughts carry electromagnetic energy. This has been demonstrated many times It's been studied scanned analyzed You know, we even use this in most modern medicine medicine today where we scan people's brainwaves And you can see an electromagnetic analysis of the brainwaves when you think certain thoughts a certain electrical circuit gets activated in the brain So that's you know, that's that's the consensus at the moment at least The next step of that is where you mix into things like quantum entanglement And specifically quantum mechanics when you think about something Particles that seem to be separate are not separate at all. In fact, there is no separation The idea that anything is separate from anything else is a complete fiction. It doesn't exist It can't be proved. It's never been proved and If you zoom in far enough at the quantum level part all particles are connected. There is no separation between anything, right? So taking that one step further It seems pretty believable pretty likely that your thoughts we know they were electro magnetic Netic energy with they have a measurable electron electromagnetic effect And we know that other you know particles are kind of Connected in many ways quantum entanglement shows us that two particles can be connected Regardless of distance and if you do something to one particle it affects the other particle and vice versa This happens regardless of how far away they are using that as like the basis the foundation It seems pretty likely and obvious that your thoughts your intention can affect other things particles people experiences reality But this is not just some kind of science fiction idea, you know, this has been known for thousands of years and This is another point I want to get on to about the video the dark the dark world of new age gurus now at a certain point in the video he talks about Specific people or characters one of them being Napoleon Hill, but that's not the only one He focuses in on specific people within the personal development world within the new age movement and Looks into their past and he's basically dug up some things on specific people that make them look not so good They can't it casts them in a bad light. Okay, you know a couple of players have been like scammers a couple of them have used New age concepts to just earn money purely earn money And a few of them have actually been involved in so yeah like scams and various other things like that Okay, now you've got to remember the new age and personal growth is an enormous enormous topic There are millions of books Probably hundreds of thousands if not millions of authors. Okay, there are it affect the the reach of this Community and topic. Okay. It's absolutely enormous. It's if not the biggest niche. It's certainly one of the top three and Within that, you know, like any other topic any niche any community or you know group of ideas you are going to have specific characters that Aren't so good that scam people that teach the wrong things or that use the topic for their own Dark agendas or purposes, right? It happens all the time Does that mean that the concept in that in that topic are false? Absolutely not and I'll give you a very simple example of that. Let's say if you have a Thousand fitness instructors and they're all teaching people how to deadlift. Okay, deadlift is an exercise You lift the bar off the ground. Keep your back straight. Whatever. Right. They these thousand fitness instructors They all teach people how to deadlift If one of those fitness instructors turns out to be a scammer Does that mean that the idea of a deadlift is suddenly false and dangerous? Absolutely not, you know, the what one specific person or even a few people within a market do Doesn't disprove the whole market. It doesn't mean that suddenly deadlifting is nonsense But that is exactly what this video is is suggesting It's suggesting that because there are a few people within the new age movement who have been found out to be scammers That suddenly means the concepts. They're teaching are Disproven which is just absolutely not the case and to think that is such a flawed line of thinking You know, it's just laughable Especially when you use the deadlifting example I just gave you what you really have to consider is not the people teaching this stuff because that's completely irrelevant Their background their personality, you know, they're Their history and their Intentions are completely irrelevant if what they're teaching is true and useful The concepts they're actually talking about the idea that thoughts create things the idea that through positive intention and mindset You can change your life The personality of these people is irrelevant. What they're teaching is, you know, the only important thing now Of course, it's nicer when they're not scammers. It's nicer when they're, you know, likable But that doesn't mean what they're teaching is wrong and I just want to make that very clear another interesting thing that comes up in this video is He's he's trying to and I see why you've done this I was James. I see why he's done this James because you wanted to make an interesting video your Channel essentially is about Exposed documentaries where you kind of reveal a hidden side to things that people might not know about so I understand completely Why you've done this but just for those of you who have watched this video and who actually are starting to believe what you're saying You're I need to make a few things very clear the first one New age although it's called new age and new thought is not new at all the idea that and I'm going to kind of summarize here Right, but the idea that our thoughts conscious attention and Beliefs affect our physical body and our physical reality. That is not a new idea That's been around for more than 5,000 years and this has been documented It's so extensively throughout history that to suggest that it's new is Is ridiculous? quite frankly this has been known throughout history and if you look at even a small handful of ancient civilizations, you know of seers wise men sages Really any sufficiently advanced civilization or even you know tribes shamans and tribal communities they all knew This concept was true the idea that our thoughts and intention At the very least right at the very least affects our reality and At the very most indeed it can change almost everything about your reality however The society we're in and the programming where I can talk for hours about this, but I won't okay the society we're in heavily focuses on Trying to convince you that your thoughts don't create your reality and actually you're a victim of something you're oh, it's the government's fault It's society or the economy, you know, it's your Background is your education They're always trying to say it's something else Something else is the reason why you're in the current situation that you're in it's not your fault. It's something out there and This is an incredibly dangerous thought a very dangerous idea because the truth is that every one of us is in Charge Responsible for king or queen over their individual reality It's all you it always has been so the idea that it's something else out there Someone someone else's fault right the economy. It's the education you have that's not true and If you believe that then of course the world must seem absolutely chaotic it must seem like You have no control over anything things are just going wrong the world's crumbling down you you know you're struggling because Something in the economy isn't quite right, but the truth is in any situation and I'll use the economy as an example in any situation You will have people who have the victim mindset They think something's happening to them and they're just like the unlucky recipients of this You know unfortunate situation and then you have the people who understand the truth Which is that they actually create their own reality? Everything they're experiencing they've attracted everything. They're feeling they've caused With a few exceptions now these these are two very different perspectives on reality One of them makes you feel helpless and sad and depressed and fearful The other one makes you feel empowered confident happy and grateful So to suggest that You know, we should only do the first one right the one where we're like the victim That's a very irresponsible thing for a big channel to suggest and for anyone watching you are in control of your reality you You do have that power And you know, like I said, it begins with your thoughts your intention everything begins with your thoughts Anyway, so then he starts talking about It's almost as if he's like tried to collect as much dirt on the put on the new age movement as possible And of course if you look hard enough, you will find it in any topic any area of life So he starts talking about like this specific. I've forgotten the name of it now the specific event. It was some kind of new age Like conference or something and there was like a like a steam sauna like a hot hot sauna And a couple of people died in the sauna, okay, which is obviously very tragic it happens a person, you know a there is a statistic somewhere For how many people every year die in saunas it happens is very unlucky But it happened at this this new age conference and he somehow suggests that because that that unfortunate event happened That it discredits the person organizing it The ideas they're teaching and the entire concept topic and market around What he's talking about? Because these two people died in a sauna accidentally The the connections and mental gymnastics that this guy is doing is is nothing short of astonishing Anyway, so I guess this kind of applies to not just to this guy's video, but to any any video really any YouTube YouTuber any, you know online Influencer who is kind of trying to debunk or discredit the new age movement you've got to understand It's not firstly. It's not a new movement and secondly You can't disprove it because it's a fundamental truth a fun a provable fundamental you know Basis of reality that we can all prove and experience ourselves Without the need for any scientific study without the need for any, you know Lab or meta-analysis or anything else you can prove it yourself, you know just start now 30 days Pick a goal focus on something really specific Only think positive thoughts if you find yourself thinking a negative thought revert back to the positive thought and You'll find that I mean I could go on okay the connection between our thoughts and our body is such a Fundamental strong connection that it's been widely documented an interesting book I would recommend you read is the holographic universe and in this book they talk about studies and and accounts and Times right times where people have used their mind To change their body physical healing And I'm not just talking about oh, I thought I would get better so I got better and you could say well You would have got better anyway.  You'll make it true and This has been the case many times a good example is if yours if you're sick or ill You should never say I am sick You should always say I am healing or I am healthy because your thoughts and beliefs and what you say out loud Especially will become true if you say oh, I just can't seem to get rid of this illness Your body and mind and subconscious will all align to make what you just said true on the other hand If you say I'm healing You will heal. It's quite quite literally as simple as that anyway Yeah, I didn't intend for this to come off, you know as if I you know have something against this guy I absolutely don't don't have anything against him His videos are actually quite interesting. I've watched a couple more of them about other topics But just with a new age movement in particular and with I guess he's kind of he was kind of attacking spirituality And it was more about he was attacking a concept or an idea The idea that you create things with your thoughts is absolutely true if it wasn't true You would you would know it, you know, you would be able to prove that but here's the kicker right? Here's the really interesting part Like I said if you think something Strongly enough it will become a belief. It will become reality That same thing applies to if you don't think your thoughts create reality. I Hope you're with me on this So if you are watching this video and you and you strongly believe and you say to yourself no Steph That's nonsense. My thoughts don't create my reality What will happen for you not anyone else but for you specifically is you will experience that as true Which kind of proves it in the first place? Because like I said what you believe will become reality So if you if you tell yourself, oh my thoughts don't create reality That's all you'll experience if you tell yourself you let your thoughts do create reality then again, that's what you'll experience and You can test it out. So a good example if you're skeptical or if you don't believe this is to really Kind of trick yourself really just force yourself to believe it just for a short period of time just for a couple of weeks or so and You'll see what happens. You'll see the results. Anyway, I could go on for longer, but I think it's been Yeah, it's been about 20 minutes now That's my debunking and my thoughts on the dark world of New Age gurus by James Janney let me know what you think and Hopefully this has been useful in some way if you see videos like this in the future by the way Try and comment and just say hold on a minute, you know, that's not quite right You can't just discredit an entire idea because of one person who said it You know, you need to look at the actual concept and not the specifics of who said it because then it's just gossip It's just hearsay. There's no substance behind what you're saying, you know, and again, it's the same with me It doesn't matter How I look what the background of my video is Already anything even my anything that happened in my history or anything that will happen in my future What matters is the concepts that I'm talking about. Are they true or are they not true? That's all that matters and Yeah, so hopefully that's been helpful and I'll see you next time


So, hello, thanks for coming to my talk here in the BSD Dev Room. Some people might remember last year's speaking, me speaking about my engagement in speaking or in teaching BSD in my university, and this is the actual CIS admin part of my role there. So, it's a nice combination because I can teach what I learned as a CIS admin, and vice versa, I can teach the students a little bit what's involved. So, today I'm talking about migrating a big data cluster from Linux to free BSD, slowly but steadily, so this is going on for quite a while now, and I want to give kind of a status report and also a little bit how I approach the whole thing. This is what I'm trying to cover in the 40 minutes that I was given, and don't worry, it's a little bit more technical and also a couple of social aspects are in there. So, why this talk? Sometimes we get at the foundation or at the free BSD table downstairs, the question of, yeah, I like BSD, I want to use it not only at home but also in my workplace, wherever that is, but I'm the only person using BSD there. I'm surrounded by other people who use other operating systems, what can I do? And so, I thought, well, this is actually what I'm experiencing or have been experiencing, so I thought, let's create a talk about this and this is an experience report, basically how I approached it and maybe you can get some ideas how to implement some of this in your own environments. Because we want to see BSD rising in the, not only in the private space, but also in the environments where you work or in the work spaces. So, as I wrote here, there's no one true approach, so I have no master recipe to give you, but I have my story to give you which is not also a successful one but also fraught a little bit with peril, so it's not the perfect approach but at least it's one way to get slowly but steadily a little bit better. So, the first thing that you do is look around and learn, because if you're starting in a new company or you're with a company for a while, then you look around and see certain things that you might like or certain things that you don't like and maybe you figure out, okay, this is where BSD would come in handy in this area that is basically making a list of things that you want to change. So, I use it also to describe to you what the environment is that I work in and yeah, so this is the first slide here, so one day I inherited a big data cluster at work. So I started there and a colleague of mine had actually started this big data cluster project that they started from the need that the students wanted to do long-term measurements of performance like for NoSQL database, benchmarking or thesis work or just projects that they were doing and so they were granted a couple of machines as called big data clusters, it's basically a rack of machines, I have a picture in the next slide. And so they built it up, I wasn't involved in that part so I basically had to take what they left me or what they gave to me and so I took over this, this is the big data cluster it's called and it's used in the University of Applied Sciences in Darmstadt Germany where I work in the Computer Science Department although nowadays the whole university can use this cluster. And the person that initially administered was a Linux person oh, how new is that and was using Ubuntu at home so you thought it would be a good idea to use Ubuntu that he knew in this cluster as well and so this might be one of the first mistakes that were made but I'm getting to that and so this is an old Linux environment so actually the whole university is mostly Linux or the Computer Science Department a couple of people are now getting more into the BSTs now that I did a little bit of advocacy work but basically back then, this is a couple of years back there was no BST at all maybe me running this as virtual boxes or teaching students to use BST in my classes but that's pretty much it. So the OS image that they were using was based on the actual computer lab machine image which is kind of weird because this is a server environment and lab machines are typically sitting there waiting for students to do some work on them so lab machines are typically more desktop oriented and the image is also desktop oriented I don't know why they choose to use that on the big data cluster but I'm getting to that later on and then the classic thing, operating system on the first disk and the next three disks that are in each of those nodes are used for the NoSQL database data files and they also bought a central file server for exporting the home directories for the students and the professors via NFS-3 so this is fairly straightforward and fairly easy to use so here's the little photographics or for the hardware geeks here out there so this is what we use so we have old nodes, these are the ones that I started with so this model, the C6220s aren't even on sale by Dell anymore because they went out of support a couple of years ago and we're now using the next generation models, the C66320 and so it's basically 64GB RAM machines, 8 CPUs and 4 hard disks that's pretty much what we're working with and so some of the newer nodes have a bit more memory and a couple of newer modern processors but that's pretty much the only difference and the file server is a little bit less in the memory area which is kind of bad if you're thinking about using ZFS later but pretty much the CPUs are there with enough power they have enough disk space to cover the computing needs or the no SQL database needs that we have so the software in use when I took it over was basically a Hadoop installation and they were using also PIG which is kind of accelerating the Hadoop queries a bit and Couchbase was in use, MongoDB and nowadays we're also having a Spark installation which has a couple of other components in there so this is basically all in the area of big data and data analytics and we have a study program now with the mathematics department and that's called data science so the student can take that for a year or for a semester course and become data scientists and they're using the cluster basically for lab work, thesis work or project in this area with companies for example and so then I looked at okay let's look around what actually is supported on FreeBSD so is Hadoop supported on FreeBSD? Answer is yes and I wrote about this in the FreeBSD journal a couple of months ago how I use FreeBSD and how you can deploy FreeBSD or Hadoop on FreeBSD with ZFS because the ZFS part makes it actually interesting Couchbase itself has no native BSD client there's only Libcouchbase in ports and there's nothing to download on the manufacturers or the vendors website so that needs a little bit of porting help I see a couple of portes in the back maybe that will be a thing to work on MongoDB there's a port available but it's not the latest version so currently there's 4.0.5 and in ports there's version 3.6 and of course the researchers and the professors in my university are always interested in using the latest versions not that they want to use it every semester the newest version because they usually also want to make sure that their slides are staying valid for a couple of semesters but in generally if you are writing research papers you always want to test with the latest versions of the software because otherwise they will not accept your paper or laugh at you the Spark7 itself we started at last year and that's kind of involved the actual playbook that I wrote to deploy this has actually over 180 steps and you can imagine how complicated it is because Spark also includes the Hadoop installation and so there's multiple dependencies there to make it all work okay so how do we start basically or what's the environment that I'm also working on so education is an interesting place to work in that there's no production environment or the production environment is less important you would say than in a commercial environment where you have to have 24-7 uptime and all these things so my production environment usually only lasts like one semester which is like 3-4 months and then I basically can reinstall again because there's other software being used in the next semester so I only have to take care of the software for like 3 months and then let's reinstall the whole thing again that I made so much work on to make it running and so lectures as I told you they tend to stay with older software versions a little bit longer unless they're doing research in that case they need the latest and greatest versions of the NoSQL software and the production again is only the time frame when the students doing their labs which is typically 5-6 times and then the other week is the other group so we have typically two lab groups and the cluster makes it actually easy to not be bound to a certain lab because they can use SSH to connect to these machines from home or wherever they are and don't have to be at a certain lab in a certain hour and can only do it for like an hour and a half before the next class starts that makes the cluster relatively flexible in its use and the popularity since we have that introduced is very popular because students see that they can run a lot of beefy benchmarks on that or make a lot more CPUs spin than they have at home and so the problem was when we talk about installation life cycles and updates that we also have to make sure that the operating system supports the NoSQL databases and so for the longest time when we were still running on the Linux side of things they were running on the Ubuntu 14.4 version and we couldn't get to Ubuntu 16 or 18 today because for the longest time Hadoop had a problem with Ubuntu 16 you couldn't run a Hadoop cluster on Ubuntu 16 that was the vendor trying to get it working and we were just waiting and waiting and nothing happened and so the students kept asking hey can I upgrade this thing I want to use the newer version of Ubuntu and I said well if you want to use Hadoop there's no chance you can try it and you would help me actually make it work but several students tried and never got there and then someday Hadoop folks got it working I don't know what was the showstopper there but then we could update to the latest version so on FreeBSD no problem latest versions perfectly supported no problems but other software was kind of difficult in that regard because it was always just available for Linux and BSD is not even a download option on their website so that's a general problem and I also give away these lab machines or these cluster machines to students to do their thesis work on and it's always okay if you're a student what do you need as an operating system well I use Ubuntu whatever at home and please install this on the cluster okay sometimes I try to install FreeBSD for them and then ask them or gave them access to the node just to get an email five minutes later oh I don't know this operating system can you install Linux for me okay but a couple people tried and they actually did their thesis on those machines and even on FreeBSD because what they're mostly running are applications they don't care what operating systems are running on them and that way I can introduce FreeBSD a little bit on the sneaky side and some people just want to run their databases and don't care what kind of operating system is running below okay so that's what the environment is that I'm working on or what I've been trying to make changes in and so the changes have been coming slowly but steadily so the rationale here is I don't want to just introduce FreeBSD because I like it I once saw a couple of things where FreeBSD could actually improve things or make things better and that's why I try to bring FreeBSD in this environment and create something valuable for people so that they don't maybe they don't see the features or I've never heard about the FreeBSD features and that's why I try to introduce this and I've told them about the cool features for example that ZFS provides or that Ubuntu or that Hadoop is running quicker on ZFS than on Ubuntu with a traditional EXT file system and so the thing that I told you about at the beginning that we were using the actual lab image for the cluster which is more as a server environment that includes we get the whole desktop craft from the labs into that environment so I had printer drivers on a server and I had graphics drivers, sorry Nicholas on a server that will never run anything graphical because you need the CPU power to do the actual benchmarking and so I took that image and actually was cutting and slashing stuff out of it which I don't need and that actually reduced the install time of that image from 25 minutes to 11.5 roughly and I think I can cut this down even further removing more Linux stuff that we don't run on the cluster anyway and the previously installed itself just takes three minutes probably less if I can tune it a little bit more because the only thing that I need is basically get the OS running partitioning and ZFS setup and all that and set up the networking and maybe some LDAP so that the students can log in with their student ID and that's pretty much it then I reboot I don't have to install software in this part I can install software later using Ansible or other software distribution or just package install that's pretty much it I don't need this as a full big image that I have to copy over the networking to the node which actually creates already some value because install times are going down the nodes are available faster again if a student is finished and the next student is already waiting for that node to become available I can say sure in 25 minutes or now in like 11.5 you can have the next node available because that's the install time that it takes and if I'm testing things and I need to reboot the machine because I totally messed it up and then I have quicker test cycles this way so first steps if you're in a new environment and want to try out FreeBSD and maybe it's a hardware or hardware environment it never worked on is the question of course is your favorite BSD working on this it doesn't have to be FreeBSD it could be other BSDs as well but I want to use FreeBSD in the first boot I was kind of encouraged because a lot of devices were detected and because FreeBSD is the actual server that's where FreeBSD is coming from it's a server desktop server operating system and there's a lot of drivers available for these bigger and more expensive hardware parts and then the question is well I can install it but that's the machine actually reboot into the new operating system or is it just forgetting about bootloaders and all the other things that are required and luckily it did so then I knew that this machine could run FreeBSD then what I did was quickly start a process to automate the whole OS installation because as much as you like your favorite operating system there's nothing more boring than just installing it all over again just clicking the same prompts entering the same information all over again and so that's the first thing that I did trying to create this little ISO image that's completely automated just load it and let the machine boot from that and it automatically installs and you just have to watch and drink coffee and so also network setup is a bit interesting in the university environment we do a little bit of IPv6 there we also have a couple of things where we try to separate networks from the students so that they don't break into our central servers or whatever so there's a separate network for the big data clusters and it's also a bit secured this way and we have an LDAP server as I mentioned that is providing the whole accounts for the department and for the students so that the students don't have to create their own accounts and remember their passwords all the time so it's just one password they have to remember and of course the NFS setup from the central file server and that is imported from them and then they have every file that they have in the home directory on each node that they have access to and of course I could reuse all of the or many of the things from the Linux desktop environment that they provided initially but I just had to move that to BSD where paths are different or where some Linux specific things that don't work on BSDs and that's the actual first steps that I took to create a node that is complete free BSD node and we're still not talking about the applications running it this is just operating system and networking ok so then we need to tackle bigger problems because then we have already established some kind of foothold on the cluster with the BSD you know that it's running and then the question becomes is the actual software that the professors are using running on that environment and again this I get a lot the first sentence here well actually we don't care this is the database group where the cluster is located so and they pretty much don't care what operating system they're running on because they're mostly focusing on the database side and don't care what operating system is running below and as long as their favorite NoSQL databases are working then they can just do their benchmarks and whatever they need for their research purposes so the question is why aren't you not just running everything in beehive because then you can have a bit more flexibility in just providing Linux machines that are actually running on a free BSD environment and you just fake that there is a Linux distribution in there and they are just running their same environment that they are used to and that could be done the problem is that the professors actually or the students that are working on that are actually doing performance benchmarking and creating papers and thesis work and as much as I like beehive and its performance or its thinness in the layer between the operating system and the virtualization there is still a little bit of overhead and that little overhead can be seen in benchmarks or could be seen in benchmarks and that might skew the results a little bit so it's kind of difficult to say okay let's fake everything in beehive. Overall we had used beehive a couple of times because as I said the cluster was becoming very very popular and a lot of users wanted to use the cluster and I said oh sorry I'm out of nodes I don't have any nodes left we have to wait until the next student is finished with his thesis or her thesis and so what I tried to do is give everyone a little bit of space on the cluster in a little beehive because they don't usually run everything on all nodes or only need like 4 CPUs instead of 8 or only 16 gigs of RAM instead of 32 so I could try to partition this a little bit and that worked a little bit when we were in the situation where a lot of students wanted to do the data science labs and we didn't have enough nodes left so I had to create a couple of beehives to make that a little bit better to allow more users because we want to let the users use our nodes or our cluster as much as possible okay so that's the research part why we cannot do the beehive thing the other thing is you saw that there are four disks in there and it's immediately clear that everyone says well of course I created a rate Z rate 10 on that and perfect over four disks that's a good performance boost you also would get ZFS compression and all the other things to manage your IO better and all the cool things that Alan Jude could probably talk about and of course you would also use a separate data set for each node SQL database it's much more flexible you have much more to manage your storage this way problem is same argument when they do scientific measurements or writing papers apparently if there's a rate layer involved somehow then the papers that you are submitting are flat out rejected the paper committees just look for not just looking for that but if there's a rate layer somewhere if it's a hardware rate or a software rate then they just say well it's all all the benchmarking results are the result of the rate layer that's one of your tuning mechanisms in the database layer and that was kind of a bummer for me because I really want to introduce ZFS into this environment because I think the professors would also like that and but when they have to do this paperwork or this research paper things and are rejected from conferences we didn't have that yet but that's the downside of this if they wouldn't do any benchmarks then it's perfect it's the environment we're probably going to use and and are always thinking well if the paper committees reject those papers then you know how realistic or production ready is this kind of setup because then you could say well if you cannot do a rate then you just have to use a single disk but that single disk will fail I mean the disks in our cluster are like five years old now so I just wait every hour to get a result from the smart CTL to tell me that the disk has died but is this actually I mean this might be good for research but is this production things that we're talking about here so I'm kind of a little bit on the edge about that okay so next thing probably for comparison optimization on top of that why so it's hard to decide where did the performance really come from yeah the comment was here that the comparison is difficult if you're different you have different rate setups for comparison okay so now we have a free BST a couple of free BST machines in there next to the Linux machines that are already running the old kind of hardware and the old kind of software stack so now let's grow and extend a little bit so the biggest issue here is the file server oh we don't only have nodes we also have a file server and since the file server has been serving these home directories to everyone and the research data set I mean we have a 370 gigabyte JSON file to load into the NoSQL databases to I mean it compresses super well on ZFS or on I mean GSIP but they don't have to they always have to ship the 300 gigabytes JSON file from one machine to the next to load this into the NoSQL databases and then do some kind of schema management or I have no idea what they're doing but it's I mean 370 gigabytes of JSON okay and the file server is basically providing all these datasets and yeah welcome to the new world so since the file server has been serving all these files all the time there was no time to actually make it an upgrade of the software and to the actual operating system bits and again the file server wasn't set up by me and what they did they basically did a BIOS rate 5 so this is a hardware rate 5 and oh my god I wanted to run smart CTL on this thing and it gave me nothing because it couldn't look into individual disks made up of that rate 5 so I had no idea whether this disk were all dying or whether they are healthy so I did only see this one big blob of storage that they were providing down from the BIOS to the operating system and there was one issue from the start there were always so you would create a file on your home directory or on the file server shared and it will always have strange permissions you would get permissions like suddenly you had permissions that the dean was actually owner of that file or that the secretary was the group of this directory and I was like where is this coming from I mean the all the data is provided by the NFS or the LLAP servers the LLAP has all the group information and the group sizes or the group IDs why are they so wrong when they just touch and so after a while of looking at this I traced the problem down to the ETC add user conf on linux previously doesn't have that file maybe that's why I didn't find it right away and in there you have a value called first UID and if you're on an LLAP setup you need to increase this a little bit so that the local accounts are distinguished from the ones provided from the LLAP server and my predecessor didn't know about this or at least didn't do that because the time stamp on the ETC add user dot conf was from the date when the server was installed no changes since then and once I bumped this all the files that were created had the correct ownership and permissions but you can imagine this was like five years into the big data cluster is live I'm not going into each individual's home directory and changing permissions that way so I thought okay let's this is a failed experiment let's reinstall the whole file server otherwise we're just coming into problems again and again I didn't like the hardware rate so I always wanted to use ZFS for this so I made a move into reinstalling the file server oh here's the first picture by the way yeah so the process of migrating the file server was actually quite easy so I said okay one of the nodes is now becoming the new file server because I would reinstall the file server in place even though we had relatively large semester breaks so where this could be possible but even during the semester breaks there were students writing their thesis on them or professors were doing projects on that so it always had to be something going on in the file server so I said okay now I take a note from the cluster and I install it the way I like it like with ZFS and test all the things that I wanted to have that server was going to become and I copy all the data files over and create the mount points for the students and then let this run for a whole semester so this file server was actually offline no one was accessing it and they were just making all the calls to one of those nodes which was now the new file server and then during that time there were some issues or some things where students were confused where the file was finally ended up but all these things went into basically a little file where I wrote down things that I needed to change or to make adjustments for the actual file server installation and once the semester was over which was pretty much painless so far I mean there were a couple of things that those were easy to fix and encouraged by those things I did the actual change to actually now say okay big announcement to the whole users from the big data cluster next week we installed the file server and surprisingly no one else was complaining because they were actually using this one node that I dedicated for as the new file server or the interim file server and they said yeah okay I'll give you one day and if everyone is complaining then well we still have the old data on this one node that I used for it and so one day me very excited going into the server room getting in front of the file server and removing the old Ubuntu installation and booting 3BSD for the first time on it in which cause I removed the the nicest thing to just go to a server and remove the hardware rate without having to worry about the data on it so that was cool so what I did is I created an individual for each disk we have 24 of those in the file server so for each individual disk I created a rate 0 because you couldn't do a no op export into the operating system so I had to so each disk has now in a rate 0 configuration exporting itself to basically the operating system and ZFS will basically pick that up and create the pool out of it and the data sets on top and ZFS installation went pretty well and in its cause I discovered the amazing MFI util so that's basically managing all the things on the operating system side that you would only do from the server bios and there are utilities like check each individual disks or run periodic checks and that's a very powerful tool and of course then after I was happy with the configuration and the file server setup that I wanted to have I copied all the data back from the node that was currently serving all the files and then said well thank you dear department now we have a ZFS based file server that I immediately saw the compression ratios and I was like yes so much disk space was saved and yeah everyone was interested in that and I told them about of course the other features that ZFS has quotas and reservations and all the other things because that's actually the interesting part then the students don't overwrite their home directories anymore and I can limit the students to only say like 2 gigabytes of disk space that they're allowed to use maybe 4 if I have good day but yeah that's pretty much the actual migration I'm pretty happy that I could make it work because the operating system on the file server is really getting old and I was worried that someday someone might be hacking that thing so I'm not actually too worried about external attackers I'm more worried about the students in computer science trying to do oh what's this utility doing and oh now I have a cluster machine on my control yeah not gonna looking good and then this is our new setup so you can see that I now export the NFS I also did an upgrade to NFS 4 because now I can serve it from ZFS and that way I can see what kind of data the students are actually saving in their home directory and also how much disk space it takes and ZFS automatically compresses all that so we have like home per slash home is the home directories for the professors and the employees like me and they're separate from the students home directories in case someone else has the same name as the professors and they have their same home directory interesting concept and we have of course home data sets that has all these crazy 300 gigabyte JSON files and other things that are needed for the data science environment and yes no wrong permissions anymore because ZFS is understanding everything that the LDAP file server or the LDAP server is telling them about groups and permissions and I'm super happy now about this file server setup but there's more, there was no monitoring when I started this big data cluster administration well there was previously a monitoring but it broke somehow and they didn't have the time to fix that so now we have a since the file server is now monitoring each individual node and to make sure that the file server who's running the monitoring isn't going down I have a little Raspberry Pi running on ZFS that's checking whether the file server is still there who watches the watchers so I get performance data from these nodes and if a student is making some crazy changes like running a bitcoin miner which has happened once that's probably my talk next year then I get the performance data and I see if a node goes down unexpectedly or if the network is having problems so that's kind of nice I'm using Icinga for that it's not it's not Nagios but I like Icinga the things that I saw and I will probably give a tutorial about Icinga in one of the future BSD conferences and it's a nice thing because I get a little bit more e-mail this time every morning because each server is sending the information about how healthy the disks are and what kind of processes we're running and how many students have tried to log into this one of course feudally and yeah this is the actual file server setup that we have and the monitoring that I added to it and again it's all now we have a free BSD file server we have a couple of free BSD nodes like 50% of the nodes are now free BSD and the actual monitoring is redundant with another Raspberry Pi node that is just checking whether the file server is still online and so in summary this is where we came from on the right side we got a desktop image node installation with Dupont 14.4 with reinstalled times of 25 minutes we had NFS free mounts with some crazy ownership and permission problems they had back then a little bit of Ansible scripting like create a user on the file server or create a home directory not so much but it was at least a start and they had a broken monitoring back then and now with the free BSD setup we have a fully scripted server centric we have much lower installation times this is just getting a coffee on free BSD and Dupont who takes roughly 12 minutes I guess I can shave a little bit more on that but I have to see whether the newer versions which are probably now with more software that no one needs on a cluster we also have a ZFS based NFS4 file sharing server which provides ACLs and all other things that are interesting in the NFS4 world and with parallel NFS because that's been added to free BSD recently because that might be giving us a bit of performance boost and maybe a couple of redundant file server in case the file server might go down one day and I have pretty much everything Ansible scripted so all the Hadoop installations are there, the MongoDBs, the couch bases that we're using on the nodes I just have to say please install this on X machines and it gets installed and a couple of maintenance scripts like delete all the users from last semester and get a new boot environment installed or snapshot done and things like that and most importantly we have a working and redundant monitoring and alerting system that keeps me awake at night telling me this one machine just went unexpectedly down and it's probably the disk that is five years old so in retrospect it was a little bit sneaky where I introduced free BSD so I didn't actually say hey tomorrow I reinstalled the whole machine with free BSD because that wouldn't work I would have gotten a lot of pushback from the professors who were used to their working environment and didn't want to have to suddenly deal with a new operating system that I've never heard of or I've never used and so you have to be a little bit sneaky at times and also patient I mean this is a process that's been going on for like the last three years and each semester I had a little bit more and more free BSD machines in there and as long as students or users are only using applications most of them are just oblivious they don't care what operating system they're running on there might be different environments and this is a very unique environment as a university but I was quite happy and this is not the end result the process is still going on as I had to experiment with other noise kill databases on free BSD and seeing what kind of versions are available in ports I collected a couple of tips and learnings at the end so as I said start small and incremental do don't do the whole exploration step into different environments that you are not familiar with start slow make incremental improvements and don't do a big bang thing of saying okay next week we're all running on a new operating system surprise and also let people the time to experiment with their operating system that they're now being given or the new way that their home directories are provided for example and let them just get a little bit of experience with that and once they have no complaints then you can just continue with the next step always testing is a big thing test everything before you roll it out so if you're not sure whether things work test again test again test different scenarios so for example for the FS mounts I not only had to test whether the mount works on free BSD but also working on the legacy Ubuntu systems that we're using and there were some interesting things coming up so that also needed to be tested and write everything down if you're a sysadmin you're probably used to that so making notes about every config file change everything that you're just doing on the command line paths that might be different from your other operating systems any settings that you might do in your loader.conf or RC.conf all these things might become important when you're doing a new server installation and back everything up oh my god if I would have lost all these data sets that I used for research I would have not my head again so if I lose some of the data that during the process of migrating the file servers that would not go well with the professor so that way I backed up everything to multiple notes to just be sure if one note would die I still had the data from that but I would encourage you to experiment and as long as we have boot environments we have all kinds of mechanisms to save our necks and you never know what comes out of that if I wouldn't have installed the file server I would have never discovered the MFI utility and what kind of cool things it provides without going into the bios to make changes to any kind of settings for the rate controllers and again don't just do things because you like that operating system of yours make sure that you provide some kind of value to your users be that they don't have to buy the server hard disks in the next couple of months because you saved a lot of disk space using ZFS compression or that they are now having the proper files and permissions that they are used to small things like that can make a lot of people smile and your users are happy that you are such a good sys admin taking care of these systems sometimes you have to be willing to compromise like the systems that I'm running is now doesn't have the ZFS setup that I'm actually initially provided because that was the research and benchmarking things for the papers that wasn't going to happen but it's the environment that I'm working with and I hope that if we don't do any projects with benchmarking then we're not going to run this other setup and the rest is just common sense really and my time is up and I guess you have a couple questions, thank you we have a couple of things about the cluster here if you're interested in hardware and some of the environment then find that link thank you


Okay, good Yeah, thanks a lot. My name is Dan Phillips Daniel Phillips. This is going to be a short talk about web assembly and syscalls Quickly about me. I am an engineer and the wisdom lead at loophole labs. We're a small seed stage startup I would do a lot of networking stuff and also some other Info tools. We just released a thing called scale function runtime. It's a suite of tools for running functions in a web assembly environment On the internet. I'm D. Filla and some variation there of across different platforms And I also started and run the was on Chicago group So if you're ever in town, feel free to stop by or come join online Okay, so we'll get right to it syscalls and web assembly is a problem for the Unix epoch epoch epoch I like the word. I like the sound of epoch better So I have an arbitrary bet that I'd like to start this talk with At least until this epoch ends. We will need to handle legacy programs that rely heavily on POSIX interfaces with system specific function calls in web assembly, right? This the epoch ends in 2038, right the year 2038 problem. So I don't know just arbitrarily I feel like we're gonna need to handle POSIX and Unix based programs in web assembly until then So if we look at this code This is just a simple function that does that makes three but technically four Syscalls, right? This is rust. So we have open right read and close So if we run this in If we compile this to wasm 32 unknown unknown, what will happen? Can we run it not not really right like not out of the box But these are some very basic file system operations. So What do we do about these system calls? There's one option. We can patch these with host functions We which is how things like the M script in VFS work Things like wasm by engine these sort of things that call out to the host runtime And then possibly on in other places those call out to the operating system itself We can also ignore functions if they get compiled To this specific target There's a second option Which is we can allow scoped access to the underlying hosts that is wazzy right wazzy is the way to do this And at this juncture, I just like to say that any wazzy people in the room just cover your ears for the next segment and Everyone else. Let's just keep this between us, right? We're all friends here I'm kidding. By the way, we love wazzy and actually the whole point of this is to use This exact thing with wazzy components in the future So the third option which I'm going to talk about today is What if we provided a wasm component or layer that acts like the underlying system? That's kind of what we're going to explore and this would mean something like a libc implementation and possibly Usable system calls even though that they're not system calls in the traditional sense So how do we identify system related calls? This is a challenge with an arbitrary web assembly module, right? There's not a ton of tooling out there for this But recently my colleague who's here Jimmy Moore worked on a project at our company Called the wasm trace which allows you to it's an it's an s trace like tool Which allows you to trace the specific calls made in in any arbitrary wasm module. There's also some other interesting Toolkits in there that do some other things with dwarf symbols and I highly recommend you checking it out So if you've ever used s trace detrace, whatever You'll know that one of the things that it does for you is It it can print out the the system calls that are being made or will be made in a specific program, right? so If we have these syscalls we can figure out what syscalls are being made What can we do with them then if we're going to think about a wasm-first environment? There's a small project that we worked on Called marquardt wasm, which is a small CLI that adds system layers Sort of virtual system layer a virtual platform layer to an existing wasm application, right? and I'm just going to kind of go through this quickly and and show you what it can do as long as I don't trespass on on my time I'm going to do this by using our scale function runtime, which is the code snippet that I showed before so this is just a This is a QR code that goes to our landing page tells you about it. It's also at scale.sh, so with that Do some demoing oops if I can type Can everyone see that okay too small? Yeah Yeah, okay great. So this is a marquardt which comes from French, which means layering What this does is this will create? Either the libc interfaces that you can then use in your wasm code or if you pass in the full flag It will also create the sort of underlying system functionality, right the type of thing that The kernel would take care of in a native platform, but this just builds it in a virtualized layer Right right with this CLI and then you can just consume that in your web assembly program. So if we try this That's in the full flag Great, okay Okay, so here we have the code that created both the libc interfaces and the VFS for those specific calls Okay, so what I'm going to do is copy this Take it over to my scale function here, which you can see this is The code that I showed before right we have open right read and close there's there's no explicit close But when the function when the rust function Drops then closes is called right so what I'm going to do is instead of just I'm just for the sake of ease I'm just going to paste this here Okay, you'll see so this is here Really really quickly with a scale function, this is just all done with our CLI in scale What to find the dependencies are this thing called a scale file very simple key values And you'll see here we need the WASM VFS, which is a small project that mark hot relies on and then also These signatures which are what we use to communicate between the host and the guest That's all covered at scale.sh if you want to check that out Cool, so we've got this now and now I'm going to build fingers crossed to the demo gods great The scale function runtime is written in go so it comes with a fast HTTP server out of the box So we can do to run this locally Do scale function run? VFS latest Lastest Okay, so that's running at port 8080 and we'll do this and There we go. We're interacting with the file system right in a serverless function, which File systems aren't usually a thing in serverless functions. So there's there's some interesting things that we can do with this, right? So here right open right Read and then close after the after the function completes. So, yeah Last thing I'll say just as a side note. We can also do this as of this week so We have a global edge network that has that pushes this to Places all over the world to deploys on scale cloud, which is available in less than 50 milliseconds Anywhere on on earth as of this morning. So you can check that out too at your convenience We'll also have some more info about that. So, okay, so that's it for the demo, but Where does this leave us right? The future right so if we have these layers, what can we do with with these virtual platform layers? We could do something like layering with diffs kind of in the manner of Container images right how layering works and how you can use different dependencies instead of having to replace the whole thing You could just use different system layers and layer them on top pull them out update them Whatever you need to do Virtualization layers for other system components specifically those that must have host access right like if we're in a wasm first world We still need a network at some point probably we still need things like block devices One of the ideas here is just like a VFS works in the Linux kernel is that we could have a VFS sort of mapping over a Wazzy first component right which then that component itself has the only scope to access every other component could just be wasm Right, this might be an easier Institutional cell to kind of like stay in line with wasms sandbox and capabilities and just saying it's just this one Component that can communicate with the actual host and its scope to this exact thing. It's just a thought Could this lead towards a sort of true isomorphism right people who remember the early days of no JS Thinking about isomorphic JavaScript This could also sort of blur the lines maybe in some interesting ways between the server and the client with web assembly running with virtual file system layers virtual system layers in many places not just the server and client but IoT devices and many many other situations and with the component model, right? This is something that we're looking forward to parametric linking will make this much easier Resource and handle types will be a huge boon for this also So when we're thinking about the VFS layer that we just created that side steps The context switching that happens in from user space to kernel space, right? So with the component models resource and handle types This could actually make things faster than native for some of these file system operations Which is kind of a theoretical possibility, but very very interesting And then the tooling can evolve with new and emerging standards. We can sort of keep this going We've kind of set this up so that as soon as different runtimes have Support for the component model we can swap these things out and get them up to speed and make things even even better So thank you very much Thanks to the CNCF and the bike alliance really really incredible groups the wasm cg and wisey subgroup for letting me be a fly on The wall to help understand some of these things better and the organizers It's if you've ever organized something. It's so much more work than it seems. So thank you. Thank you very much again I'm Dan Phillips and thanks a lot appreciate it Thank you, Dan. I think we might have time for one or two questions if anyone has them Please go ahead and introduce yourself Thanks, I'm Roman from Cosmonic I've just quick question. So you quickly show that tool analyzing the wasm binary, right looking for syscalls So my question was so I briefly noticed you there was was illypsy in there So if I were to not rely on was illypsy Would it still be able to trace a syscall or does it only trace things going through was illypsy? Yeah, that's a great question. So The question was if any if everyone heard it, do you need was illypsy to trace the syscalls the answer is no So so like this the trace program that Jimmy wrote Jimmy by the way, if you could write likes wave, that's that's Jimmy my colleague He also wrote Mibbit if anyone knows Mibbit the IRC client a client anyway, so It traces everything right and so you can sort of pick out what you need So if there if it were if was illypsy were compiled it would trace that too So it but it doesn't rely on it. No. Yeah Also, I'd be happy if anyone thinks of other questions. I'd be happy to take them in the slack later, too Or just talk to you anybody else and Speaking of organizers Dan you're prolific with what you do with wasm Chicago I know that you like all things web assembly come across you and your community and thank you for all the hard work You do I think we'll go and wrap it up here. Thank you so much. Thank you