Let's see, go in those sweats. Wow. I hope it's everybody chain-linked. I believe in chain-linked, too. All those things that I just showed you guys, I own all of them, except for Sheba, you know. I just don't get into that. But does that mean that's not gonna be awesome? No, it just means that I don't own it. I don't own a lot of them. I don't own a lot of them, so that's it. Dr. Payne is here, who came in 11th. Congratulations, Dr. Payne, in the Sweat Coin Steps Challenge. Very nice. Richard Hart is hiding out somewhere in Europe. Interesting. Dalfa Lever go back to the U.S. Someone's calling something a scam. It wouldn't be a day unless somebody in crypto was calling something a scam. ETH is the future. I don't even know. Hex, Sher, whatever. Bitcoin Cash, never heard of it. I used to have Bitcoin Cash. I actually interviewed Roger Vier, super nice guy. I just don't understand why we need Bitcoin Cash. Maybe I'm just seeing it wrong. Ha, ha, ha, ha. I'm the Ado whale. Yeah, YouTube notifications are just crazy. Sometimes you get it notified, sometimes you don't, which is how it is. Ah, Achilles, Ado is a ghost chain. Yes, yes, it's a ghost chain. Nobody builds on that. Nobody. Eh, hello. Okay, get to the questions. Mark G, do you promote Bitcoin? I do promote Bitcoin. I promote everything that I own. So you have to understand that I'm super biased on this channel. So like the things that I get into and I talk about are the things that I actually use. So like the Stonebook and like iTrust and Masterworks and Coin Ledger for tax purposes and Epic Art for crypto or just stuff that I use. And of course all the cryptos that I invest into very, very biased here. Yeah, unfortunately, that's true. And think about this way. Everything that's out there, look, everybody has their opinion that something is a scam in crypto, right? Something is a scam in crypto. And then people outside of crypto in traditional markets, will say that the entire crypto verse is a scam.  And if you hold precious metals, same thing. People say, we don't understand why you even hold precious metals.  It's just how it goes in the world. And you can't please everybody. It's just how it is. Yeah, that's pretty cool. Rob Payne and Particle, they bought a Banksy and broken a thousand NFTs and sold them. Sure. I don't know how they're getting away with that because that is definitely a security if they're doing that.  They register everything with the SEC and they put every different, every piece of art that they own, they register it as its own separate LLC and you own a fractionalized share of it, kind of like a real estate investment trust or an REIT, kind of. So yeah, I don't know how they're doing that though. And that must be flying on under the radar. Yeah, that's true. Let's be honest, we all do love a good scam. Mark, she tells what to buy. I can't tell you that. Look, you know I've been wrong a lot, right?  How many times did I talk about Voyager? How many times did I talk about, how many times did I have Alex Moskamsky on this channel talking about Celsius? I can't give anybody any advice. I can only tell you what I'm doing and that's essentially it. So I can tell you what I'm dollar-cost averaging right now with some provisions. So like I'm buying, let's see, let's go take a look at the list. And by me telling you this, doesn't mean you should go do that. I'm also gonna have a big bowl of cereal tonight. And if you're lactose intolerant, I wouldn't want you to do that either. So again, here we go. So I will be, and I'm dollar-cost averaging the regular amounts, not micro. So Bitcoin, Ethereum, Cardano, Polygon, Polkadot, Avalanche, Chainlink, Cosmos. And I should've gotten a file coin, but I just never really felt like it was the good thing. Near, here and there. Algorand sometimes, but it's few and far between these days. Yeah, that's about it. Oh, and Terraluna. Now I'm just kidding, I don't buy that, that's junk. Sorry, if you're in Terraluna, I just don't buy that. But that's about it. And there's a couple other ones that are way far down the list that I won't talk about because they're way too risky. And even like me talking about those things, people will still be like, oh, Bitcoin must be great. Let me just sell my house and buy it because people don't do their own research. So I'll keep the other ones close to my chest and go from there. Let's see, Toja. Hey, that's awesome. Rob, what cold wallet is the most easy in your opinion? So again, pretty biased on this one. I use an analyzer, I got a couple of those, three of those now actually. And that's just easiest for me, but people say they don't like it. Some people tell me that Tracer is super easy to use probably because they're used to it. I've used the Arculus wallet. I'm not for sure that is really like the cold storage. I think there's still some issues with it. But I like Arculus, that little metal. Do I have it here? Yeah, there it is. Looks like this. That one, it's pretty cool.  And just walk around and buy stuff. You wouldn't put, I don't know, $10 million in your wallet and walk around. But this, I think is pretty cool, just if you wanna buy things throughout wherever you can buy things, that's about it. So I can't really tell you which one's easy. To me, honestly, the analyzer is pretty easy to use. But some people have other opinions and it's just cause I don't really use anything else.  I like that. I like most of the beer. Barbecue, I'll actually, I'll take both of those. Can you recommend a YouTuber to do more active day swing trading? Yeah, Tom Crown. Tom Crown seems to be a pretty good one. And then, meme, who was the guy you were talking about that really does good TA? Put that person's YouTube channel in the comment section. Who else is good? Mullet's good, just another YouTube channel. Can you tell me some things? Who else is good? I don't really follow too many people with the TA stuff cause I don't really get into it. Again, it bores me. It puts me to sleep. I don't know what this means. Was there anything believing in an altcoin and roll it, Mullet? I don't know. Any news about World Mobile Token? Something that came out.  That's about it. But there's a couple of big news. Oh, there is one big news. I was talking to Mickey Watkins and they're going to, we have this sports facility in El Paso, it's a sand volleyball, sand soccer complex and pretty fun place to play. We've had it for like 13, 14 years. And I told them, I said, hey, if you guys want to, cause they're going to branch out World Mobile Token which brings telecommunications to different areas where it doesn't have it, especially in like third world countries. I said, hey, if you want to put one up at my facility cause it's kind of out there away from the city, I go, I'll definitely allow you guys to do that. And he said, yeah, sure. So it looks like they're going to do that sometime this year. I'll let you know how it goes. How would copter? Good question. What happens if crypto on and off ramps are hunted down? Well, then we just become unfortunately more underground and things slow down. However, I gotta tell you, it's a funny thing because when things start to go underground they became a hell of a lot more expensive. I mean, just take a look at prohibition. I mean, I don't, I was not around at that time, but I hear that alcohol became super expensive because it became illegal and people really wanted it. So when you have a product, good or service that people want and people can't get that because it becomes illegal and in some way shape or form then the price tends to increase. However, now that we're in the digital age all the means for us is if it's going to be an on off ramp issue you can understand it'll probably be in places like America and India that really happens. So what it'll mean, it just means that it just gets pushed like a balloon deflating it just gets pushed to another side. So America deflates a little bit and it fills up over in Poland or it deflates a little bit. And then of course, the EU takes it up or Australia or whatever else and then or Asian markets, it doesn't matter. So for me, I just see like we're not in the early 1900s where once one section shuts down and just destroys it for everybody. It just, one country gets left out. I'd like to not have America left out but if they want to do that then they're making a huge mistake and we'll see how it goes. Yeah, crypto's got a good point. And crypto will most likely outperform it's done since inception. Nothing else has really topped it yet. So there is that. Yeah, I mean since inception course done pretty well. Do you recommend multiple stores? Yes, I do. Because it's just this thing that I'm always worried about. If here's a big driver. So like if I kept, I don't know like $5 million or $10 million on one ledger I would feel kind of neurotic about that. I mean, granted, if, see, I'm looking for ledger I don't know where it is right now.  Well, it's not lost because I have my mnemonic phrase in my book. But let's just say for some reason this is why I have multiple books in different areas as far as like my mnemonic phrase is my stonebook. Let's just say for some reason I don't know what happens. They all get destroyed. It all gets wiped out. I don't know how. Then everything's gone on that one. So if I keep it in multiple I feel a little bit safer. That's just me. Some people just put it on one and they're like eh, whatever. But some people are crazy because they've got like 10 million there's no big deal to them. To me that's a lot of money. What do you think about the graph? No real opinion. I don't own it. So I don't even talk about it. Graph AI, sleeping giant. Sure it is. Thoughts on Rweave versus Filecoin or a bit of Paul Barone showing that. Talk to Paul Barone. That guy seems to know he's got a great team behind him. So meme, I'm not for sure on that one. And meme, don't forget what's the name of that guy that is the TA you were talking about? Yep. Jdedaysus.claims to be a software and not a security. They wrote, I forgot the name of the I think it's a no contest letter. They wrote that to the SEC and said, look, we're not a security. We're just software. So please exclude us from all these Wells notices and litigation. And they're still waiting for an answer from the SEC which is like what everything is going through. So yeah. Poke it out.  Hopefully it makes itself out. Who knows? That's why I like to diversify. For how I still love the Voyager price prediction. Yeah. See, can't trust anybody. Like I thought it would go up pretty high. I remember one of, it was 29 cents. I said I was going to go to 30 bucks. I went from 29 cents to a dollar to three, five and seven. And I went back down to five. And then I went down. And then of course, Voyager. Voyager is a weird one because they just did a very dumb thing.  There's a lot of things coming out of Celsius. And if you guys aren't watching Simon Dixon, Aaron Bennett or Tiffany Fong about Celsius or no stop Pete or Pete no stop, watch those guys because they have their pulse on what's going on with Celsius and just the straight up lies they were doing. But Voyager was different. Voyager wasn't a scam scam.  Who does that? Just because, you know, these guys were supposedly the wonder kids. That's the thing. And that's why I still, I look at it and go, I have to diversify. And that's it. Skaman ski took my big one, took mine too. I got a whopping six figures tied up in Celsius. What are you going to do? And I can tell you guys that because everybody knows that because they doxxed everybody and put the judge put everybody's, like information online, not like your address but like your name and how much you have. So yeah, that's how much I have. What are you going to do? What are you going to do? Just wait. That's why I really liked Simon Dixon's plan. Made a lot of sense to me. Hadera link.sand, this is solid. I own some sandbox chain link polka dot. Dr. Ryan, I don't know man because like, I mean, Gary Gender is not going to do it. That guy's really, really streamlined for me personally. Correct me if I'm wrong here. I kind of feel like Gary's making up for lost times. All the meetings that he had with Sam Bakeman Freed and all the different meetings that he's possibly had with other exchanges, he just missed the boat. And he could have done a lot of things with Celsius and Voyager, potentially BlockFi, but he really missed the boat in FTX.  And now it's like he's making up for lost time. You know, when you screw up and then you overcompensate, that's what I feel like he's doing and he's going to do moving forward because he missed it, let's be honest.  That's not my, I'm not the SEC chair. So Tom Williams says, obviously, that has been classified as software and security. If that just came out in the last 40 hours, I haven't seen it. But I know that they requested that letter. And I believe it has been granted by the SEC. So try some ethue. What about, Mr. Wolf says, what about real estate and homes waited to buy some to rent out? That's a tricky question. So like for me and the wife, we're always looking for real estate. Like we have condos here and apartments and house of the places. But like we don't try to like say, okay, well, this is the time that there's a good deal that comes. There's always a deal somewhere and it just depends. But then you have to think about this, like is real estate going to go down? I mean, it could go down massively. I don't know. I thought we were going to have a much harder landing and we could still have it. But for right now, with the rates that is as much as they are, although I will say, you know, rates, what 5.25% and then you're looking at 6 and a quarter, whatever it is now, between 6 and 7% for APR for mortgages, that's high, but it's not as high as like say the early 80s. You're looking at 13, 14, 15%. So like it's still low in that context, but not in the, you know, two and a two and three quarters, 3.25% APYs that we were getting for mortgages, not more than, you know, two, three years ago.  Are you doing short-term rentals? You're doing long-term rentals. What is the market like in your area? Are your houses overpriced? Are you in New York? Are you in Miami? Are you in Austin? Are you in Los Angeles? Or are you like in a Dubuque, Iowa, or Cleveland, Ohio type place? I mean, they just, and then of course, what's the markets? How are the comparables? And you have to look at a lot of things. So for Mr. Wolf, I can't even ask you to answer that question. It's very tough. You got to do a lot of research. Yes. Ah, that's it. So everybody look for this guy, more crypto online. I think he's got a YouTube channel and he does really good TA. I haven't watched it though. This is meme, I trust her or so. Yeah, Rob, I got a Ledger Nano S plus and it's even better on the regular nose. Great. I just like the Nano Ledger S because it couldn't hold squat. The Nano X can hold a bunch of difference or the software to interact with the crypto projects. It doesn't really actually hold your crypto, obviously. It just allows you to sign off on transactions. That's it. Check out the chart guys, do live trading. Second, day trading won't let you sleep. Exactly, that's why I don't do it. Mullet says it's Russian roulette. That's coming from a trader. Insane clown posse is also really just kidding. Internet computer protocol, really cool. DC and ICP. Let's see. What do you think about Algorand? I think it's massively underperforming right now. But I don't know. Makali, the person that's the head of the foundation seems to be quite a smart individual. Looks like they're going in the right direction, but it really just depends on adoption. Who's going to use it? Who's going to make it work? You know, who was it that was on the show? The Mooch, Scare Moochie, him and his Skybridge Capital. They invested like, I want to say it was a quarter of a billion dollars in Algorand. And they said, yeah, we did our due diligence. It looks pretty good and fantastic. And I was like, oh, interesting. Of course, they also did their due diligence in FTX and we see how that worked out. So who knows? I can't tell you. So can't get it right. It's got a great question. Rob, do you really think the bottom is in? What's your opinion? I thought the bottom wasn't in, but it seemed like it's funny because everybody said the bottom was in in June. That's 17.6. And I was like, okay, bottom's in. A lot of TA people, a lot of people look at charts and all those things. And I was like, I don't think it's in. I think there's something to happen. And then it went down to 15.6 or 15.7 somewhere around there in November, 2022. And then people said, that's the bottom. And I've been saying that I don't think it's the bottom still. And so far I've been wrong on that one. So it seemed like everybody's been wrong on this one. So not for sure. I will just say, let me show you. Yeah, I will say one thing. And that's like, if we're just looking at cycles, so like from the top to the bottom, because remember we're talking about four year cycles. 2012 was a halving, 2013 all-time high, 14 is a reset, 2015 is a, well, 450 is a reset. 2016 is a halving, 2017 all-time high, 18 is a massive dip, and 2019 is a reset. 2020 was a halving, 2021 was an all-time high, 2022 was a monstrous dip, and 2023 we reset. So from the cycles, from the top of the cycle in 2013 to the bottom in that range, 2015, it was an 85% drop of Bitcoin. And cycle two from 2017, 2018 took about a year, it was 84%. And our last one, 2021, we thought, when I did this, we thought it was 19,047. That's only, what's 71% of my only answer, that's it. And then of course, we hit 15,742 in 9th of November, and that's a 77% drop from 67,000. And I still see it going like 10 or 12K, but I've been wrong. So we'll see what happens. I could be wrong, but does that mean that I'm gonna stop dollar-cost averaging? No, I just have a theory. So I actually stopped, and we did a video about this two weeks ago, matter of fact. So since May of 2022, I've been doing what's called micro-DC-ing. So instead of spending like a couple hundred bucks a day on Bitcoin, I'll just put 20 or 30 bucks, maybe 40 bucks a day on Bitcoin, or 10 bucks a day, it doesn't matter. And I was doing what's called micro-DC-ing. Then I was buying some alts, some Ethereum, and some Chainlink, and some Cardano, some Polkadot, and stuff like that. But fractions of what I did. Then two weeks ago, I said, you know what?  Because I feel like since we're in 2023, I still live in the four-year cycles. And I think that this is the reset here. So if we go down, great, fantastic. That means that I can get some really cheap alts in Bitcoin, but we're gonna have a having next year, right? That's a guarantee, 2024. I think in March or April, somewhere around there. And usually that is a price appreciation type of thing. So I got another year to go. And then after that, maybe it's an all-time high, 2025. Maybe it's 2026, I don't know, even better. So for me, I've been, since two weeks ago, I've been just doing the same thing, dollar cost averaging again. So if I'm right and things go lower, great. And if I'm wrong, well, whatever, not too bad. I guess I could just still accumulate for two more years. Not a big deal. Rob Blaze, why are the gift card rewards always still on the Sweatcoin app? Is there a time the rewards refresh like midnight's the time? That's a great question, Rob. Let's take this up to Twitter, shall we? Great. So let's ask, hey, Sweat, Economy, and Oleg, who is CIO, C-M-O, C-E-O, one of those two. Why are there no gift cards? Rob, hold on, Baves wants to know. There it is. Okay. Now let's find out. I don't know, man. So, all right. Band is interesting. It's another article, I believe, and it kind of competes with Chainlink, but in a different way. Could be great. I just trade Sharts, Rolls-Royce, or Rolls-A-Toe, but there doesn't matter. That's good. Hey, Rob, will Ada Burn token sooner or are they? Um, Charles Hoskinson has been very against that, but I think the bigger question is, I need to read up on this, is there chewing something for staking, for the dynamic? I forgot what the term was. Where they're looking at some kind of registration, intermittent registration of something for stakers. I don't know. Watch Charles Hoskinson. It's just an idea, I don't think it's a proposal even yet. Okay. Jesse Powell's tweet just now makes good sense. Wonder what he said. Let's see. Rob, Swan, primarily, like others say, the Bitcoin is not crypto. We need to start to program ourselves to start understanding the difference. Hmm. Oh, because, well, I can see, I'm sure there's a reason for that. If there's gonna call it like an asset and not a cryptocurrency, but that was the whole thing. Bitcoin, as was written in the Satoshi Nakamoto paper, and if you've never read it, it's only nine pages. We've done this a couple of times in the channel already. It's a peer-to-peer transaction model. That's what it was created for. Didn't say anything about a store of value. Didn't say, I mean, peer-to-peer transactions. That's pretty much what it was created for. Now we're, you know, we'll fill in the blank. So I don't know. Yeah. No more Manhattan penthouse and Hamphons for Mishinsky. Probably not. Probably not.  And I gotta stop that. I'm gonna stroke out one of these days. Loans are much bigger than the 80s. I can see that. Let's see. What? Hey Rob, I ran into an old Indigo video viewers in the DGen channel. If you're looking for like the more risky stuff, go check my channel, Dan Degen. And we haven't talked about any projects in sweat coin matter of fact, but we will be talking about this project here. It's a game that I'm doing all my work on called Alaska Gold Rush. And it's a play to earn type of game. Interesting dynamics and tokenomics. But it's actually, they have a working game that's gonna come out in like two weeks. I thought it was pretty interesting. So that would be that. So yeah. Anyhow, have you gone back and used the platform? It's been live and earning rewards. Is it Avogache? Or is it against Akushi? We did two and those were the two that I remember. Yeah, CryptoJeb. I've heard this is good work for TA stuff. Yeah, Eric Crown. Eric Crown or Tom Crown? I just know Tom. Tom, Tom. Ooh, Mooch bought it a dollar for Algorand? I gotta tell you, man, I had to say anything bad, but well, I guess it's all long-term play. What do I know, right?   That's okay. I mean, look, some of us bought Bitcoin at the peak and we're down. It's a bummer. That's just how it goes. So let's see. Amy didn't miss much. I just said how over 14 years history, seems like we're growing as far as Bitcoin wallet use. And then we took a look at who's in the money, who's out of the money. And Bitcoin and Ethereum and Polygon look pretty good. And Algorand, Cardano and some other ones look pretty bad. But we also took a look at Wales and some projects look fantastic and some like Shibino look totally jacked up. And that's about it. See, there's all, Alicia says, I feel like I missed the chance to buy the bottom. So I just DCA and wait for a better bottom or DCA forever. You have to DCA forever. If things go like how I think they're gonna go, I don't see us going, like nothing goes down forever and nothing goes up forever. And with the things that are moving along, I just don't see how we don't have all-time highs within the next couple of years. And that's kind of reasonable, I think. Let's see. Yeah, Gareth Salloway is still expecting a dip. The only thing about Gareth Salloway is as much as he was a bear, he was never a jerk about it. He's just kind of like, he's like, you know, I'm sorry, but this is how I see it. I'm still, you know, bullish long-term but short-term, it's looking pretty awful. And he was right. I don't know if he's gonna be right again, but we'll find out. People making millions with tokens and trying to say they're just software. Are you kidding me? Look, Microsoft did pretty well with just software. I'm just saying. Okay, that's a good question. Hey, Rob, let's say I can only invest $20 a month to save by the top 10 coin or some S coins. These days, it might even be treasuries, T-bills. I know it's kind of a weird thing to say on a crypto channel, but if you're looking for the safest thing, I mean, some short-term, short-term treasuries began pretty recently. And yeah, safe. But for me, if it's me and I have 20 bucks, I'll probably just put it on the safest bet in the most volatile market out there, which is probably Bitcoin. That's it.  That crazy Swedish guy would be CTO Larson. He's funny. Had the show a couple of times. Very funny guy. A Pfizer, I can't, I have no idea. I'm not a big follower of that stock. No, that's an interesting comment. And where'd it go? Someone said, biometrics are the next step for cryptocurrency. That'd be interesting to see. Charles said, I'm guessing we're talking about Charles Hosinton, maybe, said to allow apps to do regulated stuff. It will acquire KYC, and I voted in by the community, SBO stake pool operators such as myself could choose that they want to run it. It's an interesting concept. I don't know if that'll work. It can work, but you have to really come out and say, well, if you're in America, and I have to register, because America is not so much the land of the financially free, that's for sure. Home of the brave, if you wanna go up against Gary in the SEC. And that's it. Diesel Kane is here, though. Any thoughts on your thoughts regarding Grayscale? Apparently, they're gonna pull it out. Digital currency group is looking to sell off Genesis. So, I mean, looks like they're going to make it out on Skate. I thought that the contagion with FTX was going to run all over into Grayscale. Coin Desk was also part of that group when they were looking to sell. Don't think they sold it yet. So maybe I was wrong on that one, too. Maybe I was just a little bit too bearish, I guess. But again, it doesn't matter if I'm, for me personally, if I'm right or if I'm wrong. If I'm right and we go down to a Bitcoin 10K, that means all it's gonna crash even more. That means that I buy, get a more frame. And even as it starts to go down, I do this thing called Dynamic DSA and I will buy even more of a percentage than what I do now. Because I think in the next two or three years, we're gonna see some all-time highs. And then when am I gonna sell that crypto? Because here's another thing. And then after this, we'll take off. But the thing is it's easy to buy. It's easy to buy the dip, it's easy, it's easy. I mean, for people who've been here a long time, when you're just getting into it, it's a little bit harder first, but it becomes just normal over the years. It's easiest to buy and buy and buy. The hardest thing is to sell. Ask traders, sometimes they don't sell fast enough. So there's a link in the description. It says when I'm going, and it's at the very top of all the videos I do now. It says when I'm gonna sell, when and why I'm going to sell 80% of my crypto. Watch that video. There's a lot of indicators to look out for. And that's it. And then along the way, maybe diversify a little bit. Maybe get into treasuries, maybe get into stocks, maybe get into real estate. Maybe you roll it into your own business, do whatever it is, invest in yourself. That's the one thing that's saved me in the crappy bear market of last time. Rolled the profits into real estate, worked out okay. But I didn't, what a hurt. So just remember that. Anyhow, that's it for today, everybody. So look, thanks so much for hanging out with me on a Sunday, almost a thousand people. I appreciate you guys. If you liked today's video, apparently YouTube likes this like button. So hit that thing before you leave. Consider subscribing. We do this all the time. And lastly, I know everybody misses the live streams. Just gets to be too much. But I have a plan to roll all of us, this community into some other, some other community platform where we can all communicate away from here. So if that sounds good, say something in the comments. But that's it. But thanks so much for stopping by. I appreciate it. See you guys in the next one. Adios. Good to do.


Oh, that was a good counter. Literally the best answer. My man just punched the Krillin in his face. How's it going everybody? Dachodoy here back with some more Dragon Ball FighterZ rank matches. This time, after a week or two of not playing them, we have gone back to the main team, basically the Kame team, consisting of Krillin leading the team off, Super Saiyan Goku and Yoncha. I don't know why I'm so excited to finally be playing this team again. It's been like what, a couple of weeks at this point? We played Krillin in that worst characters video, but, you know, you need the full experience. You need Krillin with the Beamus. Wow, it got taken out of 6M. The Beamus has got taken out immediately. Alright, knockdown in the corner. What's the mix? Ah, not the mix I was hoping for. Oh, we're getting Vigito. You can only open you up with a 6M in a system. That's Vigito's bread and butter, by the way. Just keep blocking low and eventually he'll try to 6M you and if you can reflect that, you're good to go and we're snagged and we're getting Beamus system. Alright, we're out. Yoncha... I didn't get to do the EX reversal. Tried to get the reversal, tried to use the Yoncha player signal. It didn't work out. We're probably... nah, we're good. Oh? How did you let this happen, bro? No! Kojima's health bar just got erased without him ever setting foot on the field. Good counter. That was a great answer. Literally the best answer. My man just punched the Krillin in his face. Oh, that was unfortunate. One, two. Alright, that should do enough. I'm hoping it does enough. I probably should have just gone the safer route of having everybody doing super. Alright, that was just barely enough, Krillin. The amount of destructor this guy threw almost just took me out of the game. They blocked that. Watch out for Zamasu. Watch out for 6M. Zamasu! I didn't really need that beam. Take the Vanish. Oh, why did I... I reflected the hit, but got hit by the beam? Damn. Okay, I don't think we want to go for anything too sketchy here. I'll take it. Krillin, get out of here. Level three? You mean mixed city? Let's go. One invitation, dude. I think I'm gonna fake it, too. Okay, there he goes. 6M? Alright, defense. There it is. Oh, he even stepped up before he did it. He even dashed at me to scare me. He tried to scare me into mashing, but then did the thing that if I mashed it would have been... That was like a... That was like... Man, that was a layers of mix. Keep him there. There we go. That's a hit. That's the hit I'm looking for. Keep him in the corner. Main team privilege. That 2M was the worst 2M. I wanted the... Oh! Alright, we take those. I can't sweat. I can't... Alright. I tried to do a level one with Krillin just because it does a little more damage, but it wasn't happening, so I'm glad I was able to alternate out. That was close. That was close. Man, I love this team. This team just moves, dude. Krillin might be low tier, but he can really get the results. If you're fast enough, he's got that 2L that's great afterimage is a fun tool to use. Alright, I didn't... That was a... Huh, I've never gotten hit by that part of the sword, down tech. That sucked. 6M, there's it.  It's the one thing they have. It really is the one tool Vigito has to open people up. It's not even about... It's not even about the player. It's just that's what Vigitos have to do. They can faint as much as they want, but when it comes down to it, they got a Dragon Rush or 6M. Why did I... So, as you can see, even though I can think about that, even though I think about defense, I am not capable of performing. Well, that was a bad start. Uh, what's the... I mean, if we get another Double Sunday, I'd be all four. Alright, let's just level three. Let's try to get back to a two character game. I feel a lot better about that. Go with the 2M. Go with the 2M. It always works, baby. It never fails. 2M, I wish we could kill here, but it just doesn't look like it's going to be possible. I'll take that every day of the week. Optimize it. No, I dropped the last part. The last part. The last part, dude. No, no, I can't believe this happening. Oh, we had it. I could have even switched him out if that was what I preferred. Dark Ansel, thankfully it works. And a big hit through the Vigito assist. We don't even care about that. Alright, I think he reflects here with it. I regret going for that quarter circle forward now. Didn't think we'd get the hits. Those to the other side. 6M as well. All where the one's getting Double Sunday. Alright, thankfully it drops, but still we're probably dead. Emesis? We're probably living. Maybe. I don't know anymore. The calculations are getting rusty, dude. The calculations are past the point of being accurate. Yeah, we lived that. Nope. Alright, well we recovered pretty fast. Alright, that's really nice. Hurry up and kill this guy so we can get that blue health. I, hurry, we're not gonna get any blue health. Die Sculpture. We got a little bit, baby. Little bit of blue is better than nothing. Unfortunately, he still has Sparking, which is the one big defining factor between us, but. Oh no. Dark Ansel, thankfully. I can't believe it. If we've got, he didn't bite. He didn't bite on that DP. No, Die S, don't drop it and it was dropped. Properly dropped. Kick. Double kick. Alright, fine, I'll take it. Not the cleanest. Ha, ha, ha, ha. It's just yelling my move names at this point. Kick, double kick. Health, don't take.  That's a plus for him. Oh, wow, he'd be in there. What? Super Dash, if you want to. I don't blame you. I don't blame you for Super Dashing. In fact, I would love to see more of it in my future. I'm gonna take a sliding here. You miss this for help? Yeah, we'll thank Fist. Probably. I'm trying to keep that safe, but it. I don't blame you for Super Dashing. I'm out, I'm out. You played around for too long.  Jokes aside though, all the sparking is gone, which means it's free reign. We defended it for long enough. That's probably game. That's probably the game right there. Okay, please kill one bar, one kill, two rounds in. Man, we are getting pretty high up in Rosé. Well, we're at 50,000. We're not, it's not like we're close to ranking up, but we are getting up there. Just these point gains just gotta keep on drippin' in. Please train me. Damn, with the Crility Votes, who? I gotta really set up some emails, dude. Ha, ha, ha. I'm looking for that Spirit Sword, dude. Oh, Yamcha. Nope, all right, didn't fall for that. No, I can only block so much. Thank you, thank you, Yamcha. I can only block so much, let Krillin do the rest. Let me go to work, let me go to work. Mix city. What's the mix.net? Oh, here, let's go to Krillin. All right, canceled, just keep the pressure up. Go! Oh, we would have taken those if it worked. You can only block so much, dude. I know your secrets. Oh, man, why that reflect was active. All right, do we eat a bean? I mean, if you eat this with a level three, we're eating beans, bro. You know what, I'm gonna eat a bean right now. Whoop, give me that. Don't talk to me till I have my morning bean, my morning breakfast. All right, well, the bean health is mostly gone now. Guard canceled, no blocks. I'm expecting a tag. All right, we gotta get Krillin out. His assist is worthless. Krillin has returned. No down heavy. It didn't hit. Medium slicer for the sliding. Let me out. Can I just please leap? I'll do anything to leave, bro. Okay, anything to leave, looking out. All right, we have to spend a bar, but I'm fine with that. We're out, Goku Black is gone. He can't bully us anymore. 2020, we're gonna stop the bullying, I swear. All right, again, this is not Krillin, so. Really? Wow, that was potent. Got him. Oh, come on, dude. Follow through. He can only do it if he hits, right? I'm not gonna guard cancel this time. I've done it too much. I'll acknowledge that. I've done it too much. Please stop with the pressure. Oh my God. That was really nice. That was really nice. I can't hate on that. I panicked. I had seen so many kicks. I knew he was gonna EX one of them. Just bought me over the head. Wow, okay, he has the lead. He did not do a lot there. Job choo! Best assist in the game, free. Except for maybe Kid Boos and Goku GTs, but other than that, free. Where's the competition? And the beam assist. He tried hitting me. Wow! I don't know why it didn't tech then, but whatever. You won't see me crying about it.  I should have 2M'd again. I didn't do it because he didn't block it last time. All right, hold on. I got Reads. Watch this. All right, well, the Reads were there. Perfectly protecting myself. No, can't punish that. Countcha, he blasted. Keeper met Bay, make him wanna vanish.  Oh, I'm so bad at this game. Oh wait, I'm great at this game. Oh, I'm great at this game. Not that good though. Not that good. Not that good. He's not stopping the Super Dash. Who could? Nobody can stop the Super Dash. I'ma go one more. I don't know why I'm so confident all of a sudden. I'ma go one more. One more time. Yeah, I had already come to the conclusion. Wow, that's the first time I've ever decided before then. No, oh, all right, this man is getting robbed. This game does not want him to win, unfortunately. I thought the mix was gonna be there for me. Probably should have just gone with it and then seen what happened afterwards, but you know, hindsight is 20-20. Oh, he blocked it. I've guard canceled so much. I can't be upset at the first time it gets punished. Is he gonna push me down? Okay. Understandable. It's already sparking time, by the way. Guard canceled again, which unfortunately makes it so we can't heal with sparking, but he's gonna be all right. Level three, hope it gets the full damage. I wonder if he sparks here. I don't think so. Nah, he didn't. Again, that low is proven. He does not wanna block on wakeup, at least not low. He might have been blocking high that time, but we'll never know. Whoa, I can't believe he did that. Passed through the beam. No fear. How's that Amar? Oh, pretty dead then. Sounds cool. Level three, I'm gonna block. He might go for a command grab though. I don't know why I got that feeling, but he might do it. Nope, might have been looking for a guard cancel. You know what, hold on. Let me get this beam assist out of the way. Come here. Wow, really? Oh my God, yo, he's brave. I gotta do it. Well, this ended up sucking pretty hard. Try jumping.  Now, Tech, no, we're getting comboed still. Still getting comboed. Medium, no, I wonder if you can medium slice it there. Oh, let's go. Really get the beam waiting for me, dude. Nice. Dragon rush after dragon rush, baby. Struck to the disc. Two bars, I'm cool with it. I'm chilling. This is the point character for my team, so. He's no use to me anywhere else. 6M always works. Guaranteed. Three bars, gotta spend it. At this point, I think he will spark, but I'm not gonna make that read. Again, the EX grab is too, I mean, EX uppercut, DP. I wonder if he sparks for the kill. Well, no spark, that means the game is probably over. Yep, it's over. Slice, that is 4-0, we take it. I feel pretty good about it. First return to the team in a little while, and it's still just as much fun as ever. Thank you guys so much for watching. Make sure to go down below. Let me know what your main team in Dragon Ball Fighters is, why you're down there, and also make sure to hit the like button and subscribe if you enjoy the stuff we do here on the channel. You can also check out the videos on top of your screen if you want some more right away, and just give any of those a watch if they catch your eye. Other than that, I've been Dr. Doya. You can pick up the holiday shirt in Teespring. The links should be down below as always. I have been Dr. Doya. Thanks again for making it to the end of the video, and I will see you in the next one.


Okay, so hi everyone. I'm really glad to see you here. It's four in the evening on Saturday, so I think a lot of people started getting out of here, so I'm really glad to hear that many people. And today I will try to talk to you about backups of TEF, because we know that there are two kinds of people, right? Those who do backups and those who don't. And actually there's a third group of people, those who do backups but don't know how to restore them. Yeah, so always remember to also try to restore your backups. And my talk will be mostly about our experience at OVH, the company I'm working in, and the story how we get to current point in time. So my name is Bartek Święski and my daily job is working at OVH at Wroclaw in Poland, where we have research and development center. And my current focus is on bringing awesomeness to our self-infrastructure. So quick show of hands, who does not know TEF? Okay, few people. Who does not know OVH? Okay, also few people. So very quickly about TEF itself, we've seen few introductions already, but this is an open source project. The most important fact is open source. Right now, developed under Abra of Red Hat. It's meant to be a network storage with different access methods. And it should be perfect solution for large scale deployment, so it should scale out very, very easily. It should be reliable. So if there's any problem, it should detect it. And once it detects it, it should be able to recover from the situation by guarantee that it heals the data that is lost. And last but not least, it's very fast. So when you take a look at the history of TEF, they put really a lot of effort into making this very fast platform and squeeze out the last bit of performance from your hardware. And that's why actually we like it at OVH, because we want to have fast things. And basically, how much do we like it? We like it because we have 40 petabytes of raw hard drive storage that we need to manage somehow. And it's growing. And it's funny because last time I was presenting, we had 30 petabytes. And it was, I think, half a year ago. So the growth is really fast. And it is a lot. It is really a lot. And we don't put everything into one cluster because we split it into smaller parts. Right now, we operate around 150 clusters in our infrastructures. Those are both our internals and external ones. And our base workflow are the images in OpenStack. And that's where we put focus on our backup infrastructure. So I just said that TEF is reliable. Right? It will heal itself. It would replicate data. I didn't say that, but it will replicate data and also consider your physical infrastructure. So we'll make sure that it's replicated, for instance, across racks, not inside one server. So if we have all these benefits of TEF, why do we need to backup? We have three copies. Can't we consider it backup? We could, but yeah, better save than sorry. So we always can find some kind of software backs. We don't know when it would strike bar. And we haven't yet seen something that could destroy our data, but we know that backs happen. Right? And it's better to be prepared for that kind of situation. And backs can happen in SEF, let's say, but also in the lower parts of the stack. So let's say that there's a bug in the Linux kernel. Something in the file system layer. And it starts raising data. We want to be protected from that kind of situation. Also, there is a software running on our hard drives. And it can happen that it sometimes just behaves differently than what it was expected to do. So that's one thing. The second thing is that we want to be prepared for disaster. And when we are working at the scale, like at OVH scale, the probability is top getting intuitive, like the probability of hard drive failure. So if you have laptop, if you have desktop, you can expect that you will replace your hard drive in a few years, right? Because for two years, the hard drive should work. When we have a few thousand hard drives, you can expect that hard drives will be dying daily. And remember, we have three copies. So if three hard drives die at the same time, we just lost our data, right? It's very, very unlikely. But we already had a situation where two hard drives died at the same time. And Ceph is very good at that situation because when you lose two copies out of three, it just blocks traffic to the cluster. Our clients would complain, but it would start to copy the data as fast as it can. So it has at least two copies. That's the usual setup. So Ceph helped here, but as I said before, better saved than sorry. Also we've seen because we are using Firestore. That's, let's say, original data layer for Ceph. And it uses XFS Firesystem, which tends to have problems where there is a power failure.  So that means that if we have big power failure and we know that big power failures happen, we may have problems, right? Hopefully we didn't have this yet. But let's be prepared. But when you take a look at the statistics, what's the biggest factor for losing data? It's always human, right? Always operators, right? So let's be prepared for a situation where someone has, let's say, very hard duty and just types something, mistypes in the common line. And then you have this feeling that you just feel cold because you just did something wrong, right? We want to be protected for that. And also because we are giving services for our clients. I can assure you that sooner or later there will be someone that came to you and say, I've lost my data. Please, please, can you recover it for me? Because even if we do backups, not always our clients do, right? And it will be very good to be able to help that kind of client in a situation, especially if he says I will give a lot of money for this, right? Yeah. Also, when we talk about SEF, SEF works very well when it's in a local territory where not in the cluster can quickly connect together. So if we start putting data in the data center very far away, the pink cross, SEF has performance issues and it's not yet fully ready to handle this kind of situation, geographical replication. And we, of course, know that the best backup is a backup somewhere far away, right? I'm not talking about Mars here, but the other side of the globe will be the best way. So we need to somehow overcome this and be prepared also for this kind of scenario. And I see that inside SEF itself, there is a lot of effort to bring this kind of geographical replication backup. Right now you can do this for RBDs, but it's very early and it doesn't yet scale well. So we have to have something instead, right? We take, we try to take a look into this and we see also that there's an effort to backup also Rados player if someone is interested. So I've seen some rumors, so it will be very, very good solution in the future, but we need something for today. So we know what we want, so let's do some planning. So we started planning out of VH what we want to do with our backups. So let's first think about backup software. We need to have some kind of software that will backup our infrastructure and we made few assumptions. What do we want? Yeah, we want something that would do compression. Why? Because if we compress data, we will need less backup storage, right? Less is mean cheaper and of course we can upload it faster. Also we would like to have some kind of deduplication. Why? Because sometimes we will be backing up something a few times in a row, right? So we have our one RBD image. The next time we backup it, we can compare it with the previous content and there will be only a small difference between those two. Also we want to have encryption. That was one of our main goals. So we don't want anything to leave our clusters unencrypted. This is custom data. So even if we're talking inside our VH network, right? We want to keep the data of the clients only inside our clusters. So everything that leaves that must be encrypted. Also we want something that is speed because we have a lot of data to backup. And because exporting data from SEV is using RBD export, we want to be able to process stream of data. Now the file is just a stream of data. Just send it to the backup and everything should work. And we also wanted to support open stack Swift. That's a funny thing because we already have open stack Swift of VH and this is totally separate thing from SEV. This is separate code. This is separate team. So if something wrong happens in SEV, whether this will be a SEV back or something in our management infrastructure, the separate team should still be fine. So even if in our case we just broke everything, other team should have the data. Yeah, and we didn't find that kind of solution unfortunately. So instead we tried to look at VH what experience do we have? And it turned out that there is a team that is doing something similar, backups of RBD images and they are using something called duplicity which is an open source software to basically do backups. And we decided to use this because we already have knowledge how this works and what are quirks. There were some things that they said initially even that we have to consider. But still we are also analyzing alternatives. There is very nice promising project called Rastic. It's still pretty young and it doesn't have compression but it looks like it may be our replacement in the future. Okay, so we selected software. Let's think about resources. So we have to do some assumptions because we don't know what kind of data our clients have. So we have some 30%. So if we compress the data, they will duplicate it. Let's say that we left with only 30% of original data. Also we want to reuse as much of our internal infrastructure as possible. So as I said already, we used the internal of VH Swift cluster. And also we knew that we will need compute powers because we need to do this compression, the duplication. So we decided to use just of VH cloud where you spawn view to our machines and you can very easily spawn new ones, destroy ones. So that seems very flexible and if there are any problems with resources, there are dedicated teams to handle those problems. So it's less burden for us. Okay, so let's also do some example calculations. At that time we had 20 petabyte of raw data. We have three copies. So when we divide one by 20 by three, we have 6.6 petabyte of data. And let's say that we want to backup this daily. So let's just divide the numbers. 281 gigabytes per hour to backup. So 4.7 gigabytes per minute, 0.078 gigabytes per second. So throughput of around 0.63 gigabytes per second. So it doesn't sound that bad, right? We usually operate on 40 gigabit nicks. So we should be able to handle this. Similarly we calculated this for raw hard drive speeds. You can calculate how fast it can work. And results were that we can basically backup everything in a few hours if we are happy. And this is initial design. So we have Swift cluster. Of course for the image we need to create a snapshot. So to have it in kind of consistent state. Because we don't want to backup something that's changing. Right, then we export this stuff through some kind of backup virtual machine. Also we containerize some of our infrastructure to be separate from differences between self-versions and to have local storage cleaned up automatically. We put this into duplicity and then throw out to our Swift storage. So it looks fine. So let's see how it works in practice. So we started implementing it. And very soon we started getting challenges. Okay, so first thing to hammer is duplicity. Our selection of the software has some limits. First limit is it doesn't work with data streams. It just works with files. So what we do? We take the image, just back up it locally and export this local copy of data to duplicity. We have backup VMs. And we have local storage of let's say 500 gigabytes at most. But some of our clients have 2 terabytes of images. Can we handle this? We thought that we can use whole punching in Linux kernel. So if we dump something into the local file system it has a lot of zeros. We will just eliminate those zeros and maybe we will be lucky to just put those gigabits into much lower space. But no, it didn't work out. So we actually, I think I just moved to the next problem, but we also have problems with duplicity because it doesn't tolerate large files. If you go beyond, let's say, a few hundred megabytes, our sync algorithm that's used inside duplicity start having some performance issues. So you just have CPUs at the top. Just wait for the backup. A few hours later I used to wait for the backup and nothing works. So our colleagues at OVH said that you need to split. You need to split, you need to chop into smaller pieces. Only you can backup only smaller pieces. And that's what we started doing. So when we dump stuff onto local hard drive so that duplicity can backup it, we just split into 256 megabyte chunks. It turned out to be pretty good. We don't have to split into lower amounts. Duplicity was able to handle this size pretty well. And the problem that I just talked before, we have large, large images. We have basically limited local space. So we also need to split the whole image into chunks. We call these chunks and backup such a large image in parts. Each part separately. So it took us some time to implement and fix all the issues. And basically that's the architecture. So we start with self-cluster and RBDImage and Snapshot. We just cut it into pieces. That's the first layer of the cut. Out of this cut we have 25 gigabyte chunks. And then we have to split those chunks into files. Each file is 266 megabytes. Put it into local SSD drive and then we can put it out into Duplicity and finally go to Swift. Yeah, so we made it. It took us a few months. But this is right now what's working on our production. And it works really well. So really we don't have to care what, how large images our clients have. And I've seen 30 terabytes once. So it is pretty, pretty large. And we can handle you with this system. Of course the time needed to backup will be proportionally bigger. But it works. Okay, but we also thought about some alternatives. Yeah, we have a lot of splitting here, a lot of files here, some local storage here. Maybe we could do better. And yes, we can. Yeah, fuse. File system in user space. So very, very nice mechanism in line scanning world. So basically you create user space application that just do a virtual file system somewhere in a directory. So if we create this kind of fuse client, what we can do? We can do all we did previously with files and with this chopping. So we can just initially select only portion of the image. Then internally inside the memory, chop it into smaller pieces and expose this as already chopped series of images. And just put it through Duplicity. It should take it as if it was just standard files. So we made this client right now it's in testing. It's very promising because basically we don't need any local space. So what we take from the cluster, the fuse client just sucks it into the memory. Then when the client request the data, it just throws it as directly to the some Linux kernel world, I don't know where. Basically it just travels then back to the Duplicity and it sends it back. Yeah, backing up was pretty simple. We had some problems with restoring data. Yeah, because Duplicity has its own rules, right? So for instance when you start restoring stuff in Duplicity, it just raises everything at the directory of the destination. So you need to pretend that you handle removals. And then when files start to appear, you just need to handle the situation. And of course keep in mind that you cannot show that there are some files where they are still not uploaded yet. So tricky situation but we have a very bright colleague and he implemented this. And we plan to send this into our production pretty soon just to remove the need for local SSD space because that's currently one of our limiting factors. Because we need just a lot of VMs with reasonable amount size of local hard drive space. Okay, so let's talk also about something else impact of our production because when we start reading data from our clusters, we can expect that it will affect our clients. So from the beginning we knew that we have to throttle the backup system. We have to be able to very easily shape the amount of data that falls through it. That's why we implemented few layers where we can basically just have this vault and rotate it and even put mod into backup or just reduce it. So one of that vaults was made because we used salary tasks. So salary is the task processing architectures. It has workers. So the amount of workers which we have spawned is basically a number of simultaneous backups we have in the system. So by just spawning more workers or reducing the amount of workers, we can easily just do more or less prior backups in whole system. But still we want to protect one cluster. So we implemented some limits. For example, we can set up in one cluster at a time. We also want to limit the time when we back up. So when we schedule new backups, for instance we can say schedule this only during night because there is a clients don't do much on this cluster. Also when we have those backups virtual machines. So we have a virtual machine that has let's say 200 gigabyte hard drive. And we need to put in backups there each of 25 gigabytes because that's the chunk size. So we don't want to cross some limits here. So each backup VM at one time can accept no more than some calculated value. Some calculated number of backups. And for one image we decided that there must be only one backup. So we have to either finish the backup or just cancel it. It just simplified everything. And all those limits we implemented using ZooKeeper and some semaphors and logs which are built on top of ZooKeeper basically. Ok, so let's see how it scales. Of course we have issues. And Skyling always shows some issues. Even if we test it locally, if we try to do some simulation you can never predict what will happen at a large scale at a production. So ZooKeeper our decision sometimes has a problem because ZooKeeper is good for rarely changing data. So it will distribute the data in its own cluster but when you start doing frequent changes once in a while there will be a snapshot. The snapshot takes a hard drive space so if you do a lot of changes and acquiring logs and semaphors do a lot of changes then you have alerts that run out of space. Because there are a lot of snapshots and what we did we just increased the number of transactions after which ZooKeeper would do a snapshot. And basically that fixed our problems with ZooKeeper. Also it turned out that we have problems with solar workers because it has some problems with memory. Sometimes just memory spikes also has a problem with CPU when there are connectivity issues and we put this into virtual machines so in virtual machines you don't have a lot of CPU characteristics. So sometimes you have a lot of CPUs, sometimes you just saturate it very quickly because there's some other VM that just tries to do something CPU intensive and things like solar doesn't like it because you have to respond to pings in the network and when you don't get a ping in a reasonable time you say okay this guy is dead just kick him out. And we had situations where we have half of our solar workers dead. So what we did, we just increased timeout for pings and right now it works reasonably well but still we look forward into a new version of solar to fix stuff. Also we have problems with Docker because we want to control everything to clean up automatically so it turns out like a lot of local hard drive activity you put a lot of data inside Docker sometimes it doesn't clean up and you just run out of local space. So this is something that we fight currently with and also duplicity has problems with scale. So it turns out that when you do backup you can do it reasonably fast but when you do restore it needs four times more time to do it because of how it's written. So we started looking for something better and we come up with code hold backup strategy and okay so let's think what better we can do and this is basically something that we should start with. What's the fastest way to backup a CF cluster? It's to use another CF cluster because CF is fast, right? So if you backup CF on self it should be fast and fast, so just fast, right? Okay so what do we do? We just create a separate cluster that is supposed to have just a copy of original one and just be dedicated to a backup, right? We also thought that there is a nice feature RBD export if, RBD import if, so basic something that so basic in CF you can just say export me only the changes from given snapshot put somewhere in the time and just trust me those changes yeah so we thought yeah there must be something in it and intuitively we said that we may have benefits here. I will show later yes we had. Also there is one nice very nice property which was very crucial for us. If you have this kind of spare cluster with data of let's say thousands of virtual machines if we have disaster and our original cluster dies what we do? We just take the spare backup cluster and just plug it into OpenStack and just start firing VMs back so you can very quickly recover. We don't have to just copy the data back to somewhere. Just take the cluster that we already have. Also we thought that we can use our previous architecture so previous code that we did to just chop things, manage this have these throttles everywhere we thought that we can easily use the same code and when we have this backup cluster and we use what we did before so just take the hot backup put it through duplicity we have two levels original cluster, hot backup cluster and then called duplicity so we still have this separate software separate software solution, separate team we still have the data built somewhere and if we just break everything there is a separate team that should have data of our clients. The architecture is actually simpler than with duplicity so what we do take original cluster do a snapshot of image we also decided to still keep this 25 gigabyte chunks because of that we can easily just when there is a problem network problem we can just restart one part not whole image so there are some advantage over that take the chunk put it through backup container I will talk in a minute what it is and apply it on backup of the image backup container is something that we can very easily spawn next to the backups of cluster we actually didn't need that much of processing power because there is no compression nor the duplication so it just pipe of data from one place to another yeah we have advantages of that solution great advantages it turned out that we can backup very large clusters in less than 24 hours we also don't need that much computing power because we don't do this compression duplication so we just the data just falls through and the recovery is very fast because we can take this backup cluster and use it instead ok so let's talk about how good it does work and let's bring some numbers at OVH ok so first a global statistic right now we actively backup 34 clusters and daily it's about 1000 images and it is still mostly using the duplicity the old swift base solution so that means that we around roughly 0.6 petabytes of data daily so it already works well but still we can do better so this is a case study of one of our biggest clusters which we experimented with we actually this is the first production cluster where we implemented the CEPH on CEPH infrastructure we started with the 3350 images backed up per week so that was something that we could do on this cluster it turned out that we actually wasn't able to backup whole cluster in one week using the first infrastructure because of the largest images the largest images they just needed more time once we implemented CEPH that's the second bar here we still kept this 7 day period and it turned out that we can back up whole cluster just easily within one week so we had 4776 images there and everything was being able to backup in one week and then we shortened the amount of time to do a backup in one day and the performance spikes that's where we implemented the diff strategy so it shows that our intuition was right so if we start exporting diffs only we can take advantage of the fact that actually in urban images there's only a fraction of data that changes and you don't have to reupload all the time everything else so it's around 10x performance improvement to the original implementation one more nice chart so this basically shows the age of the image so if we base age of backup so if we do new backup we just remove old one so it just goes down to the very young one on average our images initially were about 6 days old so many images we were able to actually back up more frequently than one week but still the worst case scenario was around 1.7 once we started implementing theft to theft backups we just reduced this down to around 7 days you can see some spice here that's basically where we were fighting with salary, zookeeper and that's why it doesn't always works perfectly but the most important thing is here we switch to diffs and suddenly we just see a drop that's I think the most important from this talk what you also can see that there is an initial part where we still are on theft level without diffs that's because you actually have to do a first backup first full backup so it just takes few days once we have the first backup done we have snapshot required then we just start actually implementing the diffs start applying diffs instead of full backups and that's where we can expect this reduction of time and similarly this is something that show us if we can keep up with the amount of data so this chart shows the this line shows basically new images that we can backup and once per time window that we want to have we just reduce it to 0 so that means that if we have total amount of 4700 images this line should get up to the top and cross it at some point before we can say that we can backup everything in time so initially for duplicity swift implementation we were not able together once we switch to save if we just forget about this period the trend is something that can easily cross the line but once we add the diffs we can very easily one day just backup everything just to sum up backups at scale are possible we do this we have a lot of data and we can backup it but you should do what self actually gives you just use the export diff and that will give you the fastest way to backup your data and it will you can definitely backup the clusters which are highly utilized in our clusters there is a lot of activity a lot of stuff is going on 24 hours a day and actually every 24 hours you can backup data of your clients that is something that you can easily achieve actually you can even get shorter backup times but you have to watch out for the latency on the cluster because it may start increasing but also even if you have self on just think about some alternative storage because we never know what will happen and we never know what bugs can be there waiting for us in the next few years and the bad thing that we have discovered is that self on self is good as a first line backup and then just do some kind of cold backup at the end so that will be it from my side there are some nice images so this is accreditation and few minutes for questions yeah so the question is if we have a workload like databases do we have any coordination between the stuff that is going inside the VM like dumping the data and our backup system so currently we don't do this because first we want to have the backups ready and working at the 24 hours scale but we discuss this with our colleagues OpenStack team because in OpenStack you can also do backups and it has some ability to freeze local file system do a flush, some kind of sink and at that point do snapshots so we want to explore this in the future not yet implemented but we think about this yeah so the question was about RESTIC and the fact that it supports stream we still consider it and we want to try it in next few months actually the streaming feature is very important because you don't have to have local storage right so that could be one of the benefits of RESTIC at this point in time we just focus on this hot cold strategy and want to have this first and then we will try to improve the cold backend and most likely RESTIC will be a very good candidate there have you considered other candidates than RESTIC have we considered other candidates than RESTIC there were few, right now I cannot recall how many but basically once we heard that other people at OVH are using duplicity we knew that we can use their knowledge so we didn't do very huge research in that regard yep, didn't yeah so currently we do and okay so the question was if we do self-on-self backup do we have equal sized backup at the end or do we have less replicas so currently we are using identical cluster so the same size, the same number of replicas because when we have to do a quick switch in case of failure the backup cluster will be able to handle the traffic but we also consider a situation where you have totally different tailored hardware especially for backup like more hard drives less performance and actually even two replicas out of three maybe because that will be a cost saver right, okay here okay so when we yeah so the question is when we do a chunked backup do we reapplied chunks on the end on the backup clusters yeah so each chunk in our system is a task and each task is responsible for exporting diff only of this chunk we created some small script in Python where you can very easily just export only diff of one chunk and then on the other end we do something similar we just apply something regarding only this chunk and once we have all chunks backed up on the head image then we just can finish the whole backup just do a snapshot at the end yeah we'll still using Zupipe yes yes that's something that we will consider we I think we were thinking about at CD but right now we just solve our major problems and started focusing on increasing our infrastructure so it's something definitely that will be on our roadmap okay that's all I got so thanks for the meeting


What's up, how are you guys today? We haven't done a recipe video in quite a while and that's because although I have a lot of creative ideas, they're not that simple and easy and I honestly just don't wanna spend two, three, four, five hours preparing a lot of this stuff every time. So if you guys do have an idea for something that's simple, creative, preferably meat-based, maybe you can let me know and I'll try to make my quality version of it. Today we're doing something that I've had an idea for a few months now. Now I was looking for stuff to add variety in my diet. I saw a few bean dips at the supermarket and I wasn't a fan of the ingredient list so I came up with this stuff. This is a white bean onion garlic dip that you could use for anything, whether it's bread, tortillas, chips, vegetables. Let's see how it turns out. As usual guys, everything is organic and the reason we do that is to remove as many herbicides, pesticides, those agrochemicals we don't want in our food. Of course, there's still always downsides, the can lining, the plastic, but this is about as good as the average person can get. So earlier, I sauteed up two whole onions and about five cloves of garlic, got a little bit of browning. If you wanna go crazy, you can really caramelize these but if you don't wanna do this, you can probably just use onion and garlic powder although I don't think it will turn out as well. So we're gonna throw that in the food processor. So normally you'll see black beans used in a bean dip and those aren't as good for you as white beans, the black or higher in anti-nutrients. I've tried the cannellini beans as well as the butter beans and I think these are gonna give this a much better texture, they're a bit firmer so when we blend this up, it'll have like a nice creaminess to it. So I'm gonna strain out two cans of the Bio Italia organic butter beans. So just in the strainer and then I like to just rinse them with a little bit of bottled water, nothing too crazy. Now this is a bean dip, so we want the majority of this to be beans for the fiber content, the gut motility, the texture and the flavor. So for some added fat and texture, even more creaminess, I have this organic heavy coconut cream. I think you could just use any sort of coconut milk product, even coconut oil, wherever you have that's organic and high quality and this is really nice and thick so that's why I like this. There's also a condensed coconut cream which might not be a bad idea even if it's a little sweet. So I'll just do about two big tablespoons of that. Just a pinch of organic sugar. I've seen sugar is an ingredient in some of these bean dips and then we'll squeeze in some lemon juice just a bit of acidity and freshness. So that was kind of the basic stuff. Now my little twist to this is adding some of Frankie's steak seasoning which is mushroom, garlic and onion. So that's gonna kind of reinforce the mushroom and garlic flavors we put in there. Add a little bit of salt, some more seasoning, a little bit of pepper. We have our rose salt which we will add about a teaspoon of because we need to season this. Now something to add a lot of amino acids, a lot of nutrition. This is the collagen broth we have on Frankie's syringe meat. This was the main reason I wanted to do this recipe. A very unique idea. Add this gelatinous mixture to it, add a lot of nutrition. That might be a bit too much liquid. If this was cold, the gelatin will set up. So it's a little bit liquidy so I would go lighter on the broth and definitely let this set up in the fridge. That'll get a lot of solider, become more solid. So we just got some bread and the bean dip. All right, let's see how this is. This is pretty delicious. This is pretty good guys. Now what's nice about this is I can have this every day in my current diet. I knew the ingredients would be okay for me to eat. I just wasn't sure how good it was gonna be but this is very good. The fresh caramelized garlic in here is really nice. I mean, you could have this as a soup. These are that good. Very delicious. Very, very delicious. Amazing guys. I really, really like this. If I had my own kitchen at my business, I would definitely start making stuff like this and selling. And I've always wanted some type of like Italian deli set up. I just don't have the logistics and time for this. But in the meantime, grab some bread, make the dip, get some water kefir from Frankie's Syrian Foods and you will feel really, really good and enjoy eating this. It's a nice thing to have before dinner. You don't have a ton of steak in your stomach which can make sleeping hard for some people and slow down digestion. But depending on how much collagen broth you put in there, that does add some amino acids. And as I said earlier, guys, if you put this in the fridge, it will firm up a lot and you have more of like a dip. This is kind of soupy right now. But I'm not that patient. I'm not waiting for tomorrow to show you guys this. So thank you guys for joining me today. You can check out frank-found.com to see all the products we have on the meat website and support me through all of my businesses. Outside of that, if you guys can please drop a like on the video, leave a comment down below. Subscribe so that YouTube can unsubscribe you next week and be sure to check that notification bell so they don't notify you of my videos. Thanks again, guys. I'll see you for tomorrow.