Welcome to the last set news. My name is Robin. Welcome to the one and only live stream for the week We used to a lot of these but we got away from so we're just gonna do these once a week just to for the sake of time and Then we have to go over today Just a couple of stories one is about coin base and the amount of staking that's going on and also about Super Bowl commercials or commercial and then before we get into it I just want to say everybody. Thanks for stopping by. I know we're gonna go into the Super Bowl That's I think it's like the kickoff is around 6 30 Eastern time But we'll just do a little quick review and then go from there Also, what I want everybody to do right now is if you have questions put them in the comments right now because these stories won't take That long maybe six to eight minutes, and I want to get through everybody's questions So start asking those questions now. So first things first Welcome to the live stream. These are my favorite We got away from because it was just you know time constraints, but I do like doing these things It's just just getting around for the time. So the first things first is It's like collateral damage and of course everybody's been talking about the SEC and Gary Gensler and cracking down and cracking and whatever else And it's all about staking and it comes down to an article from Blockworks. I didn't know that staking and yield generation and interest had been such a large percentage of coin bases business I can understand why they were You know so up in arms when this came about I mean not of course the whole Past and and how it affects the industry as a whole But I mean to see like 20 of it building being interest generation. That's a huge chunk to lose so Coinbase diversified revenue was staking how much great question. Here it goes Between the start of 2020 to the end of the third quarter of 2022 coinbase raked in 9.46 billion dollars from retail traders And an additional 508 million from institutions So before we get in just the yield generation part in the interest part. Do you realize what a huge amount that is? 9 I mean 10 and this is not a typo. I took a look at the shareholder Revenue report that would come out every single quarter and this is exactly what they said that retail traders that's us we Gave them the fees because they provide a service. I have no problem with that 10 billion dollars institutions 508 million and it's because the curse they they buy these large chunks and That's just how it goes. Then of course they have a custody service for them But it's not that much where they make most their money Is retail trading which is why it astounds me that places like voyager FTX and all the rest of them All's they had to do was just stick to that model and it wouldn't have screwed so many things up Just stop being so greedy.  No, you had to keep going. I digress But the weight of coinbase's trading fee revenue has been shrinking. Okay, we are in a bear market three business Three business lines have made all the difference interest Staking and subscription fees and here's we take a look at the coinbase revenues And what's interesting here is i'm just going to do something real quick, which is This is how big the retail transaction for the revenue is watch when I take this away It's like Nothing at all. It's amazing to me. How much They make off of us. I again. I have no problems with that. Absolutely no problems with that It just astounds me again about what happened with the other centralized exchanges and the greed that they had Now in the second quarter of 2022 coinbase brought in more than 38.2 million Be its interest activities, which was only about less than 5 of its revenue But then right after that the fall of the quarter That figure exploded continually 115 million or almost 20 percent of all revenues interest activities including staking. So When they heard and there was a rumor and that's what brian armstrong put out the rumor that the sc Was going to crack down. Would you like to lose 20 percent of your revenue? Just because of some guy's idea that this isn't really what it should be now There are some minute points and I understand that we talked about The things that I thought that gary got right and the things got massively wrong I did two videos on that a couple of days ago, but to me I can understand Why there is this much issue and then we can just see it here retail transactions taking that away And that's retail transaction interest Over here in q3 It's quite a bit and of course as we go into the bear market That could really hurt coinbase moving forward if it happens But here's a good point by paul gruel. He is the coinbase chief Legal officer and he says this Coinbase's staking program is not affected by today's news Staking on coinbase continues to be available and staked assets continue to earn protocol rewards. So nothing's been done yet yet In this regard What's clear from today's announcement is that kraken was essentially offering a yield product coinbase's staking services are fundamentally different And are not securities For example, our customers rewards depend on the rewards paid by the protocol And commissions we disclose rules making clear these distinctions will provide real clarity to consumers investors and the industry So there is that part But how much is that going to stand up to the scrutiny of the scc and just saying no We don't believe that's true. Now. We're going to find you and regulation through enforcement. This is the thing that we have to fear Moving forward, but maybe it'll just force Here's the thing Maybe this will just force the scc to go. Okay, fine We think and gary's been very clear about this right not that i'm agreeing with them But he said look he goes everything outside of bitcoin and sometimes a theorem whatever day he talks about it We believe that you guys Are selling unregistered securities. So if he has said that in the past Why doesn't just sue all the exchanges in America and going you guys just you did this So you guys are selling these unregistered securities. We're going to sue all of you and let's go to court.  We got my chips. We're going to go all in Let's have the courts decide it. Let's have congress decide it and let's just get this s out the way Because then we can have some real clarity and you can stop threatening us Like the nerdy bully that you are and just get this out in the open Correct me where i'm wrong. I just say we should just have a showdown now Because i'm tired of this this piecemeal stuff and just the the little nibbles that are being done That maybe i'm wrong here. That's how I see things and then also this was an interesting point by Jesse Powell former CEO of crack and he says oh man All I had to do was fill out a form on a website and tell people that staking rewards came come from staking Wish I'd seen this video before paying a 30 million dollar fine And agreeing to permanently shut down the service in the us How dumb do I look and he goes on a little thread and it's pretty interesting I'll link in the description. You can check it out But it is true because gary comes out and says oh we just shut you guys down because you didn't have disclosures That really wasn't what what it was. It was all about securities and piecemeal and just kind of keeping everybody in fearful and in in check So the big places can't come in so let me just think about that in the comment section and then also Just you know, I thought super bowl ads were there was going to be no representation For a crypto, but I guess so super bowl crypto ads are lacking, but nfts have a spot This was a statement from mark evans fox sports executive of ad sales. He said well two crypto Advertisers had had commercials booked and then those deals fell through after the crash of ftx So we were gonna see some even though we're in a bear market There was a couple different I guess exchanges that were coming out But they said after ftx that kind of closed the loop. However Ftf nfts a once little known acronym that entered the mainstream in 2021 Are said to step back up utah based web three company limit break. I don't know who they are Said in october that it bought a super bowl ad for six and a half million money well spent sure The company spot will feature a qr code In reminiscence of coin bases ad last year Directing viewers to sign up for a chance to own a digi Digaku nailed it drag and nft the company said the ad culminates a free mint white list opportunity Launched last month that is extremely limited. I gotta tell you it sounds good Then maybe we can get people to actually get an nfts again at least we have some representation So i'm happy with that sounds good and that's it for the news today And before we go in the q&a again get your questions ready There's just a couple of things that if you're looking for some pretty good information I think and especially if you're a cardano cosmos avalanche holder There's a great video From crypto ceo i've had him on the show real smart guy Does really great work and for some reason i just don't get this he's got a whopping 30 000 subscribers That's uh, honestly rookie numbers. He should have at least 100 k with the kind of quality he's putting out So he did a fantastic video about first of all about how polka dot avalanche and cosmos differ And how they all work and he gives them the pros and cons and it's really well balanced He talks about here about the cardano sidechain that's being created on cosmos And uh, just well put out. There's a link in the description I think it's like the first link that I put in here. So somewhere here. Yeah crypto ceo and cardano sidechains with cosmos So you can watch that video.  So i'm just going to give you my answer And uh, i'm i'm not going to tell you why I said this you can watch it later But here is my answer to guy because guy and ben they came on said i'm gonna have like health any which is a great choice and uh Satoshi Nakamoto and a couple of different, uh, you know gurus in the space, but this was my answer take a listen You would ask satoshi, wouldn't it? It's like what do you what do you think? Could you ever have imagined that that it would get to this? um So rob, I mean i'm dying to hear your picks on this the picks that apparently no one's going to like No one's going to like this, but here it is So like I knew you guys were going to say some of those some of those chaps You know who you're going to bring in which will be fascinating So you can probably do a video about it or you we talk about it like all this what they says So i'm like great. That's awesome. I you know, I'd love to hear about this This is who I would want I want well, of course, we got five people. So I want number one alex moshenski I want doklon. I want the two guys from three arrows capital I forgot their names and I want sam bakeman free and if I could have a six person as like the waiter I want gerry ginsler. So so that's it. So i'm gonna tell you watch the video I don't explain why it make a lot of sense and uh Yeah, I just thought it was an entertaining why just uh just to get through but a great video link in the description also I just wanted to give people a little bit of I don't want to say hope him but some good news because I know we get you know We get all this bad stuff with the sec and we're like, ah crap everything's going down You know, it's gonna be awful for whatever else look bny melan says their investors are absolutely interested in digital assets bny melan is the oldest bank in united states also Just so you know, they don't pick a bunch of losers. They've been in this space for I mean forever They've got 44.3 trillion assets under custody. They're a bank and they've got almost two trillion assets under management so this is just one more example Of a big institution getting into crypto going. Okay. We don't know what's going on over here But we know this is going to be big and I think we're in the right place at the right time Also, I want to say thanks to everybody who showed up. We did uh the amigos de los animadas Uh, we go there every sunday and we walk the shelter dogs But look at this Look at these these great views you get to watch if you're in Puerto Rico next sunday come come hang out with us And after this we all went to breakfast, but it's just fun.  So not a bad deal win win and then Also as a reminder that this wednesday is our coin ledger live event So if you've got questions, I know people are getting like 10 99s from celcius and voyager and their You know ftx and they're having them Requesting them to pay things and there's some other questions for us taxes I'm gonna have the guys from coin ledger, which is me david kemer and the tax lawyer I forget his name But we'll be asking all your answering all your questions Live that'll be around two 15 115 eastern time I'll put it out on twitter and I'll talk about it the next two days. So just remember if you got tax questions Follow me on twitter and you can ask your questions right here and we'll answer the guys and of course We'll do it live and then lastly lastly lastly Ah Clown news.  So I guess people have issues with clowns. I don't know. So I'll just use this clown face. No one You know gets frightened, but this is really the biggest clown news It's not news. It's just a statement.   You may lose your dog.  None of those things matter What matters is that you achieve success and become free and you can do whatever you like I got to tell you that's the dumbest thing I've ever heard And I'm not alone With people who have to agree with me. I got to tell you if you think that Success and money is going to be like the end all be all and freedom. Let me tell you Once you get to a point where You don't really need money. There's a lot bigger things out there and it's not just about making a ton of dough And that's it. So look that concludes today's news piece So like today's video thumb up The subscribe all that good stuff and now let's get into My favorite part, which is the q&a and then we'll get out of here and go watch a little football And we'll go from there. Here. I'll do this one. That's better. I'll stay like this all day.  First of all, it has a lot of a lot of questions Chival that's good. It's a pretty good day Good morning. Yes Okay, this was the question that I So rob hope you're well will be great to hear your view on retail clawbacks without insiders info by celsius If you don't know there's this thing called clawbacks, which is the dumbest thing of all time Well, it's not dumb. It's it's a legal president And what it says is that if you removed your crypto crypto I think the money too if you remove that At any time within 90 days Of when celsius and voyager and ftx apparently shut down that you are subject to clawbacks What do I think about that? Well, obviously, I think that's a that's a pretty awful way to deal with investors But unfortunately it is the legal precedent and it really comes down to the judge And as as they determine who is the person or who is the rifle owner and celsius they very well could do that Now are they going to do that? That's the question and I'm actually going to have I was reached out by plan c who Asked me to have one of the representations of the lawyers on so I'm going to have This lady on and she's going to talk to us about clawbacks and what it means for us I'll keep everybody in the loop and go from there. Let's see Ah, that's the wrong one. Okay. I'll turn it off. I it's a little goofy. Okay George good day Buenos dias Morning round thanks on dare dare looks great Some people say it's going to be the next big thing and it might be I just don't have any exposure to it So that's all I can tell you and you know on this channel I am super biased. I only talk about the things that I own so if I start buying hadara, you'll know it because I'll start talking about it Yeah, that's about it. Let's see Not quite so it says the coin base freezer robbery. Well, there's two things really about that first of all The fees are a little bit higher, but guess what they haven't collapsed so Everybody was talking about how great the ftx fees were and you know what their fees really were 100% if you think about it, sorry So if they're going to charge me a little bit more and they stay up, great Uh, it really doesn't matter to me. I transfer all my crypto off exchanges onto my nano ledger x or yeah x So I don't really care so much, but if you're worried about fees, there's two options First of all, you can use they used to be called coinbase pro and now I think it's something else coinbase advanced And if you want to use that the fees are are less from what I understand I don't use it personally because I just dollar cost average and I don't really I'm kind of lazy. So I just let it go Another thing is is a coinbase one you pay like 20 bucks and there's no fees up to $10,000 per month And that's what I use every single month and that's it So yeah top of the morning That's a good question road trip buddies So there's To answer your question the question is rob did you move to Puerto Rico to save on heavy taxes? It's one of the reasons.  So what's the point? So we spend six months here. We can either spend six months here or like Florida We're like Arizona But we're like, well, we like it here and there's a nice tax benefit So we chose this place and it's working out pretty well so far So there's a couple different reasons so before anybody moves here Just remember if you got like kids and you're like, well, I got to move with the kids and then, you know Depends on your revenue and your income. There's a lot of factors to think about it Don't just move here and go, okay, great. No taxes. Trust me I've done a couple of videos on Puerto Rico so far watch those videos first And I urge you please do that before you make a potential huge mistake And move here for some for some it's for you and for some it's not right on Let's see if staking took off Coinbase could have eased fees and taken equity from Binance plot twist CZ made a few calls to a certain party And whispers and ears to stop staking, you know, there's a lot of things that are going on I just I find it interesting That gary gensler Is always talking about we you know, we believe that All these exchanges here in the united states are selling our ownership securities and we believe it and then they just Every so often they really make a big move and it's like to scare people Just to scare people back into submission and go and see we told you where we can do these things and they do it That's why I'm like just let's go to court right now Let's go to court. Let's get it all in the open and then we'll see and guess what if the sec does win Okay, well, then, you know what that means. Yes, all the exchanges become Selling securities robin head does it every day in equities Stocks so all you gotta do is register and then the different projects that'll say, okay Well, I guess now we're considered a security here. We'll pay some fines and move forward whatever So let's just get it out of the way and get it done So the institutions are like, okay now I can invest into this stuff And I know where we're at and I'm not going to get a fine for offering Unregistered securities to my customers which would leave me to my next point I was talking to not meme not meme shout out today at the At the event for the shelter dogs And she said what I don't understand is that fidelity in my account is offering me Crypto products that I can buy right now in my retirement account Or I think it's an investment account and I can buy those right now But they don't seem to be bothered that it could potentially be an unregistered security And it's it's things outside of not just bitcoin bitcoin and there even some other cryptos So I'm like that is interesting It's a ballsy move and uh, I just wonder how that's going to play into effect later on So just think about that It's it's it's it's it Let's punch up like like some damn. Yeah, that's right punch up Ah Yeah, I don't know George says why is the sec pursuing legal action against coinbase they get up Real big thing for coinbase for whatever reason And uh, I'm not for sure why either so who knows They share network fees each other boards as users I gotta tell you And my concern was this and I talked about this before it seemed like before Like in Different exchanges like they would say okay, we're going to stake your crypto But we're also going to re-hypothecate it and we're going to give you x amount of of yield And it seemed like you didn't really know where the yield was coming from was like Were you really staking because you said you were but there's nothing there's no proof of documentation And Are you re-hypothecating are you co-mingling nobody really knew and now here we are so I just think like okay fine Let's just put on the open Get some disclosures like jessie powell was talking about what she's Does he did make sense in that statement?  I am not your dad. That's very true Uh Cookiness says celsius has confirmed they will claw back retail who withdrew their coins the nine days of vk What's your view if that happens who would buy sell krypton u.s? Injury exchange if you do Exactly Exactly, so it's like This is Man, I gotta tell you I now i'm definitely gonna have that later on If this happens, it's a huge black eye for everything, but you know what? It would make Where there is a hole or where there is a problem? There is a solution somewhere and I know that Not me, but people in the crypto Crypto field are smart enough to figure out a way around that to transfer your your crypto some way into Are your fiat into crypto and then you just use dex's that'd be a big a big question So if uh, if celsius does claw it back I Will go out of my way I'll go out of my way. I'll make it a mission To make sure that everybody is reminded of what they did if they ever did a new company or what's gonna happen Hell i'll stick around for another year just for that I'm a nice guy and tell Court would still be a huge risk though if it was all taken to court if we lose and good chance We would since we're against the ac. That's not true. Then we'd really be screwed. So think I should only Here's the thing. Cory you have to understand the sec doesn't win all the time It's just that mark cuban beat the pants off them in a lawsuit. Go look it up Not only was it uh a drubbing It was a fast one. So the sec accused mark cuban of insider trading. He was like Well, let's just go to court and see how that works out and it worked out great for mark cuban and the sec Was away. They don't always lose or they don't people don't always lose against the sec It's what i'm saying. So If we all got together I mean, let's let's be honest the exchanges do have a little bit of money a little bit power and sway I think we should do just fine and let's not forget that just because we're against the sec It doesn't mean that we don't have friends in high places the luminesces The ted cruises the tom emmerers The hester pierces. I mean these people are not just like Average jane and joes I think we do just fine. I think I think we went actually and we'll see Uh sec g just finished what f2x trying to do all year to stabilize crypto.  I'm actually very boring uh Yes, george the So-called, you know what's ben had a good point the people that I wanted on my dinner party Which was alex michinsky?  That'd be the first thing and the rest of people would come in. I don't really care and then you had uh, the two guys from three arrows capital and uh Sam bachman freed and then del kwan and then the waiter would be gary genzo So ben had a good point. He goes isn't it amazing goes if you would have said that a year ago Besides gary genzo people and like oh those guys are really good up and standing citizens of crypto and people love them. I remember Being at guys event. No, no, no, sorry. This was uh, this was this was another another private event that that I attended And uh, there were people I was up on stage with a like that martini guy and a couple different people And somebody was talking about you know Uh, we need we need people who will stand up for for bitcoin and crypto And we all said yeah, we need more people like that And then one guy stood up in the audience and goes you guys are like you pretty much was he started Yeah, and he said But we already got that and his name's del kwan. What do you guys don't know what you're talking about? I'm like only that guy's really that good and then like three months later everything collapsed So I think it was that I don't think it was really that good for the for the uh, crypto market in general Nah, you know I would want to have dinner with the illuminati, but that's just it anybody be illuminati. You never know They're like i'm the illuminati. You have no idea 200 years of jail Yeah, exactly definitely maybe I like the first clown too Road trip buddy says dude.  I've ever had I've lost both I mean there was a time when me and my wife split and uh My best friend just passed away My my dog of 14 years. I gotta tell you I'd give up Everything I own to have to take that back.  It just doesn't it doesn't matter in the long run. That's how I see it Uh Ah teeny tesla Hello Tika ben rinsberg. It's a good question. Hey rob. Will the staking usa problem affect us that are staying with you? No because I am just the face of dnew staking. I Am uh, just a cog in the wheel. Uh the people that's Our response not responsible, but we all have a certain amount in it But uh, our staking is out of net netherlands in australia So for us, it's uh cardano staking. So we don't Have anything centralized here in america Uh, we used uh kubernetes and we just Space it all out globally. So for us, it's not a big thing But if you listen to what what genzler said about the staking prospect and how that all works it's um You have to understand like Think about how aetherian runs their staking, which I gotta tell you is the worst Not the worst, but it's not the greatest. We'll just say that I've when I heard about the staking I was I did a video on it and I was showing people how to do it and halfway through I just said I'm not doing this I'm just not going to lock up my aetherian for that long. You can do it, but for me. I'm like, I don't get it I don't For me it didn't make any sense And then with cardano because like when you lock it up and you're moving your aetherium you're giving it essentially Locking it up someplace else and you don't have access to it But with cardano, I mean it never leaves your wallet ever you have your private keys You are in control of it. There are no lockup periods. There is no steak or there's no slashing. There's no I mean the worst thing that could happen is that you may not you may miss if you're designated Stake pool operator isn't Doing their job. You may miss one epic not giving rewards, but you'll never have your rewards taken away So for me like personally, I think cardano knocked out the park with staking. It is the cleanest easiest Way to do things and the most consistent. I don't There's and then of course everybody will say well, well this you know for well, this other project is great I don't really I mean I stake I've staked avalanche. I've staked Cosmos I've staked near I've staked salana and so far the easiest and best way I've seen with cardano And that's why we just stuck with that stake pool and that's it Let's see People who love money say that Yeah, some people are so are so poor.  I don't know I think if if the coinbase card rewards would be effective. I think all credit cards would be affected in the future uh Yeah, joseph's got a great question.   It would just shut down. I'm like Why does one of the biggest crypto exchanges on on united states shut down? I mean, they should have this already figured out and I got I was always giving them guff, but lately They've been doing pretty good and I will say one thing about brian armstrong I mean he really believes in crypto and really wants to make things work from what I've seen maybe behind the scenes He's plotting a huge takeover. I have no idea But uh, people just hate coinbase because they hate coinbase and they always talk about the fees and the fees and the fees look man these days i'm just happy that coinbase has not become insolvent and uh Collapsed I got to tell you it's just like the army just show up with the right uniform and get promoted So in this situation of coinbase just keeps moving forward and doesn't collapse when I see them as onboarding many crypto investors around the yeah me and lure in the same the same boat I gotta agree with you coinbase will be good It's wrong Ah Yeah, it's crazy ryan, right? Uh I know mr. Wonderful does have kids. That's what's crazy He's got a wife. Maybe he hates his wife. I don't know Allegedly, I don't know Yeah, d tom says i'm in new york, so I basically need coins y'all can f off wishing to see his downfall I don't wish his downfall. I want those guys to succeed as bad as anybody else does This question Curious as to why you and ben ditched james. We didn't ditch james the thing was like when my tribe Gets together with ben's tribe. We don't really have too many problems pretty much. We're all aligned But when my tribe and ben's tribe and james's tribe gets together It was toxic and when we did these these the dca shows It just wasn't a good mix after time over time And that wasn't my plan my plan was to alleviate your stress You're in enough stress as it is when you go to the shows I just want you just to enjoy it and relax And when you see things on live stream and it's really negative you're like well, this isn't what I wanted So I said we and james and ben talked me go. You know what? Maybe our tribe's just don't align and now james has got a new tribe and they seem to do pretty well So that's great. And then me and ben picked up guy and we're doing pretty just fine. So that's it How's porto rico? Man in porto rico is awesome. Um I like how in porto rico like stop signs and stop lights are suggestions And not something that you actually have to do that's fun And besides that the weather is perfect and people are great and the food is Yeah, but I mean it's a great place to be that's all that's all I do like it here I'm probably keep living here for the rest of my life Uh is the staking ban official just rumors. So the staking ban for kraken. That's official That's gone through 30 million dollars later. They're not offering staking. They will not be moving forward unless Gary ginsler comes out and says here's the criteria But as far as I know that's it now with coinbase and all the rest of them Seems like they can still do that by hand to us. Sure Uh Ah red crayon red crayon Keep up the good work by your insights. Wish I could have found you in the last bullet run so I was talking to me about this this this morning at the dog rock. I think that's what I'm gonna do so I'm I have a plan to sell my crypto And people if you're a diamond hands person You can do that but I did a video did a video and it was called And why and when i'm selling 80 percent of my crypto And I linked it in the description of every one of my videos at the very top in the description And I want you to watch that video because once that those criteria happen I'm gonna sell 80 percent of my crypto And once that happens, I think I'm gonna step away from this channel It's It's a good idea to do that To just make things final and then kind of move a little bit and what I'll do is I'll say once I sell Like this is this is just a thought I'm having it's just to go. Okay. Look everybody. I sold This is it you can do whatever you want to do you can ride this out But I'm gonna make things final right here And I'm gonna step away and tell the bear market comes and I'll probably just take a break for like three months Because there's a blow off top and who knows three months six months Who knows Then I'll come back But I want to make it like uh like a final so people are like oh rob is serious He really did these things maybe I should you know really think about that Do my research and see if this is a good time to do those things because it's always the same thing It's always a four-year cycle And it seems like whenever we think like it's just gonna keep going up and up it never does Obviously, right so I want to give people something to think about Instead of just sticking around going at everything's going good. That's great I don't know We'll see if that's a good idea or a dumb idea.  It's true SCC's from my banks Uh, and that's staking in japan is great. Yeah I probably It's a good idea T-shirts not your dad Is that true from april 2023? India is basically going to find and jail people who do transactions through peer-to-peer On foreign exchanges. So now the only option left is domestic exchanges that prevent crypto withdrawals So you can't that's interesting. Is that going to be a law? They already put that in if that's the case as far as crypto withdrawals nothing in cold storage Interesting jail. Wow Ryan says are you DCing more into bitcoin or eth bitcoin? There's more of an upside potentially for ethereum, but I'm Trying to play it a little bit safer Didn't know that Hey rob me time I walk on the shelf dogs Don't forget to mention scientifically proven to boost endorphins. Yeah, and you can also lose weight and you feel better What are your thoughts on lido staking? I wouldn't do it. It's just me I don't like having things locked up. I don't know never had an issue with eta Deadless is the best way to stake. That's what I use but people sometimes don't like because it takes so long to download I get it Mofangor tostones neither I don't like either of them, but everybody seems to love it here But that's just me because I'm not from here But everybody loves it here Ben had no mods Ben Ben doesn't like to Doesn't like to ban people He's just like me Like if you make if you're I think a scam bot, then I'll ban you but I don't even people would disagree with me I don't ban them I've got plenty of those. Let's see. I think that's it guys Selling sunset selling may and go away. I'm gonna put that to the test one these days That would have worked out pretty well actually for the last bill run Okay, and that's a good way to say goodbye tesla goodbye tnt and goodbye everybody So look that's it for today Uh, I gotta do some things gotta get out of here and get some things done But enjoy the big game watch that uh super bowl commercial should be interesting But that's it for today. So like today's video thumbs up before you leave. Please it helps out a lot apparently Subscribe you can subscribe, but it's not like youtube's gonna notify you all the time But good luck, and that's it for today. So thanks so much for stopping by. I appreciate you And I'll see you on the next one. Adios


Okay, so hi my name is Chris. I already mentioned I work as a production engineer at Facebook London working as a member of Web Foundation I'm gonna be giving a whistle-tock tour of the new version of control groups added in Linux 4.5 Don't worry if you haven't the faintest clue what control groups are yet I guess since we're in the containers their room probably some of you have at least some kind of idea But I'll go into kind of what they are where you may have used them and a comparison of the old and the new in the next few slides So like I said in this talk, I'm gonna be giving kind of an introduction to control groups I'm gonna go over what they are where you may have used them Where you might have encountered them before if you already know something called control groups like you've you fished around Previously or you've used them before you almost suddenly have been interacting with version one of control groups control groups It has existed in the kernel since a long long time ago since around 2008 and it's been kind of building our Love of containers. It's one of one of the building blocks of containers as we know them So it's definitely got a whole bunch of good things in it. It's also got like a whole bunch of Problems usability foibles other kinds of things which are not so great So I want to go into like what those are and when you might encounter them And and how we try to improve that in security to security one is currently now kind of in mostly in maintenance mode Where it's secretly to is is more in active development. They share the same core It's mostly the the user-facing API which is kind of different And yeah, I'm also going to go over kind of what's what's still to be done And what's what's already been done the core of C group B2 is already stable since kernel 4.5 But we have a whole bunch of work. We want to be done a lot of a lot of the core work has been to enable Future work to it to enable new features in C groups, which we're already working on So first a little bit about me. I've been working at Facebook for about three years. I Like I said I work in this team called web foundation Technically where foundation is the team responsible for the web servers at Facebook But web servers are kind of not not usually extremely complicated like they're stateless. They serve you cat pictures and that's about it So we actually ended up becoming kind of a team which deals with the general reliability at Facebook This means we get involved with production incident discussions and all sorts of we generally act as kind of the the guardians of reliability at Facebook Yeah, so I like I said I work at Facebook London. We have a we have office there We also have an office in Dublin in Europe and in Tel Aviv Yeah, and we have a whole bunch of different kinds of people in web foundation We have people who are experts at the Linux and positive learning stack, which is basically my expertise We also have people who are experts in our cache architecture RPC And say like the web push and stuff like this especially a cross-functional group of experts Which all come together to deal with when shit really hits the fan So that kind of brings up the question why why do we as a team like care about syrups? And why does Facebook is a company care about see groups? So we have many many hundreds of thousands of service And we have a bunch of services which run on the service some of them co-located on the same service Some of them spread throughout service. It's all kind of all over the place. So Most kind of outages at Facebook are a few there are a few kinds of outages But like one very common one is failure in multiple systems and there are a few things you need to do to mitigate that another one of course is Actively mapping out your dependencies and making sure you have an understanding of your dependencies But another big one is making sure that when two services run on on a single machine You don't end up with a situation where one service completely Overrides the other and results in it becoming completely useless. So that's one of my main concerns about Secret between one of the reasons why I'm super interested in it So a typical use case for this is like you say on a on a normal web server We have we have three types of running processes same goes for most other kinds of web services other kinds of servers as well So you have your core workload your core workload is the thing which if you were to describe to somebody else What your server was actually doing you would say it does this thing it serves web requests It does load balancing But usually you don't end up only with this on your server You usually end up with a whole bunch of other stuff Especially if your company has been around for a while or you have some ideas about how you should architect stuff You end up with a whole bunch of non-core services, which can probably be used interchangeably with kind of system services This might be stuff which just comes with Linux like also the kernel workers or it could also be like Say cover us demons or it could be like stuff which is related to your business needs say metric collections that we can work out What is going on with our server like whether we're managing to serve users correctly?  Same goes for cron jobs and chef I care about my server being up-to-date, but I would rather have some system Which which actually acts in a reasonable manner if chef starts taking a bunch of memory I would rather it got degraded service and still managed to run then it took all of the memory or took all The CPU and then everything was fucked like I would much rather have that that outcome Then we can have this third class like ad hoc queries and debugging These are typically things you don't know that you need and don't end up getting run on the majority of servers These are like the things which you end up realizing you need only when an incident is already happening And we kind of want people to be able to dynamically be able to determine the importance of those things As the incident is going on whether they want it to take precedence over the core workload of the machine or not So see groups is a very good use case Sorry, this is a very good use case for see groups Like I mentioned if you had some interaction with see groups You've almost certainly be interacting with version one version two has been in development for like five years now It only just got stable in the Linux kernel But even on recent kernels version one is typically what's mounted by default and the reason for that is as you'd imagine With a different version number the actual changes are backwards incompatible So I'm going to be going over kind of why we've made them backwards incompatible in a moment So the fact that we typically boot with your in its system only mounting the version one hierarchy Is is a testament to why I'm doing this talk like I know we have a whole whole Group full of container experts in this room And this is also kind of a sell to you about why you should give a fuck about see group V2 And why you should take the time to care about it and invest in it in your products So yeah, so in the previous slide we talked about multiple processes fitting into each of these see groups as well So a see group can be set as tightly or as flexibly as you like it can be all of the processes which are related to one service Or it can be a single process. It can be however many processes you like from zero to how many you like And yeah, the idea here is we don't impose a we don't impose a structure on you The one of the guiding principles behind V2 has been we want you to be able to choose how to use it We don't want to have a hierarchy which imposes how you should do things and we want the easy way to be the correct way So a see group is a control group. They're one in the same There's there a system for resource management on links like I mentioned a resource here It means like CPU IO memory and management here can mean for example Accounting like we know how much memory some particular some process in some particular secret for using it can also be limiting like We hit a particular threshold and we we take some violent action to curb that it can also in V2 There's also been some work towards throttling which I'll go in as well Generally, um killing or whatever is quite violent. So we want to have some Remediative actions instead of just just straight-up killing stuff Yeah, so The way that secret V2 secret V2 and V1 work are essentially you have this hierarchy at CISFS C group It's a book just a bunch of files and directories. We don't have a system call interface There may well be one in future, but we don't currently have one The reason the reason this is kind of a good thing is it's really easy from some user space application Even if it's not written in C or C++ and you don't have access to C library functions or system calls easily You can easily go and find out the state of your system when it comes to see groups You can just look at some file like I would hope that all of you are using Languages which could like open a file Make a directory remove a directory like yeah That's I would hope like whatever the new head kids are using still supports those kind of things So yeah, so each resource interface is is provided by a controller. You'll probably hear me use the word controller Resource and domain kind of interchangeably the idea is you write files in like some particular that apply to some particular Controller this controller takes the values which you provided and it provides them to the kernel and the kernel makes some kind of Decisions based on what that one values you gave it And so it's essentially the backing store for for all of the stuff which you input to see groups so as mentioned previously like Workload isolation is a really big use case for see groups So you might have many background jobs on a machine, but you don't want them to override the main workload That's the case which we talked about first There are also some other kind of other cases like say you have a tier which runs Which runs asynchronous jobs in the background we have we have that at Facebook And some jobs have higher priority than others priority is a very abstract concept But priority usually has something to do with resources It usually means either you expect it has more memory or or less if you or something like this It's very much up to you like what exactly that priority means You also have shared environments say you're a VPS provider or something like that and you don't want Some particular user to be able to use that container and steal all the resources from all your other customers Which are now going to go and leave you a bad review So yeah, so there are kind of a lot of use cases for see groups here And you might ask hey my my favorite my favorite application X like it already does this Why should I give a single shit about see groups? Well, the answer is if you have been using any kind of software in the last eight years I would really pray that it does it through see groups It probably does it transparently to you and you don't ever have to like talk to see groups directly But the the backing for all of these is see groups like that's how they do resource limiting So let's go concretely over how this works in in version one So in V1 Sisyphus C group contains the names of all the resources which which have a controller So this might be CPU memory pins that kind of stuff and inside these Inside these resource directories. There's another set of directories which are the see groups themselves So these see groups exist in the context of this resource and you put processes into those see groups For example, we have the PIDs one at the bottom. So because this is to do with PID PID resources it contains files in those directories and those files are related to how many PIDs you can have in a see group for example and we also have the concept of like Having its own hierarchy for resource distribution So in see group in see group V1 you have a resource and then you have a see group hierarchy underneath each resource So even if see group 3 here was called the same as see group 1 say they were both called foo from the kernels perspective They have absolutely no relation to each other This is kind of important because if you look at how for example system D sets out see groups typically You often end up with quite similar looking hierarchies in different resources and you might even be inclined to believe They have some relation to each other. Well from system D's perspective I'm sure they do but from the kernels perspective They do not and that results in a huge whole slew of like subtle issues, which cause problems at scale, which I'll go into in a moment See groups see groups are nested inside each other in this example So when a see group is nested inside another typically what it means is it can control some limited amount Up to the maximal amount of its parents So if you have if you have a memory see group And then you have a child of a see group, which is also that which is also another see group then you can limit Up to the maximum of its parent so The resource that the Controls these see groups or the see group hierarchy that they're in determines what kind of files there are I already mentioned that if you're in the memory hierarchy, you can only access files related to memory Like for example, that's this file memory dot limited bytes You can read the limited bytes from it or you can write to it and set a separate limited bytes And one pit isn't exactly one see group Per resource in see group you want so p2 here is is explicitly assigned to suit to resources a and see in secret one of five respectively But because we don't assign it to anything in resource be it actually goes to this rootsy group The rootsy group is kind of a special concept in in in see groups. It's essentially unmanaged territory How exactly it's manages up to the controller? But the idea is you don't really have the opportunity to set really any limits in the rootsy group because it's just The the starting point for resource distribution of this resource across your whole machine So yeah, you do get some kind of accounting, but in terms of limiting. It's basically useless So here is a concrete look at how this looks in see group V1 So you have Citifest see group and then you have the resources which are like block IMM mpids and then you have the see group names And we have nested see groups here a inside BG for two resources and be inside ad hoc for two resources So once again just as reiteration because this is really important that you get this from the Connell's perspective naming has no meaning like If it's in a different resource, even if it has the same name, it has no meaning and that has also some weird implications So here's how it looks in see group V2 by comparison So in secret V2, we actually don't see the resources anymore if you remember how it looked in version one We have resources under Citifest see group and now we actually have the see groups themselves under Cicifest see group So how do these see groups understand which resources? They are supposed to apply to if they're not inside a resource hierarchy. Well, the answer is kind of see groups at global now They're essentially a global a global set of see groups and you enable resources inside of the see groups This means that we have one hierarchy to rule them all and the idea is like you write to a special file You tell us what particular controllers you want to enable and we enable them for your see group We don't require you to create disparate hierarchies each time you instead create one hierarchy and enable see group controllers at will So this is how the previous example now looks in see group V2 So in secret V2, like I mentioned, we now have the see groups directly at the bottom but you write to this special file see group dot subtree control and that enables the In the shot in the children of that see group that those controllers are enabled Essentially if you were not to enable them there, but they were enabled the next level up It means they would compete freely for those resources that you didn't enable So yeah, so here's the version one hierarchy again for comparison and as you can see here. We have resources first and Remember that in version one See groups with the same name again don't have any relation to each other In secret V2, we have this unified hierarchy And you enable resources for a see groups children by writing plus memory plus page plus CPU plus IO that kind of stuff To see group dot subtree control and when you do this the files appear in that directory instantaneously Oh, and another thing to mention is in real life You also need to enable the memory pids and IO controllers at the top level for this to work But for the sake of simplicity, I'm not out of them here So the fundamental differences are obviously the unified hierarchy Resources apply to see groups now instead of see groups applying to some some distance across the hierarchy for a resource This is very important for some kinds of common operations in a nooks like for example You have pagecast writebacks and pagecast writebacks Transcend one particular resource they they happen across a whole bunch of different resources And we need to be able to we need to be able to consider these actions together in unity to be able to form reasonable limiting or other actions We also have granularity at the thread group ID not the thread ID level This is a contentious point, but it's it's kind of important In cigarette V1 you could essentially put different threads from the same process into separate see groups This has a whole bunch of weird implications like for example People would put Say different threads from the same process into different memory see groups I don't know how the fuck that's supposed to work like You have like basically the entire shared memory between these two fucking secrets I know people have done insane shit with see groups This is this is the main the main thing is like we want to guide towards a reasonable implementation because it's not that these people are Stupid that's not the problem. It's just that see groups in V1 were like quite over complicated. So it made people do some insane shit So now limiting the paid level kind of gets us like a more reasonable approximation of what people generally want Also without extensive cooperation even for the resources where maybe it's would in theory make sense to to have Different different threads from the same process in different resources. It often ends up being like You you have to have some way to communicate which thread from your process is doing what and there's no standardized way to do that In the next right like you can set the the calm of your thread to like Some value and then then look at like the value of somewhere But it's not standardized and it's like really really fucking hard to reason about So yeah, it usually doesn't act in any reasonable way in general Like there's been a focus on simplicity and clarity in in v2 over like ultimate flexibility v1 was invented Kind of the dawn of containerization people didn't know what they wanted. They just know that they wanted something and they wanted it now So v1 was kind of a solution to this problem and v2 was kind of more like a more a more developed approach to the problems We know we're having for sure now So another new feature in v2 is the addition of this no internal process constraint So this means that C groups with processes and controllers enabled cannot create child C groups Essentially, this means that in in simpler words These red C groups either have to have no processes or they have to have no controllers enabled in that part of the hierarchy This is for a few reasons it's kind of hard to reason about how that should act usually in in v1 this in v1 this was allowed and The problem is now you have two different types of objects kind of competing against each other So say you have processes in I and then you have some child C groups Underneath I here Now you have processes which are in I Competing against C groups which are its children and it's difficult You have to make some kind of you have to make some kind of judgment about how we will treat processes Compared to C groups, maybe we can consider each one its own C group Maybe we can consider them like I prime like a separate C group But it's quite hard to reason about and for most cases the better solution is just create another C group So this is another guide towards like helping people create a sane hierarchy And the root C group is a special case the the Controllers themselves have to decide how they're going to handle resources in the route So clearly breaking the API is kind of a big deal like C group The secret papers is pretty fucking big deal Like the fact that we want to create v2 instead of instead of just improving v1 obviously needs some good some good reasoning there So v1 works okay for basic situations, but it gets kind of exponentially complicated when you're getting more and more complex as I mentioned in v1 design kind of often followed implementation and Trying to rework kernel APIs after the fact is really really really hard Like you can't change the kernel API the fundamental nature of the kernel API which people rely on day-to-day in production That's just not something you can do Even for stuff, which was designed upfront like I mentioned like generally the use cases for containers and C groups We're not really that well fleshed out yet. They originally started as as like only for CPU And then it grew and grew and grew kind of naturally It was generally hard to gauge at the time how C groups will be used So now this is an opportunity to redesign them and work them how how we actually think they should be So to fix these kind of fundamental issues you need to have an API break and that's why v2 was created So I want to go over some of the actual practical improvements because I've talked a lot about like the theoretical how we've designed it But I also want to go over why we've designed it in that way and what that actually means so Okay pop quiz when you write to a file in Linux what happens It's not a direct question. That's where you're gonna follow this. Okay, you get a file descriptor Okay, that that that was that was possibly before what I wanted. Okay, so does it write directly to the disk? Okay, so where where does your data go? I got about five different answers. I'm not sure what note they were Okay, so it goes you write a dirty page into into the into the page cache You have like some some some set of dirty pages now and your rights his call came back and it said everything's everything's great And from your applications perspective now you can go on pretending. Yes, it was written to disk And my rights is cool succeeded. So it must be written to this can you can have all this class of wonderful belief But ultimately it's not been written to this right?  We don't have any account We don't have any tracking of this. So if you if you wrote dirty pages We don't know where they came from afterwards when we flush them to disk So that IO goes to the root C group and we can't account it to your process or your C group Simply because it wasn't tracked. It wasn't tracked where this page comes from in v2 It is tracked and we can actually count these towards your your limits And we can also make kind of reasonable decisions about like oh you have IO contention And this is what I should do based on that or you have memory contention This is what I should do based on that when you're trying to do a pagecast right back V2 is also generally like kind of better integrated with subsystems So most of the actions we could do based on thresholds in version one work Kind of crude or in the case of the memory subsystem like quite violent You set like a limit with memory dot limit invites and what happens is you you So your application has like a one-minute spike in memory usage and then the umkiller goes along and goes Oh, I'm gonna kill you and that's like the standard method of dealing with things like Usually like processes don't particularly like being kill-nined I I don't know like Maybe like there's some kind of statistic processes which like that, but I Mean ultimately, it's not not a very a very good way of limiting results uses What you really want to do is like tell it okay Can't the fuck down and like stop allocating memory or in in the better in the better case where you can't tell the Can't the fuck down you want to tell the operating system.  We've come past the point where like the penalty for any kind of failure is instantaneous death So so yeah, so now in secret v2 we have like generally kind of better thresholds here We have we have a new thing called memory dot high Which instead of killing a process we still have memory dot max which is very analogous to memory dot limit invites Which uncaled your process, but we also have memory dot high and what memory dot high does is when you pass this threshold We start to do throttling and reclaim when for every single map memory allocation So when you go over memory dot high and you want to do another mallet or you want to get grab some more memory Then what we do is we break into a separate path in the kernel and we say hey I would like to I would like to dial back this user So what I'm going to do is go to the tail of the inactive list and start reclaiming pages So if you fail to reclaim any pages, it's still kind of good because you took a while to to scan the page Cache it took you a while to scan the page cast We slowed down your application now and we've done it in a way, which is kind of transparent to your application But if you do manage to reclaim pages then we also win because now you've retained some pages And you managed to get some memory free again In fact, this is like a generally a much-sane a way of doing things and this is like Using this on web servers was like a big win when you see like these spikes in resource users So notifications, I don't know like how notifications are like one of the more edge cases for C group since it usually ends up being people Like system D which end up using them But notifications are essentially a way to say hey something in my C group change state It could be like oh, I have no more processes in my C group. So all of them have finished It could be like oh one of my processes ran out of memory and I'm gonna take some action based on that Ultimately, it's a way to get information about what is happening in your C group system D Uses this for example to track which processes are running and the state of the state of your system and the state of the services Which you're running The problem is on v1 for for release notifications Which are the notifications which are sent when your C group has no more processes, which for example means like oh we exited You have to designate what's called a release agent and this release agent You it's just like giving a call dump utility you tell it like here's the path to my executable And when you have no no more processes go and execute this this this thing with these arguments The problem is now if you have if you're using C groups as a utility where say you have C groups Expiring like a thousand times a second. You're now like forking a thousand times a second as well Which is pretty bad like it's generally pretty expensive And it doesn't make a whole lot of sense since the rest of the the rest of the C group API used slightly more sane methods You using a bent FD So now we have I notify support everywhere Since if a C group looks like a false them it kind of makes sense that it supports our notify We still have event FD support so you can like poll and find out what the answer is The idea now is you you can have one process to monitor everything you like you don't have to fork a new process every time An event is created. This is just like a straight-up grade really So utility controllers are another thing utility controllers are controllers essentially most most controllers in in C group Related some particular resource say you have memory CPU IO there are other ones. However, like save perf Or freezer, which I'll also into us in a second Which are not related to some resource, but you put the processes in that C group based on some actions You want to take to them as a group? So the idea is like basically you want to group them together so some user space utility can take some action based on that group um Perf is a tool for performance monitoring and tracing and I guess quite a few people probably probably heard of it And the way it works is when you say I want to have a certain set of C groups Is you give it a C group C group path and it says okay? Here is like some particular set of processes, which I'm going to map into my own C group hierarchy So now you have sysfs C group perf And inside there is a completely separate C group hierarchy, which only relates to perf This usually doesn't make a whole lot of sense usually what people want to do is monitor an existing C group hierarchy not create a new one So people result ended up resorting to like tons of hacks like oh, I want to like copy over this hierarchy to the other one They would run like a tool to copy it from one one hierarchy to another and like you end up with all these race conditions And like horrible and it was like really really bad So now having a unified hierarchy means we don't have to sync you only have one hierarchy So there's no way this could possibly go wrong touch wood So in v1 there's also like a lot of inconsistency between controllers This usually comes in in kind of two forms one you have inconsistent API's between controllers, which do exactly the same thing So you have like the CPU Both CPU and IO are essentially weight-based or share-based you give a certain amount of some resource Based on a relative amount to some some other particular C group But the API's were completely different the API you had to learn two API's to do one thing Which is really really not ideal So there's been a lot of focus on trying to unify the API's and also unify the naming like and generally like It was a bit of a crap shoot in v1 And now we have the opportunity to to rethink those names and generally standardize them a bit more So V2 is generally more intuitive up front. Oh Another one is is kind of inconsistency group semantics for example root C groups for example, sorry most C groups Inherit their parents limits So when you create a when you create a child C group of some particular C group it usually it usually can only use up to its parents limits But some some controllers didn't do that some controllers did their own thing for some controllers This whole idea of a hierarchy was like an imaginary thing and it just created a new C group and didn't care where it was It was all a bit of a crap shoot. So now with one unified hierarchy. It's it's more difficult to fuck it up I guess So V1's V1's over flexibility also contributed to like a whole bunch of API problems For example, when memory limits were first created They only limited a few like a couple types of memory and they were they were in this file memory dot limited bytes Then as more and more memory types were added They ended up getting like their own files one by one So you ended up with memory dot limited bytes memory dot K mem dot limited bytes memory dot K mem dot tcp Limit bytes member MSW the liminabytes and the really bad thing about this is now Yes, I have very granular control, but it's not very useful Because now say I want to set a limit on the maximum number of TCP buffers which is set with memory dot Memory dot came on dot tcp dot limited bytes now Say I have like 10 gigabytes of page cache free and I set a limit on the number of TCP buffers if I said like Oh, you should only get say X amount of TCP buffers if you go one over we will kill you Even if you had 10 gigabytes of page cache free or something like that It's not a reasonable way of operating usually like most people don't want like most people don't care like you allocated one TCP buffer too many they want to give some kind of idea about the overall memory use and like reasonable memory use Generally unified limits are only reasonable way to approach that so yeah again another trade-off kind of in favor of in favor of usability over ultimate flexibility Generally if you do want to limit these things say you wanted to limit some particular kind of resource like the pit controller is a very good example The pit in the early days of C groups it was considered Maybe we could limit the number of pids by like limiting certain types of kind of memory, but that turns out to be really really fucking hard Like so another the way that was fixed was we now have a pit controller and that specifically controls this resource So if you do want to do very specific kinds of limiting like some TCP buffer or something else Then you should do that through a new controller. That's that's the reasonable way to do that So if you go to Facebook comm right now, there is a one in ten chance. You are going to hit a server with secret v2 We are running a secret v2 pool in like the tens of thousand machines now And we're we're investing heavily in secret v2 for like a number of reasons My main concern like I mentioned at the beginning is limiting the failure domains between services I really care a lot about making sure that we don't have cascading failures or anything like that on a machine And also being able to kind of manage the resource allocation in your data center, especially at Facebook sale is really important Like if we can suck at this that little bit more resource efficiency The data center then that's that's a really big win We run secret v2 manage with system D One of my teammates David Capico was sitting back there did a talk about this at systemd.com for last year Called I believe it was called deploying system D at scale correct me if I'm wrong. Yes, it was So yes, we're really big contributor to to the core of secret v2 We have two of the core maintainers working at Facebook And we will continue to drive innovation here like this is a big a big thing we work on right now So I already mentioned that secret v2 is kind of new Secret v2 has been usable for a little while now and the core is all very stable But that doesn't mean that there isn't still work to be done here Like a lot of this is kind of building the building blocks for future work So the core API is a stable But there's definitely functionality to be worked on when thinking about see groups like most people think of kind of three things I guess which is IO memory CPU Those are like pretty much the biggest ones and two out of three of those are merged currently Pid the PID controller is also much for the CPU controller like it's very important But there have been some disagreements with the CPU subsystem team about how to merge it They have some disagreements about the no internal process constraint and also like Having process granularity instead of instead of third granularity and stuff like this Yeah, there's a very juicy drama filled thread at that link As all Linux kernel mailing list threads are it's probably still better than usual We also may end up with some thread based API for some particular kinds of thread operations that make sense That would be like in the works Another kind of big bet that we're doing right now is what one thing that the Linux has never really had is a good metric for memory pressure Like we have many kind of related metrics like the amount of memory have free or that the amount of memory used You can also look at stuff like Certain kinds of page scans or like yeah, it's it but it's all very heuristic and ultimately none of these metrics prove That you actually are encountering memory pressure because they can also happen in a bunch of normal scenarios So our proposed measure is to track page refalting so we the essential way this will work is we track pages which are consistently refalted back into the active queue and Essentially the way to explain how this whole thing works So you have the inactive set those are pages which the kernel considers Are probably not being actively used by any process Then you have the active set which are considered are more likely to be used by some some process And it's essentially one big list and the way it works is when you have a page fault You go to the head of the inactive list if you got if that page was accessed again Then it gets moved to the head of the active list which means it's protected from from reclaim And when we do a reclaim we go from the tail of the inactive list So we we take the pages which we consider the least likely to be used of a reclaim So what happens if we keep on? Faulting pages in and they get so far to the end that we keep on reclaiming them and then they fall back in again immediately That probably means that we have too many pages like at once for our system to handle It probably means that we can't we end up pushing them so fast off the edge that we simply don't have the resources to be able To deal with this number of pages that is probably not a bad metric for for for memory pressure And it's one which is currently being worked on as part of the security effort as well Because we do want to have those metrics around memory pressure not just memory usage Which is kind of only tangential to the thing which you really want to know so Sorry question. I Thought I had a question so we We also have Tracking now of page caps right backs what this first point means is we have the tracking of memory in Iowa But we don't have tracking for CPU yet Like you spend some time on CPU waiting for network packets to come in on the network We and we don't know who it's for yet because it didn't get rooted yet And we can't account for that yet And we also can't account for the CPU we spent like doing a doing a page cache right back yet That's something which is going to take quite a bit of effort But it's something which we're definitely working on at this point Yeah, and as for the second point I already mentioned like the the refaulting generally the idea here is that we want better metrics for Memory pressure because right now we only have something tangential Yeah, those you who use V1 are probably also know about freezer freezer is like an alternative to umkilling for example like you can freeze some set of processes in their state and then Go and decide like oh, I want to raise the memory limit or I want to kill them or I want to stop So some new processes it was essentially a way of freezing them in time and having some other process come and decide What to do about it? in in secret we want this didn't work at all basically like if you if you Used freezer one of the most common things you might want to do is go and like get a stack trace and look out What they were doing when like they kept on allocating memory or whatever it was that you froze them for But like it's a very common situation that you would end up with like say GDP if you would try to attach GDP It would end up in D state, which is not really the ideal result if you want to like find out the stack Of some process which froze like that's generally the complete opposite of what what I would like The reason is like the freezer implementation in v1 doesn't guarantee that we stop anywhere reasonable We we often stop in a stack which makes absolutely no sense in the kernel So in v2 the idea is to have like a more kind of sick stop style mechanism sick stop is very well defined And where it stops is very well defined as well So it's it's kind of a more reasonable solution to use for stopping processes So the the v2 implementation of Friso will be more along those kind of semantics So I talked a lot here about trying to sell you a secret v2 But I should actually tell you how to get it probably at some point during this talk So hopefully you're interested in trying it out yourself So here's what you need to get started with version 2 first. You need a kernel above 4.5 Before that we do have a developer flag which you can go find I won't tell you because it'll like eat you if you try and use it and you don't know what you're doing But I really wouldn't recommend using it before 4.5 4.5 is the first point. We have a stable API Once that's done more or less two things to do you need to turn off all of the controllers for v1 And you need to turn on and mount the the file system for v2 Typically you want your init system to do this for system D use it with this flag You basically put both of these on the kernel command line But if you're crazy or you want to try it yourself You can also manually mount it and cry when things break So yeah, so if you're interested in hearing more about c-groups come talk to me I'm happy to go over any of what I've been talking about on v1 or v2 And yeah, I'll be happy to go over any questions you might have and if you've used v1 in the past Which I guess many of you have and you found it lacking in some areas Please do try out v2 and let us know what you think. Thanks


Thank you, Yashua. Just to introduce myself, I'm David Cox. I'm the director of the MIT IBM Watson AI lab. And we're thrilled to have you here. And we're going to have a brief fireside chat. Apologies that there's no fire here. I was actually told that technologically it was possible, but we thought that that would be tacky. So Yashua, I just want to start off before we dive into the technical discussion on the fascinating talk. Since it's been a while since I've seen you and since then you won the Turing Award. And I just wanted to take this opportunity to congratulate you publicly. It's a tremendously well-deserved honor. So I like that. So and also just as a bit of a plug for some of the other activities that are happening at AI Research Week, we're running a workshop on causality and transfer learning, which is constant with many of the ideas. We'll share some of our work, but also the work from the community in the Boston area. We also have a workshop on neurosymbolic reasoning and machine common sense, which I think is also in a lot of the similar themes. But I just wanted to unpack two things. And one of them starts with that word, the S word. So you didn't say the word symbols. And yet in many ways symbols embody a lot of the ideas you have. So they're compact, low dimensional. What are your thoughts? So in deep learning, we don't do symbols. We do distributed representations of concepts. So what I personally want to take out of the classical AI work with reasoning and logic and so on actually is not the symbols. I mean, there's discrete concepts for sure, but it's much more profitable from a generalization perspective to think of these discrete concepts as having attributes that make cats and dogs close to each other so you can generalize across them. But the thing, so that's like an old idea from Jeff Hinton, right, in the early 80s. But the new thing is with the attention mechanisms I've been talking about. What the attention mechanisms I think can bring to deep learning that is also inspired from classical AI is the ability to manipulate variables. The ability to do computation on objects and not always the same objects. Just like in programming we have functions and we can call those functions with different arguments depending on the context. Whereas in a normal neural net it's always the same neurons sending their outputs to the same other neurons. But when you put in an attention mechanism it's like you're saying, oh, I can choose which neurons are gonna send their outputs to this guy, right? And now you have to start thinking in ways that look more like classical programming or classical AI where in addition to values being sent you have to tell the guy which receives who's that guy who's sending? Like what is my name, right? What, and names now is not gonna be a symbol it's gonna be a vector which is like a key in transformer networks and attention mechanisms you have key inquiries. So the key becomes like a type or a name that allows the recipient NLP to know what is this variable that it's getting not just its value but who's that, right? What is that variable? And now you can do things of course with recurrence to recurse and do these things. So I think a lot of concepts that have been developed in classical AI including like chains of reasoning and so on can be transported to the neural net world but in that way transforming that the way we think about neural nets from these vector processing machines to systems that can handle variables, recursion and all these things. Yeah, so we've been investing heavily in what we're calling neurosymbolic and other people are calling neurosymbolic systems and we're actually agnostic on what we mean by that it's always good to have a little bit of a vagary if you're not sure yet what you wanna brand something as but there's a version of it where we say, okay how can we teach neural networks to process things that are symbol like, like what you're describing but there's another view which says, well if we have symbolic systems and we can get from neural networks to get from the world to those symbolic systems. That's not gonna work. You don't think that's really good. So David Cecillo has an interesting quote which is, you know, reasoning, human reasoning and logic is crappy Turing machines running on amazing neural networks and neural networks today are crappy neural networks running on amazing Turing machines. So, you know, if we actually combine the two and again, we're agnostic on this. We're doing a little bit of both. Yeah, nobody knows. I mean, we're exploring. Exactly. So what are you talking about? My feeling is that just like sticking classical on top of neural nets is not sufficient. So one, you wanna keep this soft probabilistic distributed attributes and not suddenly move to the pure discreet logic thing at least to mimic the kind of reasoning that humans do, right? You can do amazing things with classical programming and optimization and search. But I mean, if your goal is to emulate the sort of reasoning and high level condition that humans do, I think you have to build in the system one part, like the deep learning part inside of the reasoning, right? So it can't be totally separated. For example, the control mechanism which decides where to search because when you do reasoning it's basically like the search, right? And right now our reasoning systems they are like super expensive in terms of search. Like they look at zillions of things. Multicollow research is probably the best integration we have right now and human reasoning is very, very different. We just look at two or three things. Well, it sounds like magic, right? And it's because we have a controller that decides what to focus our attention on, our thoughts on and it's very, very efficient but it can only do it because it's grounded in sort of all of that intuitive background context that the system one provides. Sure, and then increasingly there are formal reasoning systems that rely on internal network to do premise selection for instance, to reduce search spaces and things like that, so lots of confluence of different ideas. Probabilistic programming is another sort of intermediate area where you can actually have program, you invoke arguments and as you imagine increasingly sophisticated systems, probabilistic program gives you the ability to represent things that are not fixed and discreet and don't have that brittleness. How do you feel about? I think it's an interesting direction but again, there are many ways you could do it and I would favor the ways which don't lose these distributed representations, right? If you do it purely in the symbolic realm, even probabilistic, it's not sufficient I think. So another, just to dig into another area which you invoked a lot is biology. So in the audio laid out this idea that there's this sort of intrinsic motivation of deep learning, originally came from biology but it's a fairly loose inspiration. I know this is an area you work in, we've talked about it before. How important do you think biology is? How does that flow? I mean the work you do in biologically plausible back propagation, is that meant as a service to neuroscience or are you looking for, is that scientific discovery or do you think there's real kind of gold there and inspiring us for new approaches? So in a good fraction of the talks, my first slide is about this hypothesis that there might be a few simple principles which explain biological intelligence and that we use to build intelligent machines. If you believe this hypothesis, which is a hypothesis of course, then the kind of work that tries to build things that are both compatible with our scientific understanding of what works in AI, machine learning and what seems to happen with biology should bear fruits on both sides. And I would say these days I'm more interested in talking to people who work with child development or high level cognition because I'm thinking about these things about causality and reasoning and so on. But I think in general we should look at potentially all sources of information about human intelligence including even the social aspects, right? People are starting to look at that in order to serve as inspiration for our machine learning approaches. Yeah and the connection to cognitive developmental psychologists is interesting. We're now participating together with MIT in a DARPA program on machine common sense and actually a mandate of the program is that we actually include developmental psychologists as members of our team and actually running violation of expectation tests on agent-based systems that we've been asked to build. So one of the areas of neuroscience where I'd like to see more work done that would really be helpful for the program I've been talking about is regarding memory and attention because the kind of memory that we've put in neural nets these days, I mean like the extended memory neural nets like neural train machines and all this, they're not the kind of memory that you find in human brains. It doesn't look very plausible. So it looks like a lot of our memory and the way that we select like attention is more of a process of the dynamic coming from the dynamics of the circuit which leads to sort of concentration of activity in a few selected places rather than copying information in some memory bank and then retrieving it. I mean another area of intersection with biology of course is you mentioned sample efficiency and natural selection has a sample efficiency problem insofar as your samples are babies and you only have a few of them and they're expensive to make and choosing the wrong sample means they die. So do you think there are tie ends there?  True, true, true. But we care. We care, we care. And the process as a whole also needs to be efficient and so in many ways evolution is another layer of learning on top. So actually the work on meta-learning started in the early 90s and with my brother Sammy we worked, it was actually his thesis topic. We worked on inner-optimizing, inner-learning, outer-learning where the outer-learning was meant to be like evolution but because we didn't have supercomputers we said oh let's back prop into the outer-learning. And so that was the initial inspiration. Now it is people are thinking about meta-learning potentially as something that may happen within the lifetime, within one brain you could have learning to learn going on because we are exposed to changes in distribution as I was talking about in my presentation. Yeah and changes in distribution, arguably one of the natural selection is an example of non-stationary distributions and trying to adapt the architecture. Of course there are connections to neural architecture search and methods like that. Except that traditionally people have been using meta-learning and architecture search to optimize for IID generalization but there's nothing that says you can't use the same mechanism for optimizing for transfer learning for what happens when I use my learning in a new environment. Very good. I just wanted to end with a bit of a sociology question. So the first time I heard you speak was at a workshop at Snowbird and I think there were- That's the pre-Iclear conference. Yeah yeah so and I think there are probably a couple hundred people there. Yes. At most and- Good old times. The good old times. And now we live in a world where NeurIPS now we're doing a lottery this year, last year it sold out in 12 minutes. I was in a cab coming from the Sao Paulo Airport desperately trying to register during those 12 minutes and I didn't quite make it. I eventually got a ticket thankfully. But how has that changed? I mean as a field explodes the mean time somebody has spent in that field suddenly dramatically decreases and you get all these people who are new to the field and their history starts post 2000. Right, right. How has that changed? Is that all positive? Yeah so- But it has a lot of positive. I mean progress has accelerated for sure because of the sheer number of people doing research in this field. At the same time of course they don't read the old papers from the 90s and that's just deep learning or neural net papers but they don't read other papers and they don't even know what an expert system is or search, they just even not learn it in school. Like it's just very strange. And there are some negative effects also in the culture, it feels like everyone is in this crazy race for the next first author paper. Instead of thinking about the longer term picture. So when things were slow and there were just a few hundred people working on this you could dream of problems that might take years to solve and quietly hack at these things but now grad students are afraid of someone else in another lab doing the same thing and so they're rushing like crazy. So I don't know, there is something that needs to be fixed. I don't know if we can fix but the culture is too much focused on short term incremental progress rather than attacking really, really hard problems that might take a decade to address. So how do we give the next generation that space? How do we? So I think we should move from the conference publication centered mechanisms to more journal like things where there is no deadline you submit your thing when you feel it's ready and that gives less of a pressure to submit, submit, submit because it's a deadline, I have to submit something. Otherwise I'm not a good human being. So we need to reassign self-worth. Yes, yes. This is what you're saying. And also I think it's important for community service. I mean it used to be that doing things for the community was highly valued and now because there's so much pressure to publish we're losing some of that and it's very, very dangerous. Okay and one last question. So I've seen you give talks before where you end a few slides later and you talk about social good. That's right. And the importance of. And that's exactly connected to this. If we want people to start understanding the impact of their work in the broader society they need to step back a little bit from the next deadline and start talking to philosophers, start talking to legal experts, start talking to medical people and learn about the ethics of what they're doing. So I think it's very, very important because the progress in AI is leading very, very powerful tools and these tools could be misused and they will be misused and so we really need to pay attention to this. Okay and two minutes, 40 seconds on the clock. We have many students in the audience today. What's your advice to them? Don't fall prey to this sort of pressure to publish short-term, work on hard problems, be ambitious and ambitious doesn't mean more papers it means hard problems. Don't go for what's popular. Don't do the things that I say you should do but think by yourself. Fantastic. Well on that note, thank you guys so much.


Here's another home court advantage from Georgia Power, where great customer service is always a top priority. When you need help, stop by a local office, call, tweet, or chat online for support. Learn more at georgiapower.com slash contact us.