So I got to a point where, oh my God, I have to make a video recapping how I did today. I must make money. I cannot make any mistake. So that became like such a mental pressure on myself that I felt like I missed out on a lot of opportunities I could have maximized on in 2021. What's going on guys? We're back with another episode of the After Hours podcast. Today we have a very, very special guest. We have Humble Trader who has become incredibly popular in our trading community. So thank you for coming on. Hi, hi everyone. Thanks for having me. Thanks for being on here. So I wanted to ask you, Shay, a lot of people know you from your YouTube personality. You I think recently crossed a billion subscribers. That's awesome. Well, what a lot of people don't know is that before you were a trader, you were in the VFX scene. So how did you kind of transition from the professional world to the trading world? How did that transition? Yeah, like how did you go from like being in the VFX world to finding a day trader? Yeah, so I was in Los Angeles at the time, working in VFX. I got you working in some studios. And what happens is when you're in the film industry or TV commercial, you kind of are on the contract base. So the contract would last like two weeks, a month or two months. And in between, you're kind of like, unless you're very, very, very good, which I wasn't, you wouldn't, you kind of would have to spend some time looking for a job. So that's kind of why I was inspired to find another way to pay rent is because I have so much gap time in between the job I just finished and the next one that I need to find a way to make money. So that's where I used to like trade and learn trading during those gaps. And then when I kind of have like, at least some idea of what I'm doing, then I'll do that in the morning from, because on the West Coast, I'll wake up at 5am and then I'll kind of get ready, trade a little bit. Market opens at 6.30am. So I'll trade till around 8 o'clock, 8.30 local time and I'll go to work at night or 9.30.  Did you have anyone in your family that was involved in the markets or had you discovered the markets? I found, I found one of the really famous gurus with a Lamborghini some like, I'm sure you guys know who that is. Classic, the class of the same person that got us all started. Yeah, yeah, yeah. So it's him. So I think that he's like the origin is like where everyone got started and how everyone got attracted to the Lambos, right? So that's how it got started. Nobody in my family was ever involved in like the US stock market. We're all immigrants. So they kind of knew something about the, we're from Taiwan, like the Taiwanese stock market, like they will invest, but US markets always been like a very foreign concept to us. Do you ever get to work on any movies that we would know? Yeah, Planet of the Apes, the second one and the third one. Lego Movie, I worked on the second one. So it's not the cool one. The cool one is the first one if I did the second one. Angry Birds Movie, I did both of them. And the rest is like some really lame ones, but those three are the only ones out of the industry that we don't know about. Like what's some like secrets that like no one knows that happens like behind the scenes and like VFX. So there's a lot of drug culture, unfortunately. Yeah, good and bad drugs. Yeah, because sometimes you really stay up all night till 2, 3 a.m. and you need a way to kind of stay motivated. Yeah, makes sense. Yeah, and it's yeah, what else? You know, everyone's really underpaid. You do a lot of favors to make sure you have the next job. You can do a lot of favors.  What's that word in what's that?  That's one of the reasons I hated the industry. It's like, you cannot get ahead by working hard or doing good work. It's all about like who you know. Yeah, maybe you could kind of shift gears to like trading a little bit and we could talk about maybe like daily process because like I don't really know like a lot about like kind of like what you trade like maybe it's just like me, but so I was generally curious when we kind of had this interview because I was like, oh, like we get to learn like a little bit more better. So what's kind of like your kind of like daily process as far as like coming into the market? Like what type of strategies do you trade? What do you look for? Yeah, maybe we could just kind of start there. Yeah, I think the my I guess my process has kind of changed a little bit over the years, but I think the bull park is still the same in which I still wake up early because I'm still on the west coast. I'll scan. I'll usually like just open up my scanner and like trade ideas with Benzinga. Look at the gappers and then I used to focus a lot on the small caps. That's what changed. Now I focus more on the large cap stocks. So I didn't have a lot of money like a small caps, a dollar to $3 stocks, $5 stocks. That's kind of where I was used to focus a lot. And nowadays I trade mostly like the earnings, like the large cap gappers like AMD today, Meta the other day, you know, Google that stuff. That's why I focus on now. And then before it will be like small cap stock and low floats, like the LFIN, the BPTH, those really good ones and that kind of transition during COVID pandemic market to like a lot of mean stocks. That's a lot of large caps. And now I trade like the mega caps. Are you struggling to find the right stocks to trade? Are you making money for one, two, maybe three weeks in a row and losing it all in one day? Are you struggling with sizing up on your trades? Well, I found that the best way to fix these problems is to trade alongside a millionaire trader live on stream with no delays. My investing clubs new live trading stream is the missing puzzle piece to your trader. Every day at 9am, I share my screens, find the best stocks to trade and build my watches live. Then I execute those trades live on screen. It's like you're sitting right next to me as I trade every single day due to the high demand of the screen share, we only have a few spaces available every single month. So if you're seeing this now, it means that space is still available. And here's the best part if I do not generate $20,000 in realized gains per month on live stream, you do not pay. So go to myinvestingclub.com slash live trading to get 50% off your first month on the live streams. Spaces are limited and filling up quick. What do you have to lose? This may change your life. That's really cool. Was it hard for you to transition from like starting out with some small caps and getting into large caps? Because it obviously is such a big difference. And like one, it's like less pre-market prep and like, you know, obviously it's just different structure. Was it challenging for you to make the change or were you already like, were you not finding success with small caps before you made the change? I don't think it was difficult, but I think I think I've always been, I'm a very organized person in trading also outside of trading. I like to kind of categorize things into little buckets.  So I'll categorize that. And then large caps, it's like, okay, like if I'm looking to buy the dip, I'll buy it instead of at the open, I'll buy it around like 10 o'clock or 12 o'clock. So I kind of always kind of categorize them separately. So I can kind of know that, okay, if I'm trading a large cap and you do just that, small caps, I only do that. So I think it was also a forced transition because small caps, you should kind of, this summer is a little bit different, but small caps, it will be summers before a pandemic that I wouldn't have any single green trade on the small cap stocks, because they will move like 10 cents. And, and I'm not used to that. I didn't realize they're so cyclical that there might be like months in between from, you know, June to October before I placed any successful trades. So I kind of was forced to adapt. If I still want to make money during summer, I kind of have to trade the mid caps, large caps, mega caps. So kind of out of necessity, I think. Yeah, was it hard for you to find like a strategy with large caps, or like, did you kind of base it off something that you've learned in the past, or have you tried it all? Have you tried shorting, like longing? Like, how did you find that breakouts or the like earnings plays or more your, more your speed? Yeah, so for the large caps, one was trading part-time. I started out with just at the end of the day, after I came back from work, I look at the large cap gainers on the day. And if it's during earnings season, you get a lot of like big runners that trend out really nicely on the first day of their earnings breakout. So what I used to do is a very, very simple strategy. Anyone can do this. The next day, I'll look at like, what's like the key support from the day before, the first day of the large cap breakout, and then I'll try to buy the dip, or I'll, if I miss the dip, I'll wait for you to continue the trend and break out through the higher day from previous day. So essentially a breakout strategy on second day, and it'll go higher. Not always, obviously, but the probability of that success succeeding is really, really high because usually earnings, like if you really, really beat earnings, the stock will run for multiple days. So that's the very, very simple strategy I used to trade a lot. And then I kind of developed that into, okay, then why don't I just buy the stock on day one overnight? Why don't I just swing it overnight? So kind of developed into also a swing overnight strategy as well for the larger cap stocks. So you mostly primarily just been like kind of like longing large caps, forgetting about small caps. Was it, was it kind of like hard to let go of small caps? Or you were just like, no, I'm into large caps now. I'm not a small cap trader, kind of like that kind of motto or? No, I would say I kind of made that transition after analyzing my trades. Like all my biggest losses in 2020 were on the small caps and the biggest gains were not small caps. So I just, I understand they are like the fun, fun, you know, dangerous stocks to trade. I feel really excited. And, you know, I feel like I feel really excited to trade them, but I have to kind of face the reality like small caps when they, if I was shorting, because I got really into shorting small caps in 2019 and 2020. So and then 2020, everything went to the moon. So it was really difficult for me that year shorting small caps, especially a low float. So I kind of just, yes, nowadays I still trade them, but it has to be like after it's really extended for four or five days, like TUP, I didn't trade it until it reached $5 yesterday. So all the way from the, from when it went from $1 to $5 the four days before that I didn't touch it at all. So I'm just a little bit more selective on the kind of low float small cap stocks that I want to trade nowadays. I still trade them like once a while. I just want to, I just want to like trade it. You can't get away from it. It's amazing. They're so exciting and there's so much money to be made, but a lot of the more conservative day-to-day money could be made from large caps and it has no locate fees. Most of the time, the commissions are low, if not free. So there's a lot of opportunity there. Have you ever thought about trading forex or crypto or anything like that? Not forex. I have thought about trading crypto, but by the time I was so late into the trading crypto game, by the time I was trying to set everything up and I figured out how I was going to transfer money into my wallet to buy this and that, the market started tanking. And then with what have happened with the FTX last year, I was just like, I'll forget it. There's no point. What are your points about your trading? Because I grew up in the Middle Eastern family and unless I was making a bunch of money, they just kind of rejected the idea of trading.  I'm going to become a full-time stock trader. Did they give you any backlash? Well, I knew they wouldn't understand. So I didn't tell them. Oh, wow. You didn't tell them? I didn't tell them until I had already quit my job and was full-time trading for a year. Yeah. I mean, I knew it's not like I needed them to support me or financially and I was already living on my own. So I just knew it would just create unnecessary stress for me if I had told them. It's my money, I'll do what I want. Yeah. Yeah, I just gotta be smart about it. I mean, I'm just saying a family, obviously, but I know my parents, what I worry a lot and like nag at me, so I just tell them. Yeah. Did you have some sort of savings built up from the career before you said, you know what, like, let me just have a little bit of a nest egg first and then I'll kind of transition over. Yeah. So I saved up about one year of living expenses outside of my trading account before I kind of quit Co-Turkey. Yeah. Because a lot of people think that trading, all right, I'm going to become a trader, I'm going to quit my job tomorrow, but the reality is that it takes time to be able to save up a little bit of a nest egg, have a little bit of a cushion so that you don't feel like you're trading to make money. How many times have we traded forcing trades to make money and we end up losing money? So I think that's very important. I'm glad that you built up a nest egg before that because a lot of people listening, I feel like they may have the wrong idea of trading because we got into it for the Lambo, the Get Rich, I'm going to make money, I'm going to travel the world, I'm going to trade on the laptop and this and that, but the reality is that trading is more complex than that. If you want to make more money than a doctor, a lawyer, or an engineer combined, it's going to be a little bit tricky and because it's going to be a little bit tricky, it doesn't mean that you should be quitting your job. In my opinion, I think most people should just trade part-time. For me, at least, I make most of my money in the first hour. So if I didn't have MIC or community or other things to do, it's a perfect way to stay busy throughout the rest of the day. So I don't think anyone should quit their jobs or trade full-time. If anything, I think that trading should be a way to supplement your income. Yeah, for sure. When I first started trading in LA, I was just trying to make enough to pay rent. I remember I was living in a two-bedroom apartment with two other roommates, so one person was living in the living room and we were paying, I think, $600 or $700 each. So I was just trying to make that in one month. So it started out doing that and obviously it scaled up slowly, but that's why I wanted to pay rent in LA. That's it. Of course, in the beginning, I thought I was going to get rich, but once reality hits, it's like, okay, I'm happy just paying rent and then help with student loans, eventually I pay more bills. But yeah, I think part-time trading is a way to go at least for the first two years, for most people. Was it hard for you to mentally adjust to having a set paycheck before, because I assume you're doing decently well at your other job and then to go to trading? Well, I mean, you were able to take up a year's worth of money, a year's worth of money in a trading account, so I think you were doing all right. So I think I will say this, if I had a cushy job, I don't want to offend any engineers watching this, but I know the engineers down there are working 20, 30 hours a week getting paid at 300 grand. If I had that job, I never would have went into trading. Like, come on, there's no need to. But for me, it was really unstable. Like when I had the job in film production, yes, I'll make decent amount of money for a couple of months. But it's the in-between, like the one month where I don't have any job that's extremely unstable and stressful for me. So because of that, it was hard for me to save a lot of money at once. I kind of work job, and then I like flip stuff on the side, I flip couches on the side, and I did all the things. Couches? Yeah, yeah, furniture. Oh, okay. Yeah, couches. Yeah, yeah. I'll go to the, like the... Goodwill. Goodwill, yes.  I found the Burberry Jacket ones. Yeah, yeah. So grandma passed away.  So you went from a very stressful environment, which was the film production world where you have a job today, but you don't know if you have a job tomorrow. You don't know if your funding is going to get cut from the movie. You don't know what's going to drop here or there. You don't know if they're going to change anything. Yeah. And you chose day trading. Yeah. What led you to make that kind of like, this is really stressful. Yeah. Why not do something that's equally as stressful? So what was it that all of a sudden you went, you know what, if I can handle this, I can do this? I think you are right. The stress level for trading is probably even higher. But at least, you know, after the first six months, yes, I lost like pretty much blew up a couple of times. At least I understood that if I lose money or make money, it's because it's all because of myself, right? There's no one else I have to rely on where I could blame. Whereas when I was working in the film industry, sometimes, sure, it's me who get it late. But a lot of times it's like due to some other people's, you know, progress or the producer had a change of heart, director want something else. So a lot of different changes because of outside scenarios and outside reasons, not because of me. I pretty much prefer it to just be myself responsible for my own outcomes. So more it was it was more about that ability to control it. Yeah, really to control whether you made it or broke it. Exactly. Yeah. So I like that. Let's say let's say someone goes out, they quit their job. What would be the first thing you suggest them to learn? Like would it be like technicals? Would it be fundamentals? Would it be like supply and demand volume? What would be like the number one thing where you know, like that you would say like, hey, focus on this and not like Bollinger Bands or something like that. I tell them to focus on how to get a new job. Yeah, that's all I'm thinking about. McDonald's hiring. Yeah, I mean, I don't think I never recommend people to quit and like start trading. I feel like anything else is probably a better idea than trading right after you just lost a job. Yeah. The problem is the industry is so like get rich quick money. Yeah. So like you don't even think it's stressful. What are you talking about stressful? You make money in your sleep, right? So I think a lot of people don't understand what comes along with trading at the real professional level, which is not easy, but it is extremely lucrative if you can do it the right way. And something that I think is great, Shay, is that you make a lot of content on YouTube to help the newer struggling traders. So I know that they're in your comments all the time. They're messaging you, they're talking to you, but for the new and struggling traders out there, is there something that you see that is the most common issue that people are struggling with? Yeah. So they usually will try something for like a week and then immediately jump to something else. So they'll try stocks trading for a week. Oh, Longling isn't working. Let me try shorting next week. Oh, stock shorting is not working. Let me try options. Oh, options not working.  Yeah. Yeah, you're right. So that's the biggest mistake. I think you need to like at least stick with something for at least a couple of months to really know if it's working for you or not. Yeah. And then they open an Amazon drop shipping account and that's going to change everything that they do.  How did you know I did that? Go from trader to drop shipper to forex trader, right? Right up the ladder. Yeah. We were that guy that we were making fun of back in like, what was it, 2018, 2019 trade neck? Oh, and then it was all the CFD business and stuff like that. And what was that guy? Trade trade net. Trade net was the guy. Yeah. And then there was that dude that came out with the meme that's neck was like from his ears to his shoulders. And everybody started calling it trade neck. It's to be honest, I've not met one successful forex trader. Most of them are freaking scammers, man. Every single one.  I met one that made some money, but he decided to switch to his trading stocks. There you go. Yeah. There you go. So what drove you to start posting things to YouTube? I know that before you answer that question, I'll preface to this. We mentioned this before we started the recording, but Humboldt Traders YouTube channel has now passed a million subscribers. So props to you. Thank you for all that you do for the community. What was it during your trading that encouraged you to start posting? Yeah. So after I went full time trading, what happened is I kind of lost that creative outlet. So I was working in film and VFX, then I stopped doing that. So there was, I'm naturally a very creative person. I love making videos, telling stories. I used to write a lot of stories as well. So I kind of lost that outlet. So that's number one. And number two is that because I started out trading part time, I used to finish all my trading before my local time would be 8.30. That'll be 11.30 a.m. Eastern. So that was where I found my success. But when I went full time, I was like, sit there from nine o'clock market time to four, and I'll sit there for an entire day. I'll make money in the morning and lose everything. Yeah. That's usually what happens. Yeah. So I realized I needed something else like a site project to either keep me distracted or just like a creative outlet. So that's why I kind of turned into YouTube to do both of those things. Yeah. And it was really fun. It started out like a little hobby project that would work on the evenings and Sundays. It takes so much time. And then now eventually, it's like, what, three, four years later, it's like a full-on business. Yeah. That's crazy. I mean, it's cool. But is it hard for you to do this because you're so public and your videos are great, by the way? But is it hard like when you're having like a rough stretch of trading, like when you're like having like some down days or whatever, to still post and be like honest? Because like at MIC, we post like day in, day out, no matter how good or bad. And do you ever struggle with that? Yeah, for sure. I used to like post a lot more about my trading recap. I used to film recaps every day or every other day, because every day would be a little bit too much for me. Every other day, I used to do that. I did that for a whole year. And then it got to a point of like, it became a little bit too much for me. I understand like, I'm sure I'm old for transparency. People would love to see your wins and losses. I did that for a whole year. And I realized it was kind of affecting me and my trading performance. Because whether you made money or you lost money, like at the end of the day, you should be posting and doing the recaps for yourself. So I got to a point where, oh my God, I have to make a video recapping how I did today. I must make money. I cannot make any mistake. So that became like such a mental pressure on myself that I felt like I missed out on a lot of opportunities I could have maximized on in 2021. I feel like I made a lot of money in 2021, just like every trader in the market, but I feel like I could have made a lot more. Had I not been so focused on, oh, I got to keep up with the video recaps. So I think whatever you do, whether it's a recap, a post, written or a video, you have to do it for yourself. And you have to do it in a way that serves you as a trader. Yeah, I think that's super, super, super true because it's a lot of pressure to do that stuff as well.  It kind of hurts you. And then the flip side of it is that if you're always posting and always posting, you almost feel obligated to post. And you may trade a little bit differently based on your wanting to post. So I used to post a daily P&L for years, daily, daily, daily. And I found myself kind of getting lost in the sauce every time that I did it. And then at the beginning of the year, I was like, you know what, let me start that small account challenge, let me grow that, let me do the daily P&L. Then after that, I'm just going to relax and do whatever I want. But for me, it's just, I totally get it. I totally understand the pressure and props to everyone that does it every day. But at the end of the day, no matter if you post it or not post it, no one's going to be happy for you. Everyone's going to be jealous.  Everyone's going to say it's fake. Everyone's going to say, you're not real. So at the end of the day, what's the real point? What if it's for you, if you're doing it for yourself? I think that's really important. Something that I was talking to Jack Kellogg about is his losing days.  I don't need to be doing this right now. I have this, this, this to improve and he keeps to himself. He keeps it to himself. And whenever he is feeling down or whatever he needs that reassurance, he just whips out that video for himself. So I think it's important for people to do it. But to post it publicly, I don't think it's necessary. You know, yeah, for sure. I agree. I think it's fine. It was a lot less pressure when I was smaller. But now with over it's with with this much subscriber is I'm great. I'm grateful. I appreciate everyone supporting me. By the same time it's immense amount of pressure and like everyone expects you certain things from you. You get recognized in public a lot. Yeah. It's like usually at like really inconvenient times. One time I'm trying to catch a flight at the airport. Yeah. And then and then I one time I was like playing pool or sorry playing ping pong with my friends at a bar and then we were all like super drunk. And that's not the best. The best one is when you're middle of a meal and you're just like yeah. Yeah. That's me and photos. People are like let's take a photo and I'm like I got a hot dog stuffed in my mouth. A lot of people don't get it. They just think you show up every every week and you make a video and you just record it and it's ready to go. But like I think a lot of people don't understand how tough that creative process could also be because like how many times are we going to cover level two how many times are going to cover VWAP how many times are going to cover this like eventually the topics that you know are interesting almost become obsolete. So how do you find that creative motivation to create this type of content that people have not seen in content that you also enjoy creating because I learned that if I just make content just to create it is not like it's like a job. Whereas if you're making content that you're actually interested in like we're doing this like this is fun to me. Yeah. So how do you how do you what is your creative process like in making these training videos. I think I would say because it's now my fourth year on YouTube it does get difficult. I want to say every year and a half or so like I'll run out of completely be out of creative ideas. So I'm actually currently going through a stretch like that myself right now. I'm like oh man I don't know what to make you know I'm I'm so tired of these strategy videos they work but at the same time you know I only do like five strategies like that's all I have. I cannot create strategy just to put on YouTube and I don't want to do that. So sometimes I do go back into my earlier ideas or videos and like that will sometimes inspire me you know like oh four years ago I presented certain things this way. I can try a different way of presenting it. So you're repackaging your original idea or sometimes I just talk to other traders, talk to other friends, talk to non-traders. I think I think it's important to talk to people who don't trade and see what fascinates them about the stock market. So that might help you out too. So I do that I talk to my friends who are not in trading who are not in the stock market but they want to learn they go ask them oh what do you want to learn about like how in what kind of way would you learn the most what like are you what's your focus on are you trying to make you know set hustle money like a hundred dollars a day or are you trying to you know just like pay for your bills pay for a mortgage like what's your goal. So just really trying to understand that that also helps and then and then otherwise taking some breaks this helps a lot. Like when I was in Europe I had some couple of decent ideas when I was away from the market and from work.  I probably have gone through that twice now. Do you have any psychology and discipline tips for any traders like who are kind of like struggling maybe with like the psychology of it all or just like the discipline to just execute trades the right way and execute kind of strategy the right way. Yeah so execution is the toughest one because you cannot learn execution from reading a book. You have to like I tell everyone to subscribe to the Dash trader replay it's like ten dollars extra per month and then retrade everything you either messed up on or you could have improved that day. So I used to do that I still do that quite a bit I'll probably do that once every two weeks nowadays that's how I got better with execution because you cannot learn execution from yes you can kind of learn from seeing other traders charts but you need to be pressing the buttons yourself. Yeah. So that's that's I feel like that's the only practical tip anyone can use to learn execution. Now psychology wise reading books does help a lot with psychology there's a there's a lot of books out there like trading the zone trader trader read and what's that book the the I'm like daily trading coach. Yeah that one helps a lot but there's a there's some atomic habits. What's that atomic habits that one helps a lot too with by building a really simple and lean trading routine and not get distracted that one helps with my discipline a lot but yeah otherwise it's just I think also outside of trading if you have like a I don't know if anyone here is interested like has like a workout routine so I used to be really really strict with my workout routine that helps a lot because if you have like a really strict regimen that's gonna carry out into your trading as well. Yeah that makes sense and I have a pretty strict restaurant routine. Eating routine is pretty strict. It's strictly terrible for myself. Does that help with trading? I have a lot of haircare routines. That helps too. You did some like pole dancing and stuff like that too didn't you that's pretty wild. Yeah that helps like I think a lot of people don't understand how much like just any exercise at all just can help relieve stress so aside from exercise do you have any tips to relieve stress after a stressful like a tough day trading? Yeah go on walks as lame as that sounds. I pretty much go for a one hour walk every day after the market. Yeah just to think reflect on trading and you you feel a lot better after you've done the walk because you're like all by yourself right you're like in your mind a lot thinking and it forces you to be along and reflect on what you did well what you did wrong. Rain or shine? Sorry? Rain or shine? Yeah yeah yeah in my case when it's raining you still walk and we don't need some dollars. So now what you need. Yeah go for it. Sorry. I was gonna say so you said earlier that you're super like organized and like you have everything like set in place and I feel like listening to you like you have your trading and that you have your YouTube and like what are your goals with both of those going forward like what is it what is the next five years look like for you? So for my trading I still want to trade daily but I want to start I kind of started dabbling with selling options on the side that's kind of I want to create like an income generating portfolio in which you kind of it's like a you kind of set it and forget it for a couple months because you're selling the premiums so I kind of started watching some videos learning a few things so I want to do that on the side because the more you make whether it's from trading or your businesses the the higher risk you have and I don't really want to keep on risking that much money what I used to be able to risk 50% of my portfolio now I don't really want to do that because the more you make the 50% becomes a lot more so that's kind of what I want to do and and start diversifying more into other asset classes whether it be real estate or other businesses I don't want too much money tied in my trading account anymore because I think with the especially what happened the last two years with the crazy crazy low floaters like the the amtd the the top there's a couple of trader friends I talked to unfortunately they made a lot of money but they kind of blew up just for with one trade so that kind of like I'm sure they'll be okay to make that back but they lost a lot right so for me I'm just I just also realized that the us traders we are our the number one threat to our trading account so I just don't want to put that much money in my trading account anymore I want to start diversifying and put it somewhere else where I cannot touch it so that's kind of my goal at least with trading is to actively trade less and be even more selective with the trades I take now with YouTube I do YouTube is where kind of like I don't really care if I make money or not with YouTube the like oh like I have a team now so I kind of reinvest everything back into my team so I want to do a lot more like traveling and trading content that's something I want to talk to you about later Alex so I want to do more of that so I'm less tied down to my office in Vancouver and be more kind of have a more global footprint with the content I create and like visiting a lot of traders around the world that's kind of what I want to do next step with the YouTube side that's great I mean is there anywhere special that you've been aiming to go to that you haven't gone through yet uh in terms of well I think eventually I'm going to start out with like the North American cities and then eventually I want to hit up like Dubai Singapore there's a lot of big traders there a lot in the in the future space so I want to see them and see how they trade and what is it like living there and trading there versus in North America yeah that's cool I like I can't even imagine what it would be like to live that life because I've been like in Boston so long and like I like travel and stuff but like has that always been a passion for you like always wanting to actually travel and like because I think part of becoming a trader is having the freedom right and you're the ability to make money wherever you want to be and also with your other job with YouTube you can kind of do that too was that something you wanted to do forever or is that just kind of more of a recent thing now that you've kind of like quote unquote made it and like just have like a new kind of goal and aspiration um I think I really only started doing that in I started doing that during pandemic travel a bit more um but I don't I think I kind of realized with all the stuff I've done whether it's VFX or flipping furniture like like I'll do that for a certain phase so I don't know if there's anything I would do forever even even with trading like you know trading I think I will evolve eventually who knows I might stay at what I might stop day trading and just swing trade and sell options who knows um so I just know that if there's something I'm very interested in I'm going to do that to the best of my ability for now a couple years and then see what's next so yeah I'm just being kind of following what I really want to do and what I'm passionate about and traveling trading at this right now that's kind of what I want to do is would you say that traveling is like your favorite hobby outside of trading or do you have any other hobbies that you like to focus on when you're not trading um so pole dancing is one I still do that um it's really fun but also I'm actually really dangerous um I'd say well you if if you lose your grip you can fall right just screw your head up you're falling in very odd positions yeah yeah so that's that everything else just on top of you crushing your body in its game over sometimes you're speaking from experience yeah well I attempted and that's pretty much what happened at least you tried did you make some money no no I got a lot of pity I got a lot of pity though that paid for itself in the long run that's pretty much how I got married I think actually that's how I met my wife I was injured after attempting a pole dance move and then yeah I was like you like what you see sign here press hard three copies you scan the real good then Joe would you say that it's been or hasn't been hard to kind of like find a partner or like dating wise like what has that been like trading because I know like for most guys that I know that are trading it's been like really difficult if not really why I want to know a lot of people that really struggle with like relationships and stuff like that especially in trading because like you're trading all day and then sometimes you don't want to talk to anyone or sometimes like I have a girlfriend myself and my girlfriend just kind of gets it you know like even if I'm in a bad mood like she loves to read so she's like I'm going to read so like that's kind of like how we kind of manage that which is super good but yeah like for you like has it been hard or has it just been like easy like anything else you know I would say it's it's not hard now in the beginning while I was struggling with trading I would say it's too like kind of before and after right before when you're struggling with trading you're really insecure like there's a shame when you're losing money day in and day out you're ashamed of yourself so even if you try to go and meet people whether you're a man or woman you just feel really insecure and when you are insecure that kind of carries out to like how you interact with other the other people so obviously it was really really crap when first that was when I was in that phase and then I would say it got a lot better but I'm really introvert so I don't like talking to people so so when I would say one when I feel like talking to people it's not it's it's easy but when I'm in that phase where I just don't talk to anyone then obviously I'm not even trying so when you get to go on YouTube and your personality on YouTube is so bubbly and bright right what was it that you had to do did you have to physically tell yourself like this is a character I'm playing this is a role and I need to be this I don't get to be reclusive and just be by myself I have to be out there no no no I think for introverts it's like you're at home I would say four out of four days out of the the week then when you do go out or when I do go on camera I'm like yes I like conserve all these energy okay yeah so I feel really happy talking to you guys I haven't seen anyone the last three days I didn't talk to anyone so so yeah that's okay I have a podcast schedule today three days before I'm doing anything everybody who's texting her gets left on red they're so pissed they're like what the yeah yeah mom I'll call you back after this before we wrap up Shay I wanted to ask do you have any questions for us is there anything that we can answer anything that's particularly on your mind um yeah how's how's how's your training this year I know you guys focus on small caps which I like look at once a while now but not a whole lot how's the small caps this year so small caps the beginning of the year started off really strong for me I turned my 35 I do like a challenge at the beginning of the year where I started like a $35,000 account and I see how far I could grow it at the beginning of the year I was able to grow that $35,000 into a million in 55 days because like a record it's like the most unbelievable thing ever for me but towards like the middle of the year ish like I would say maybe about like after the first quarter I started to get a little bit sloppy in my trading because I was so confident I was so overconfident that I started over sizing on some stocks I started getting a little bit sloppy on some stocks and towards the middle I kind of got you know up down up down and now um what I'm trying to do this year for myself is I'm trying to size down more and focus more on finding internal happiness outside of my P&L because I would judge my happiness on the day based on how much money I made right it's it's pretty sad to say but like that's what I used to do if I had a green day I'd be extra happy if I had a red day I'd be extra sad and I felt like that that was kind of controlling my life for a little bit too long so now I'm kind of focused on sizing less making a little bit less money but focusing on finding happiness outside the market so that's why you know eventually I want to move to Florida that's why you know eventually I'm gonna take it a little bit more serious with my girlfriend but aside from that um I've it's been a little bit summer's kind of been a little bit up down I would say the beginning of the year was strong and now it's kind of just like chopping around and yeah okay yeah honestly for me it's been pretty well I had a big kind of like last like three years in my trading where like I really like did really well um and now I've been focusing on like swing trading and uh more kind of like strategies like that you know because like as you make more money like you know you just don't want to scalp for like ten cents anymore no offense to those people you know but um that's kind of what I've kind of been marking on jam uh my biggest thing this year is like I'm I've always been like I'm going to knock on every piece of what I have like I've always been very consistent and my biggest thing this year is just kind of like fighting greed and you know we're very lucky and like I think we're very blessed to be able to make like a few thousand dollars a day like kind of no problem and it's just kind of fighting that urge to like really push myself and like just being really happy and enjoying what I'm doing now so but it's been good and the market's been all right so okay I was probably the least active one out of all three out of all four of us for 2020 and 2021 I burnt myself out I was pushing as hard as I could with the large caps which is what I do is large caps and options I was pushing really hard and uh a buddy of mine that we trade together both of us just we burn out and for 22 I pretty much did very very very little like the bare minimum and then in 23 man we moved from Texas to Colorado and I don't know what it is but this time zone just works better for me and it like reignited reignited all that burning urge to blow all my money gamble risk you know you know bet on crazy shit and um yeah come back to small caps show what I said why don't you come back to small caps with me because we did so well the first time he's betting on a horse in the background I had I had James transitioned for a moment I had him I had him thinking straight for a little bit and then Carvana happened to him and he gave up on me Carvana was great oh man no I have to say this is kind of a quick change but like I really do appreciate you coming on uh Shay because I I'll be like scrolling on Instagram or YouTube and like I see all your videos all the time and it's like it's cool by doing these podcasts that we actually get to like talk to you and like talk to these people and like there are real people behind like the the trading videos and all that stuff so I really appreciate you coming on yeah thank you for having me yeah that was awesome this is awesome thank you thank you yeah and it's obviously it's nice to I met Alex that was fun we had uh you and bow I want to say that I you know when I first started like I obviously didn't have the youtube channel till 2019 so I've been trading for eight years so from 2014 to 2019 the five years I kind of just like looking at you guys silently um quietly in the background like on twitter and I follow you and bow and a couple of the traders it's it's been really really um great to see your progress what you and bow have done and you guys were also my inspirations in my early years as well so I just want to say that and that's a while about uh meeting Bao and then him visiting in Vancouver that was also I don't know if I told him this that's why I was hoping you'll be here I'll tell him next time I see him like he is also because because I don't see a lot of Asian traders until recently so for me to see someone like him this is like eight years ago that was that also made me feel like oh I can also do this like he has black hair I have black hair he looks like that I look like that I could do this and that's that's that's kind of how everyone passes the elevator back down I'm sure there are so many female traders as well you know there's probably so many people that hit you up like hey like you're a girl I'm a girl like I could do it too right yeah so yeah like I think in trading besides the money meeting traders and where and learning about everyone's stories and where they came from how they found trading that's being one of the most enjoyable experiences um in terms of this industry yeah love it that's really cool thank you Shay thank you for coming on thank you guys thanks for having me and thank you Joe for the dad jokes that's a specialty that's all it's always got it's all I got is self deprecating humor


That's a good one. This is him. Yeah. I'm going to pause it for a second. I'm not going to be claimed that it's quite easy. We will thank you. It seems it's true. Luigi, are you a French refugee? I'm where we are. Hello, everyone. And welcome to Ned's seminar. Today's speaker comes from Italy. Luigi Rizio, he comes from University of Piz and he's also spending some time with the Google for six months. Luigi has done quite a lot of work in networking and systems, so if you use dummy net at any time in OSX or to GSD, you have to thank Luigi for this. He's also done work in packet scheduling, congestion control, and today he's going to talk about some recent work that he's done on a framework for fast IO called NEPA. So, Luigi. Okay, thanks for the introduction and feel free to ask a question. We're going to do it in the talk. I'd like to answer or postpone the answer. I'll cover the topic later in the slides. So, the goal of today's talk is to show you how we can improve the performance of network processing in standard OS in ways that probably are not matching the original design of the network stacks in a few weeks' systems. And the reason why I chose this picture is because in the end you would see the solution that I used and you would say, oh, was it so easy? And yes, in the end it was so easy. But I've been working in this area for a long time, 15 years and I felt so stupid when I designed the network because all the technology that I've been using could have been used 10 years ago so it was easy but after you have the solution everything is easy. So this is joint work with some of my former students and colleagues and also it's funded by the European project every now and then there is good outcome from European projects. And if you are bored by my face you can look at the slides online. So the question is the following suppose you have to do some packet processing applications what are your options? And I'm referring to this here what are the possibilities that you have with the mainstream operating systems? So one option is to use the sockets. The socket API is very flexible you can do almost anything including sending and receiving grow packets. It's been available for a long time on the operating system. It has only one problem, it's very low and it's very low for a number of reasons that we will see. The other option is to try and use some direct pocket systems that exist. For instance Linux has a pocket or a viad which is called pf ring. There are other systems that also expose the hardware and use the space so you can program the registers of the car directly. The problem is the following here. If you use one solution such as pf pocket pf ring the underlying mechanism to move packets between the network car and the operating system is still based on SKBUFFs and NBUFFs depending on the operating systems and that's a very low mechanism. So it's like having a very fast car and a very fast engine but you're using this car on a road which is very slow so in the end you would not be able to exploit the performance of the system. The other option is to run your code directly into the kernel which is great except when you make mistakes and try to run more locations and crash the entire system. And of course there are manufacturers who sell you hardware and software together for instance Solaflair is a library which is called open on load which is a West Bypass library which lets you use sockets from a user space application and control directly the hardware bypassing the operating system Intel has the PDK it's a similar library this is tied to vendor hardware so you don't know what's going on inside the system you're right unfortunately if you're moving to a different target or the vendor goes out of business you're in trouble because many of the extensions that the vendor has supplied are probably not available anymore. So I was looking at this program and trying to figure out is there any way we can implement direct packet I.O. from a user space application directly to the one in West Bypass and can we do it in a way which is fast, which is safe and it's software from crush risks that you have and it's all very independent and yes there is a way and we eventually built a prototype which is quite flexible it's even quite small in terms of code size which is good it's maintainable so I don't have to rewrite the device driver from scratch which is a major piece of work that would be a major piece of work otherwise and the results that we achieved are shown in these slides for instance if you say you have an application we want to send packets row packets to network interface if you use sockets you have this kind of performance so I'm using I'm plotting this graph with variable valuable clock speeds because our system is so fast that it saturates the link at the top speed so if you want to see how fast is our system we need to underclock the CPU to get the exact performance point but anyways if you take a socket based application you are getting more or less one minimum packet per second at the top speed and then decreases linear with the clock speed there are better solutions for instance in Linux there is a kind of model which is called packet gen some of you might have used it and it peaks at about 4 million hearts per second and with our solution with NetMap we separate the link and clock or at 9 other megabytes so there is more than one other margin so I'll probably talk about this later but I guess one other way of sort of measuring it would be to change your packet size and use like minimum size packet this is all with minimum size packet awesome so yeah many papers you read you will find the performance in megabits per second or gigabits per second here in the medium of packets per second so one 14 million times 64 equals 10 gigabits if you include your framing so yeah 14.88 is the magic number for the gigabit 64 by 1 yes I was saying NetMap has this performance it uses its own API which might be a problem if it forces you to write all of your software fortunately the API that NetMap exposes is very simple and compatible with LeapPickup where you have a file script so you can select on it and in fact we have a LeapPickup migration library which is very small to run an unmodified application even without very compiling on top of NetMap now the rest of the talk will be on how we managed to do this and hopefully some of the ideas that we used can be applied to some other complexes and some of your projects and so the first thing we did actually it wasn't the first thing but after we built the system we started to figure out why it is so fast so at some point we measured the time of the stack and trying to figure out where the time was spent in sending the path and you know immediately if you look at the code the network stack is heavily layered so data from the socket down to the network card pass through several layers and in principle you might think ok maybe there is one place where most of the time is spent and I'm good and reality is not like that unfortunately but to realize that what we did was run a set of measurements and the thing we want to measure is the time spent at every layer in the network stack and those times can be very small tens of nanoseconds or even less so small that it becomes really complex to measure them even using the TSC timer on the CPU so we did it in a different way we instrumented the code so that at various stages in the flow from the socket to the device driver we basically insert some written statements freeing resources if it's necessary and then we measured the packet rate aborting the processing at different stages and by the receiver power this number is the time spent to do the process the packet up to that level and then by difference we can compute how much time is spent at every layer and of course in order to do this we needed to run a fast traffic generator in user space make sure that wasn't the bottleneck and then move down the stack and see how much time is spent at every level and the measurement went down on relatively cheap system with a tiny interface at the bottom you don't need a interface until you reach this point you don't need an echo card until you reach that point but anyways and this was some of FreeBSD Linux is somewhat similar maybe slightly more efficient in some cases because it uses different mechanisms but for instance the device driver is as expensive in Linux as it is in FreeBSD now the numbers are like this so the way to read it is stable basically in the second column you see the list of functions that are coded from sender to down to sending the packet to the wire and this column tells you how much time is spent for sending a packet if you abort processing at that level so if you abort right before send to it takes nanosecond for the user space generator to create a packet if you abort after IP output sorry before IP output it takes 330 nanoseconds and so on and if you get down to the wire you spend on FreeBSD approximately 150 nanosecond now if you look at this table you see that apart from having 10 or more different layers that there isn't a single place where a lot of time is spent there are many places and you see them emphasising this column there are many places where there is a significant amount of time spent in that layer for instance just in the device drivers 120 nanosecond consider that at the top packet rate on a technique interface your budget is 67 nanoseconds per packet so it's already a short stop and the ITER output the macabre system for the packet spends another 160 nanoseconds IP output computes the IP header another almost 100 the system call at the beginning takes 96 nanoseconds and that's basically something you cannot avoid if you want to move from protected ring to ring zero you need to pay that cost can you just explain again measure dates so yeah, okay modify the kind of sources for instance at some point the code calls the IP output right before the code calls the IP output yes, right before the code calls the IP output I return instead of calling the function that's one point and then I can make the return right before the IP address and so on so I don't need to collect time stamps on every packet but I just do the measurement on one second I see how many packets I can make in one second and then now, it's very similar to the picture I showed before with all the stages now, how do we fix that how do we move the code or how do we modify the code we can send the receipt packets 16 nanoseconds that we have available as I said, the system call is the one thing that cannot be avoided you need to pay the cost if you want some protection and you do want some protection because I want my system to be safe so the way to reduce the cost of the system call is of course amortizing it on a batch of times some things can be safe for instance a lot of time spent in the device driver is for programming the device to send one packet at a time and again, if you try to move patches of packets from the application to the network card you can amortize some costs again some other costs can be saved, for instance those related to data copies and buff allocation the internal representation in the kernel in the network packet turns out that your current system allocates entries about every time you want to send a receipt packet that's a very expensive operation for certain application you don't need to do that or you can do the allocation in advance and then recycle the buffers as you produce traffic or receive traffic and so on, there are many, many tricks that you can use in different places but mostly it's about removing unnecessary operation and amortizing the cost of the performance one thing which is a particularly bad idea is the fact that the internal representation of packets in the kernel it's very complex basically the end buff or the escape buff can be a list of chunks of memory where you store the packet and that was very convenient 30 years ago because memory was expensive and the packets were by step basis you start with the payload the TCP adder in front of it and an MP adder and so on so you didn't want to copy data at the time and so the most natural way to create a packet was to create a link list of buffers with the small chunks representing all the bias layers and in recent times what happened is that people found that the packet processing was expensive so started to offload features to the network card for instance when it turned out that IP or TCP checksum were expensive cards came out with hardware checksums when it came out when VLANs came out again cards implemented VLAN tag insertion and removal in hardware and in order to tell the hardware whether you should use or not these features the packet insertion was also extended so in the end it's like playing a Tetris game with the packets that have very different shapes and the network card has to double with those shapes it would be like it would be a lot simpler if you had a Tetris game like the one on the all the blocks are the same size very uniform and you will have to think when you process them and this is exactly one of the things we did in the Tetris game so now the measurement was for sockets when you do raw packet IOC you don't need many of the layers that the socket implements you don't need to add the headers you don't need to add mac headers the packages are very preferential so we started from that simple problem and then some of the ideas that we use can be also used to improve the performance of stack and this is what we have not done yet one of the reasons to work on raw packet IOC is that it's useful to determine the maximum performance that you can get from the system it's also poorly supported by current APIs because whether you use BDF or raw socket etc you still have to spend a lot of time processing the packet so there is a lot of return working on this problem and as I said the guidelines were the following first of all I didn't want to rely on special hardware features because I would leave it to myself on a given vendor or anyways I wouldn't be compatible with some of these if I have some cost that cannot be avoided then I want to amortize them if I have some things that can be avoided and in general reduce runtime decisions every time you need to make a decision in the code you have to take a jump possibly have a cache miss on the instruction cache or in the professional unit or in the jump predictor so these things are very expensive which should be avoided and the other thing I was a little bit concerned was in the operating system we have many device drivers so if you do something that requires more different applications to the device driver you you are going to do a lot of work to make your system work with all the hardware that is supported so modification were permitted but as long as the code could be maintained so as long as the modification were small it didn't require a lot of work to be implemented to be ported to the new hardware now the basic idea of that but also the misleading name that I chose is to use Shem memory vision that both Canon and user space and the user space application can access and this is represented in this art and this Shem memory vision contains three type of options the first one is pocket buffers basically where you put your table and pocket buffers are a big size sufficient to store full size pockets so if you have 2K or if you have jungle buffer the second region is called the network ring the network ring is basically a ring buffer and it mimics the ring buffer that is managed by the hardware the network hardware in fact both the network ring and the nick ring point to the same buffer this is the Shem memory vision so the hardware has to send pockets with fetch the data from here and when it receives the pockets will send and we copy the data here and these buffers are also accessible from here the network ring is a C4 ring it's accessed by the Canon and by user space and there is a third data structure which is called network IF and the reason why we have this third data structure is that we support multiple rings one of the features introduced by nick venues to perform us was to support multiple users so you can have multiple threads sending or receiving data from the other without the need of synchronizing and so we need to support it that's why we have this further structure which has pointers to all the rings that are supported by the hardware being a simpler ring we have a couple of information in the network ring and in particular I like the basically this information and the tail of the queue I particularly like this representation but you have a current pointer to the insert or read or write position and another field which is just the counter of available slots and then for every slot you have a very compact and hardware independent representation for every buffer in the system you have eight bytes which contain the after all buffer size if you want to send targets if you want to receive packets how many bytes are received by the hardware and something that translates to a pointer to the buffer I write here index and the reason why I write the index is that this memory region is shared between user space and the current so there are different types of spaces and they can be mapped different points in memory so if I had put a pointer here the pointer wouldn't have the same meaning in user and in current I think they are just an index it's very easy to translate the index to the proper pointer in user space and in the current without so now you have an interface if you want to control it using netmap there is this data structure that lets you manage the interface the netring which I show here is a data structure which is also in main memory but it's not accessible from the user space it's managed by the NIC and by the device header and basically points to the same buffers the network ring and the NIC ring sharing information for protection I said I want to be safe I want that mistakes in the application of the crash system for protection we use the standard OS mechanism so we make sure that the only region exposed to the user space program cannot be accessed in a way that crash the current for instance one thing that pointers to the physical addresses of these buffers and the user space program is never allowed to write pointers to physical addresses the transfer of information from here to here and vice versa is done by the operating system I modify the device line and also when this lets the kind of validating information that the user space provides the size of the region the size of the region when you attach to when you open the device the kernel defines how the user space is it's allocated only once it is allocated only once is there a way to prevent the data from changing between the time that you validated and the time that you use it from the payload node I mean there is no point in making sure that so this is an API that lets you send raw packets and makes no restriction on the content of the buffers that you send the network ring yes the network ring is like you read information here only once and so once you've read it even if the user space changes it no effect so the other thing we should be careful about is that we have memory relationship between user space and the kernel and so how do we protect from braces in access to this data structure here the contact is supported whenever the user space program is running so it's not blocked on the system call then it has full access the only area which is available to user space is accessible to user space and whenever the application is blocked on the system call which is related to that marker then the kernel assumes that nobody is going to touch this information this said before all the information here is only read once after that so even if user space changes some of the information you can touch the system the most that can happen is that you have embedded the data which is playing against the system another important part in the operating system you have interrupt service routines and those run as increments with respect to user space unfortunately they never touch this share region so there is no chance of that great system to access a device in network mode basically the system is very similar to a software you try to get a handle to manipulate the device and then you issue system calls to configure the device so the handle is written by an open on a special device name then you have an IOCTL which does a system which which device you want to access in network mode to get to 0 or 0 and if the card has multiple rings you might also specify that you want to access a single ring with that file script or all the rings in the card or you want to attach to the host stack I mean there are some multiple features that you can specify once you have issued the IOCTL basically the network card is disconnected from the host stack you have full control on the packet and you can send and receive packets using this data structure of course you have to map this region using an MAP and the size of the region is returned in the argument of the IOCTL for transmissions the first thing you do is fill the buffers with data the information, the length information in the ring current and availability tell the system how many buffers you are going to transmit and then once you you have done this you can issue an IOCTL to tell the operating system ok now take these packets queue them on the device and start operation and this is a non-blocking system receive there is a dual IOCTL which reports the newly received packets so you can ask the system if there are new packets of it from here to here basically the important part of the length of the packet and then current and real updated if you reflect the moment of course you need synchronization you don't want to spin on the IOCTL to transmit and receive and there are several mechanisms and units to do that so that MAP implements that so basically when you pull on an MAP 5 descriptor you can specify pull-in and pull-out to decide whether you want to block on reads or on the writes and whenever you have data available for reception or buffers available for transmission the poll will return poll is whenever the poll blocks it is broken up by an interrupt coming on the card so if the card supports interrupt mitigation and moderation those delays are propagated in the user space process which is very convenient because you can tune the amount of work that you do in user space basically every system call is going to cost you a lot we have seen 100 nanoseconds plus whatever power it is in the current to implement the machinery for pull-in selector so you want to in this way you want to reduce the number of system calls that you do in every second assuming you have a lot of traffic coming in and it was designed exactly to limit the amount of work that the system does by amortizing the cost of interrupts in that particular case and the system call is now a case there is no KQ support KQ is the mechanism previously which is similar to the E-Poll in Linux support for that will come later KQ and E-Poll are very important when you have many interfaces many file scripts to handle within a single multi-core support yes, we have multi-Q and multi-core support multi-Q is to again basically the net map IAC points to the multiple rings and a single file descriptor can be attached to other rings so in which case E-Poll and E-Poll wakes up when any of the rings are available or you can attach a specific ring to a file descriptor so for instance if you want to share the load of multiple cores you open multiple file descriptor attached to one of the rings in the current program the configuration is done using the ISTL now the next problem is how do I bind those processes which manage the file descriptor to a specific core there are standard system cores to do that there is P-trap set affinity in general there is something similar which I don't remember there is no need to introduce any new mechanism you just use whatever the operating system provides you can bind the process to a core and then you can have that process handled on a specific file descriptor so you can decide how you map the load on your system so in this kind of system how do you prevent packets or are you classifying packets into rings that's another of the things that netmap doesn't do netmap is only concerned with moving packets up and down any classification is done by external mechanism for instance the some nicks have RSS or other features where they do hash and subdivide traffic into the values queues and you just reuse that ATH2 are support for that so again I didn't want to reinvent the wheel there is something that works and people are familiar with just reuse them how do you talk to the host stack now the question is when you start the system the operating system believes that the card is attached it has an address possibly they are processing trying to send packets to this out to this card or expecting packets to come in from this card when you start the interface netmap mode you grab the packets that are coming from the card and you take extremely control of the transmit queues but the operating system still believes the card is there and the way netmap supports this is by creating another couple of netmap queues which are connected to the host stack so you can open another netmap which is attached to the host stack and you can move packets from the card to the host stack just using the same rings that I've shown before so from the operating system point of view the card is there it's able to send and receive packets whenever the operating system wants to send packets these packets end up in this web queue the netmap application can read packets from here and if it is the case it can decide to forward them to the card in a better direction when there are packets that are coming from the cards and you want to send them to the host stack you just reject them in this particular queue and they will go up to the stack so for instance if you want to implement a very fast pipeline or something that protects you from attacks or a stuffy schedule or anything you can use this interface and without changing anything in your applications so when you have multiple interfaces typical thing you want to do is route traffic between interfaces so do you need to copy the packets or not and the answer is no you don't need to copy packets because the payload is in these buffers and each interface access these buffers so the thing you can do is just swap the indexes of the buffers between the transmit queue and the receive queue by swapping the indexes you achieve two results the first one is that you do a zero copy forwarding and the second result is that the buffer that was attached to the other queue becomes available for receiving the new packets so basically with the buffer swapping you transmit the packets and you make a buffer available for reception without allocating anything I'm sorry in terms of coincides not that we care much but the code is really less than 2,000 lines of code and the device library modification I really commented that over the range of a few hundred lines of code consider a typical device driver is between 4 and 10,000 lines of code so the modifications are really great and there is no user library for that one because the API is really, really simple so I just have this little copy user tiny layer underline the code half of them are probably copyrighted and those are markers to access the buffers from the network now in terms of performance what I was saying before when you do packet processing most of the cost is by packet because you have a system call which is metadata not with the data itself so the relevant metric is packets per second so you find a lot of papers measuring performance in terms of gigabit per second but when you take it into the paper you find out that those tests are run with 1500 bytes of packet or jump of grams which are almost 10 kilobytes the numbers here are measured with packet source in terms of packet rates is 20 times higher than what you would have with 1500 bytes and the other thing is especially in the first measurement we only measure the packet IO if your application is doing intensive data processing of course it's going to be slow but the part we are concerned with NetMap is after the application from the application to the network or by steps that's the part that we are covering here and the other thing to notice that there are many cards that we cannot do line rate I wrote this report in 0.88 if you buy 10 giga not all of it is able to do line rate and no configuration most of them can do line rate with 5 packets but as you move to short packets they are not able to do and the reason is sometimes the buses don't have enough capacity internal buses in the car some cars use microcode and microcontroller within the car is low etc now this is the graph we saw before you see that NetMap can send minimum side packets at line rate even with highly underclocked of the CPL so I guess one of the reasons that the performance is done in piece per second is because when you do it in packets you have to assume some average packet size so the piece per second seems to be more like an absolute metric than packets is kind of like relevant depends on the packet distribution depends on the party distribution I guess that's why we use it depends on the application I think this NetMap and all these rise by pass are mostly useful when you have to implement firewall traffic monitoring etc and there you have to deal with worst case equations which are tiny grams that's the reason now there are if I have time which I don't there are some cases where you also need to show the performance because there are other bottlenecks which are dependent on the packet size it's also done for comparison reasons yeah I know well but if you take routing hardware for instance there are some RFC which specifically you should measure performance with 64, 156 and then you can pass away now in terms of a lot of performance in NetMap comes from the if you send one packet at a time you already have to pay the 100,000 seconds for the system call and you cannot do line rate now the only way to get decent performance is to send a packet in matches how much is this affected how much is performance affected by the batch size and so we measure the throughput with different batch sizes and we see that we can reach line rate with a batch of 8 packets 64 bytes 1500 bytes is not a problem even Linux or 3DSD can do that but with short packets you are not some problem so fortunately the batch size is required to achieve line rate is not huge on the receive side you see some articles the receive side basically the problem is that some packet sizes can do, you can get line rate for some others who can and the reason is that if the packet size is multiple of cache line size all the transfers are easy if the packet size is not a multiple of 64 then you need to do multiple transactions on the bus and those are going to exceed the bandwidth available on the PCI Express Bus most cards use 4 lanes for each tanking port 4 lanes have a raw capacity of 16 gigabit per second but of course if you have to you are tanking a bit per second and the metadata and you need to do two transactions instead of one for each packet you are going to exceed the available bandwidth so for instance the Intel card that we use at this point so you have to keep that in mind when you run your machine at least if you know that the card is limited by the bus size you don't waste months trying to do performance for something that is not under the control in terms of forwarding performance we do connect map and compare the performance of packet forwarding with other solutions so previously bridging or even Linux bridging all in the range of 6 to 800 900,000 packets per second if we take two interfaces run them in network mode we connect them we just put packet forwarding between them with the 0.3 that I showed before you can move packets at the right rate assuming the packet size is one of the compatible ones with one core at 1.7 GHz and of course if you do some complex processing in the middle the performance is going to be the right if you use the leaping up emulation library so there is some overhead there is an extra copy etc to 7.5 million bytes per second these are all data I think we are slightly faster and then we took some application and tried to modify them in order to use NetMap for instance, for open with switch we took the user space forwarding plane and moving it to NetMap even on top of the pickup library so it's not the most efficient solution that we can have we reached about 3 million bytes per second so I think a lot faster than the internal forwarding plane we took click which is probably something some of you have used click as kernel implementation which is quite fast the user space implementation which wasn't so fast because it was based on the pickup and just adopting it to use the pickup emulation library we reached about 4 million bytes per second and so higher speed and the internal implementation and the convenience of working in the user space the code is available for this part of FreeBSD since the beginning of 2012 there is a Linux version for most kernel from 2632.6 on my website there are even bootable images so you can take them around without having to install things it was very important to have bootable images because it's very critical in order to get that performance you need to make sure that the system is configured properly the device drivers the correct one and so on a lot of feedback I receive when people try to apply our patches is oh, I applied the patches I cannot get any performance I cannot access the device etc how comes that bootable images work the documentation was not good enough but at least this gives people confidence that the numbers are real supported hardware I said I need the device driver modification I have changes for the Intel 10GB and 1GB card some pieces for real tech and for NVIDIA and for other card I really need programming documentation without the manuals of the card open issues how can we use NPUB apart from accelerating the packet processing application click etc well, for instance you could try to use the the NPUB API in the host stack simplifying the the format of packets and speeding up some something that we should do is to modify the model of operation right now when you open the interface in network mode the card is owned by the user space applications not in a sense available to the rest of the system other than forwarding the packet through network I should extend this in a BPF like way so client opens the card in network mode the rest of the still goes to the card performance in many cases is limited by the cost of the system and some of the system especially when you interact with a scheduler to go to sleep and wake up those system calls are very expensive I'm talking about several microseconds for simple operations like scheduling a new process or going to sleep and waking up etc and those are things that are important for instance when people care about latency in the communication we have only covered the throughput which is great but the latency is not improving with the network it's just the same latency that we have with some of the system and that's mostly something that has to be done within the operating system not in the right side current work that we have done we have for instance used the internet to implement software switcher which implements with a local internet and can be used to interconnect with our machines we have a QM of a can to talk about the switcher can be used to interconnect applications for instance if you have a packet processing tool and you want to test its performance without this tool you would need a real speed or very fast network interfaces etc with this tweak you can take your packet processing application attach it to one port of this software switch and attach the packet generator to the other port and test the performance of your system this is also part of the standard network distribution so you can create dynamically many instances of switches like each switch can have dynamically created ports and on top of this for instance we have invented a test of the user space version of the IPFW and dominant packet scheduler and network emulator which does several million packets per second per corp which is 5 or 10 times faster than the internet version and so I think I'm out of time if you don't I can show you some sites on the IPFW switch otherwise I'm happy to answer questions basically we try to use the same ideas that we use in network for implement this IPFW switch and what we have so the abstraction that we create is this IPFW switch and the idea is whenever an application wants to use a IPFW switch we issue the same course that we would use to access physical interface open that in GeoCTL instead of specifying the interface name you specify a special name which is Valexy where hex is the switch name and y is the port name within the switch and the port is created the switch is created dynamically and the switch is the same switch that the switch implements the learning bridge up and the kind of performance that you can achieve is about 18 million packets per second or more frames and in terms of bandwidth because this is one of those cases where the throughput of the packets because you eat against memory bus so this switch can do about 70 gigabit per second on the bike face again it's part of the standard 3DSD and the NOS model you don't need any hardware to use this switch and we also have QMU and KBM back end in terms of performance or operation mostly this is also important so all the work is charged on the sender of a packet so when you want to send a packet the packet is processed immediately and sent to one or multiple destinations and if you implement the switch in a naive way you will take a packet and complete a set of destination and copy-rich destination and this operation will be kind of inefficient when I say inefficient operating in this way would still reach by a million packets per second or in a speech but we can do a lot better we can do almost 20 million packets per second and the way to improve the performance is to try to amortize the locking overhead and the cache misses that you would have to do with this scheme and ok so whenever you transmit a packet what happens is that you kind of receive an f-up ring with a set of packets that to the various destination so the naive operation would be taking packets, compute the destination then one at a time block the interface and make a copy of the packet unlock and move to the next one instead of doing this we first collect a number of packets into a pool of pounded maximum size then for each of these packets while we collect the addresses we also do a perfection of the payload so we amortize the cost of potential cache misses we compute a set of destinations but we don't forward immediately we just compute this array of destinations with one bit means that this packet goes to the first port and then where else this packet goes to all the ports etc when it's time to forward we take the entire batch and we scan the list of interfaces instead of scanning the packets so we first select the first interface and we lock the interface once and we forward all the packets that are marked as one here then we unlock, move to the next one and forward all the packets here and there are some optimizations so we are scanning interfaces that have no privacy directed to them we can start processing not from the first packet but from the first packet and by operating this way we, as I said we can reach almost 4 times the speed that we got before now there are a few graphs about performance the cost of this system is proportional to the number of copies that you do so here we compare the performance of our baleswitch with other solution for learning to multiple destinations sometimes are based on the use of the NIC many more than for multiple queues that can be attached to the computer and the NIC can do the copy from one queue to another one and this is basically a memory to memory copy which is done by the NIC and it's kind of stupid to do these things this way because if the NIC has to do a memory to memory copy it has to copy the data to the PCI Express bus and back into the memory so it goes through the botanical network which is 16 gigabit per second botanical much much smaller in the memory bus so the CPU is a lot faster and it shows here for instance if you have our baleswitch depending on the number of parts this is the throughput that you get if you want to do the forwarding to the PCI Express bus you get this kind of throughput it's good and for instance if you take the Linux bridging code it is done here no surprises the Linux bridging code basically pays the cost of the system core and the map allocation throughput versus packet size of course the throughput of this bridging solution depends on the packet size because you do a memory to memory copy either using the CPU which is the case for the baleswitch or for the Linux bridging the PCI Express bus and the different slope depends on the memory on the bandwidth that you have available on the bus that is used for the copy and then again a lot of performance comes from batching so if you move one packet at a time you're going to be very slow if you move multiple packets at a time you're going to have a higher throughput however the delay also increases because there is a delay roughly by the batch size so you want to have some control over the batch size at least at least I mean to be able to say I don't want the batch to be larger than 20 packets or 1,000 packets or 10 packets because I want to control my delay so there are parameters in the system that let the system administrator control the batch size and so I have some graphs that show the performance depending on the batch size whether or the batch size used on the transmitter or the batch size used on within the packet and I mean you can have a look at the paper for a complete discussion on why the slope is different in the various cases in terms of latency which is another thing that matters the Linux page has a latency you send the request and then the response depending on the packet size the latency for Linux bridge is between 7 and 8 microseconds for the batch switch is between 5 it's around 5 microseconds but most of this cost is operating system cost in both cases so batch is slightly more efficient because it doesn't do the location but the bigger chunk is definitely the resistance so we need to fix the operating system in order to do this should we wrap it up for some more questions the final slide just to get an idea of what kind of performance you can get using the batch switch with virtual machines basically you use Vataion and KVM which are the most efficient solution for doing network communication on the virtual machine you get between 3 and 400,000 packet per second if you have a faster back end which is the batch switch you can go up to over a million packet per second and of course here you are still limited by the speed at which the application within the virtual machine can send packets if you have a much faster packet generator using the batch switch you can even go higher I think so you can find a lot of information on the code papers that are on my website so you had this slide at the beginning with the breakdown of where time is spent in the traditional stacking in 3BSD and from what you said it seems that you are doing only the lower stack yeah what in the in the application I have been using basically I don't have this part the initial part of the IF at the time so this part this part is going away ok and from this what is the how much do you reduce another question is based on this architecture in mind would you do things in the higher stacks differently so you said that there is this chain and the chain also goes on how the layering works so each of the layers go and process a different one thing in a standard stack for instance what I would do is instead of adding headers one piece at a time I would write well the first time you send a packet to a destination you probably don't know which option is the option the others is etc but a lot of these operations could be cash and recyclable for instance when you send packets the typical thing that is done you make a copy in the buffer and then you do the segmentation when it's time to and the segmentation is expressive you need to touch once you know the MTU the others etc you could as well do the segmentation and similarly on the receive side you have a socket buffer you could probably allocate memory initially and avoid the escape buffer location and the location at the time you receive a packet but in terms of savings basically all this cost in the device driver is gone the cost per packet is around 20-25 nanoseconds right now I'm using batches so I don't need to program the interface on every packet I need to program the interface on every batch so this is amortized the cost up there is amortized some parts are not there the memory code is there the logic is not there anymore so most number of applications use TCP which sort of requires the Linux algorithm do you have any comparison of like end-to-end TCP for good over avail or valid do you have an experience that doesn't really affect it or is that pretty much what you were showing that it cuts out the packet delivery algorithm with most of the arguments in the OS it cuts the packet delivery and so for 1500 or 9k buffers you don't care because you're already saturating the the memory bus if you're using communicating between different machines so it may not actually it makes a saving for the types of CPU cycles which is always good but if you just measure it too quick probably you're not going to see a lot of difference so I worked on MiniNet which is a network emulator and apparently we use OpenVswitch and I actually want to congratulate you for writing one of the only good things I've read in CACN in years it's one of the few good things that's really good and I want to switch to the OBS using NetMap but NetMap's not supported in Linux so I guess two questions one is why isn't it in Linux or I guess when will it be and why not cut off that no I don't know why it's not supported in Linux I submitted it to the Linux tab at least I wasn't even considered I'm not playing by the rules I'm just sending the code and saying if you want to it, be free to use it or not it etc there is one reason why it might not be accepted NetMap is basically written for FreeBSD it uses the internal FreeBSD APIs for the convention etc it's another language so people reading Linux code are not familiar with the FreeBSD instruction I have a small layer which loves FreeBSD codes to the Linux codes and that's very very convenient for me because for instance when I wrote Vale I developed it on FreeBSD and once it was ready to build on FreeBSD I just compiled it without any changes that was great if I had two different versions I would have wasted a lot of time to this part etc so I have a stronger incentive of not having two different versions and the Linux people probably don't like the styles whatever the good part is that Vale doesn't make any change to the system so if you don't need to use the the right driver so being outside the mainstream Linux is not the main problem it's only for device drivers and for device drivers the patches are really small so they mostly apply to multiple versions so one of the things I want to do is to extend Vale and replace the learning bridge that I have decided to open this feature for Vale code so thank you very much applause