Twitter ban lifted. Should we really be grateful? If I sit back to think about it The events that led to the ban of Twitter. It leaves me wondering who lied to us about being in democracy Where is the freedom of speech that was promised to us? This reminds me of the film Legend of the Seeker where one man called Dawkin Rao controlled everyone within his jurisdiction Can our leaders be humble enough to admit their unguardedness and misguided actions? Can our leaders be humble enough to accept that the highest office in the nation is the office of the citizens? Several businesses had already been hit by the COVID. The Twitter ban came as a corresponding blow The arcs that we contribute to the socioeconomic development Yet they shut us out from one tool of effectiveness that facilitated this development The Twitter ban has been lifted after 223 days or even more. Oh, yeah We should be jumping in jubilation because our fundamental right to speech has now been restored Who knows for how long though? This leaves me asking two critical questions. One What was the essence of the ban? two Why was the ban lifted even at this awkward time? Elections are closed by. Are you seriously thinking what I could already be thinking about? Peter, if you have to say it out loud, what we are thinking, but of course I mean like really Of course that's exactly what I was thinking But to answer your first question, do you know there's something about the laws of power? When you have the power to do something to punish some people, so just show that I can do what I'm going to do You know how they just say so I feel I mean to answer that for the second one obviously Twitter is such a powerful tool and Twitter Advocates like all of you sitting here minus me So there are a lot of PR campaigns, social media campaigns, everything and they already know what else are you going to use How do you want to get it done and then how do you buy good will so now they're going to give you You don't need it you don't need Gary Abbey. You need to treat as your own take it and be happy The Twitter thing for me I look at it from different angles. Let me take it from the most Sassy positive one or negative one so that I can have all your buying Now Twitter ban The way I feel okay I'm just getting I feel now I feel that Twitter itself or social media itself needs to be regulated personally I Feel Twitter needs to be cautioned because they kind of care and caution They bring to bear in other countries. They leave it up in a consulate of African countries in terms of plant I mean, you know I'm muting this and you know what the flag and all that stuff Facebook does it better For Africa than they do so I feel that yes They need it, but we need seven months to prove this. No, do we even need to ban for a week? No, there are other things we could have done to call their attention and definitely not be not immediately after the President speech was pulled out that one is a no-no However, what pisses me off in all this is that and it do stills into what we're talking about having the strength the Resilience being open-minded the same I can bet almost everything I have on it that in a country like China India even America if A platform as big as Twitter were to be banned by now You have had alternative platform created by the indigens of those countries Two things to what wasn't curry created. Yeah curry. No To prove that listen one you cannot stifle us. We have the power. We are the Gen Z like we say We've done answers like nobody has ever done in the history of Nigeria. We are supposed to prove it with that number two It's supposed to be an Opportunity for people to know raking funds that are this space. Let me take by quickly lashing through it If you say curry or whatever was created how much impact has it made how many people? Oh What I'm saying is that Yes, government has done the undoable what should not have been done But as a people and that's what part of things I see will affect affecting us in politics Sorry, I'm bringing it back to politics because this is how we sit down Even where we have power and what things imagine the youth for some group of guys just came out and said They want to come up with something chat Sorosuki and a new a social media platform is born with the same kind of ideology as Twitter Filling up that space. Do you think government will be able to ban it? No. Yes. Yes. No Change and revolutions don't come with professor Everything has the same thread. We're just not ready and we're not strategic and we're not just thinking You know how that immediate comfort Those are that thinking and a smart have taken that that's the hierarchy of needs and they have used it as a tool It gives us and we're there We're not making it It is of the fact that you mentioned that change and revolution does not come with first attempts completely true, but would An alternative platform could say by the Nigerian that says Propose be banned as that's No It's having banned and the people that have been arrested but it's not it's not where the resilience comes in Yes, all that people come up. So the question now is how would how read how ready are we to even keep When affecting change there's there's a consequence power dynamics and if you can't win on that I mean from getting this wall the wall is the reason you can do about it So I think there's no need to keep doing the same thing the same story. Let me keep pushing Let's all say something wonder I'll make it. You might never make it So I don't even one day I will just make it you will never make it if you don't For me on this Twitter ban, I guess the same conversation is Why now obviously it's obvious Would businesses that have suffered the effect of Twitter recover is this an opportunity for them to recover Or do we need to do something better? I think we need to something better to become a better business I'm thinking as a business person right now because yes, it's just it's transcends policy politics It's has to involve with other things in social issues as well. So I Well, I guess maybe the whole Twitter conversation is really linked to business has just shown us one of the rules of life that You can't put all your eggs in one bus. No one would do that We want such a simple comment in primary school But the reality is that and as business we need to also go beyond being pedestrian in how we execute anything Because of your business. Yes, sad it got affected by Twitter ban But it's not all isn't there more I couldn't you have a country if you have to it let to Evolve and be dying diversify and be dynamic. That's a lot to it. I'm not a Nigerian before All right, it's one thing that we have Generated from these conversations is that we need to start re-engineering our thought process Don't just sit down there You need to get up and do something no matter how little we've come to the end of this Episode of the advocate American politician Patrick Moffitt said the political process does not end on election day young people Need to stay involved in the process by continuing to pay attention to the conversation and holding their leaders Accountable for the decisions they make the advocacy continues on our social media platforms and Facebook plus TV Africa Hashtag the advocate Angie and on Instagram at plus TV Africa hashtag the advocate Angie to catch up with previous broadcast Go to plus TV Africa dot-com forward slash the advocate Angie Do not forget to subscribe to our YouTube channel at plus TV Africa You next week same time on this station. Let us keep advocating for a better society


Hey everyone, this is Dan. Taiwan Semiconductor TSM jumped up by more than 10% on November 15 mainly because the disclosure of that Warren Buffett bought $4.1 billion worth of TSM shares during the third quarter of 2022. Yet today, TSM is still down by 36%. Is it the right time for us to buy TSM? I looked into the fundamentals of Taiwan Semiconductor and analyzed the technical indicators. I found a few very interesting things I'd like to share with you. Let's get into the details. First of all, let's look at how TSM has been trending since the beginning of the year compared to the broad market. You can see, TSM is still down 36% year-to-day even though it recovered by about 10% in the last few days. In comparison, SPY representing the movements of S&P 500 has been down 17% and QQQ representing the movements of NASDAQ 100 has been down 29%. The Semiconductor ETF, SMH has also been down 29%. If you zoom into the last month, you can see how TSM shot up after the Buffett 13F disclosure. The trailing PE ratio of TSM is at 15 and the PIC ratio is at 0.58. Compared to the other leading semiconductor companies, these ratios for TSM are certainly very good. Let's look at a few financial charts for TSM. Sales have been going up steadily in the last few years. EPS has been going up also steadily. More impressively, TSM has avoided the dramatic dips of the industry in the last 10 years. That means TSM management are very prudent with what they do. The net margin for TSM has been heading up since 2020. It has reached 40% recently, which is very impressive. That's because TSM is very advanced technologically, which allows them to command premium prices for their products. I will talk more about that. If you think what you've seen so far is interesting, I'd like to suggest that you click the like, subscribe and notification button that will enable you to receive notifications when I post my next video. It will also encourage me to make more videos like this in the future. Thank you very much. Let's continue. We have a lot of interesting stuff to cover. In March of last year, TSM announced that they would invest $100 billion in the next three years in response to the worldwide chip shortage. Notice that the world economy has slowed down and demands for semiconductor chips have also cooled down. As a result, TSM has reduced its capital spending for 2022 by 18% to $36 billion, which is still a lot of capital investment. This flexible approach to capital expenditures also reflects the business savvy of TSM management. I see the following arguments for investing in TSM stocks. First of all, they are the number one advanced chip manufacturer in the world with 50% market share of the overall contract foundry business. And they have a 91% market share of the production capacity for chips that are less than 7nm. As you probably know, the smaller the dimension of a semiconductor chip, the faster it is and the less power it consumes, therefore the leading chip manufacturers have been racing to produce chips with smaller and smaller dimensions. TSM is a long term supplier for customers such as Apple, NVIDIA, AMD, and Qualcomm. These are the leading semiconductor companies or semiconductor design companies. They rely on TSM to manufacture the chips. They have planned volume production of the next generation 3nm chips will start in 2022. In the meanwhile, the top competitor Samsung is also planning to start the 3nm chip production in 2022, whereas Intel will not be ready for the 3nm chips until 2024. TSM is building a $12 billion fat plant in Arizona, which will be operational in 2024. They are also building a $7 billion fat plant in Japan, which will be operational in 2024 as well. Overall, TSM is protected by their technological lead, as well as the high capital investments required in the chip fabrication business, therefore it's not easy for any other competitors to take away their businesses. They have registered steady increases in revenues and EPS in recent years, very steady performance, and of course the recent purchase of $4.1 billion worth of stock by Warren Buffett is a big vote of confidence for TSM. In the meanwhile, there are some risks associated with TSM stock. For example, there are the geopolitical risks related to the China-Taiwan relationship. Some people are worried about the potential invasion of Taiwan by China. In my opinion, the invasion will not happen for at least a couple of years, if ever. Besides, the strong resistance put up by Ukraine against the Russian invasion and the support provided to Ukraine by the US and NATO will serve as a deterrent against the potential invasion of Taiwan by China. The building of TSM plants in the US, in Japan, and soon in Europe will also diversify the geopolitical risks. The US ban on sales of TSM's advanced chips to Huawei is another potential concern. Based on the revenues and EPS increases of TSM in the most recent quarters, the Huawei ban apparently has very little effects on TSM, so I'm not too worried about that. Then of course, we always have the competition from Samsung and Intel. So far, TSM is by far the leader in the field. It will take a while for Samsung or Intel to catch up. Besides, TSM is not standing still. The company continues to invest and innovate. And then the Fed, quantitative tightening and interest rate hikes are causing a bearish market trend. This is quite a valid concern. In the next few weeks, the US stock market might be testing the bottoms reached in September and October. And in mid-December, the CPI numbers and FOMC interest rate decision will very much determine the direction of the market. I will talk more about this when I discuss my investment strategies in the next few minutes. Potential worldwide recession and slowdown of the semiconductor business is not a concern. This is due to the quantitative tightening by the various central banks. It is indeed a valid concern. I will talk more about this in a few minutes. Let's look at the valuation of TSM. I assumed the EPS growth to be 21.5% a year for the next five years based on the historical growth of 26.59% for the last five years. Therefore, the assumption of 21.5% is fairly conservative. The current PE ratio of TSM is 15, but at one point about a year ago it was as high as 32. I took a conservative assumption of 19 for PE ratio for the next two, three years. And based on the trailing 12 month net income, the current stock price, PE ratio, current market cap and the above assumptions, I extrapolated the stock prices at the end of 2022, 2023 and 2024 to be $115, $139 and $170 respectively. If I took the more conservative assumptions of EPS growth of 15% and PE of 15, I arrived at the stock prices of 90, 104 and 120 for the three years. From the above calculations, I set a target of $95 to be reached by the end of 1st quarter, 2023. Let's look at the professional analyst ratings. First of all, the current price of TSM is at $82.27 and my target price for the end of 1st quarter next year is $95. Yahoo Business rates TSM as a strong buy with 1.7 out of 5, the smaller the number the better for the buy rating and the high target is $160, average target is $97, the low target is $60. Lewis-Nevillea rates TSM overall as a D rating, that's not a good rating by Nevillea standards although Nevillea puts a lot of emphasis on the chart and as we know so far yet today, TSM has been going down only to recover a little bit in the last few days. Then the quantitative rating which is more driven by the chart is a D and the fundamental rating is a C, looks like Lewis-Nevillea doesn't like TSM much. However, tipRanks.com begs to be different, they rate the stock as a strong buy with a high target of $110, average target of $99 and low target of $89, then CNMoney rates the stock as a buy with a high target of $113, median target of $89 and low target of $60, the street.com rates the stock only as a C+, which is a HODE recommendation and they didn't mention a target price although the rating was given on November 10, 2022 which was before the Warren Buffett disclosure of purchase of the $4.1 billion worth of TSM stock. I'm pretty sure when the street.com refreshed their TSM rating, it'll be substantially better. Let's look at the charts for TSM. This is the daily chart, TSM is at the top of the bowling japan. RSI is very high at about 75, that means it's overboard in the last 2-3 days. Both the DMI and MHCD indicators are bullish, I believe TSM will most likely drop in the next few days. I'm bullish on TSM for the long term however, I've bought some shares after the Buffett 13F disclosure and will buy more shares when the RSI gets lower. Depending on how much TSM and the broad market will fall in the next few days, I might sell some of my TSM shares bought in the last few days for a quick profit, then I will try to buy the shares back in a couple of days or a couple of weeks at lower prices. Based on the technical indicators, this is definitely not the best time to buy TSM shares yet. One of my strategies, I'll wait for the RSI indicator to drop to less than 60, currently remember it's at 75, I'll wait for it to drop before buying more shares, and I'll wait for the November CPI numbers to be announced on December 13th, if the November all-item CPI decreases by more than 0.4 of a percentage point from October, the market will most likely go up, that means TSM will most likely go up as well. And if the November CPI comes in too high, the market will most likely drop, and TSM will drop as well. I will monitor the December 14th FOMC press conference. If the December rate hike after the FOMC conference is higher than half a percentage point, the market will most likely drop, and a tone of the Fed Chair Jerome Powell in the post FOMC press conference will also affect the market. If the market is bullish after the December 13th CPI and December 14th FOMC press conference, I will buy more TSM shares and other long positions. But if the market is bearish after the December 13th and December 14th data, I will sell some TSM shares, and I will sell long positions and probably buy short positions such as SQQQ and SPXS. I am bullish on TSM for the long term. At this point, I would like to suggest for you to subscribe to my Twitter account, which is DanMarketL, in addition to subscribing to my YouTube channel. By way of my Twitter account, I keep my subscribers updated on important news developments about the market and related to the companies that I invest in. I also share some of my trades in my Twitter messages. For example, on November 16, I tweeted that I sold half of the AMD shares I bought on November 10th at 15% gain. And I sold half of the ASML shares bought on November 10th at 9% gain. And then I bought SQQQ because I believe that the market was ready to drop for a few days. Thank you for watching all the way here. I would like to remind you to click the like, subscribe and notification buttons. As usual, I will very much appreciate your comments, suggestions and questions. I would like to remind you that I am not a financial advisor. I share my stock trading strategies and analyses for educational and entertainment purposes only. If you want to buy or sell stocks, you should make your own decisions and you should definitely consult with your financial advisors before you do so. This wraps up my video for now. I will chat with you again in the next few days. In the meanwhile, I would like to wish you the very best of luck with your financial investments.


So, what is summer internship? So, what do you think?  So, you have to ask questions, you get answers, same.  So, then you understand. Towards the end of the course, you come to know what is programming, what is artificial intelligence. But before that, you have a very narrow or very vague idea of what that course is. It's very singular situation. If you are done internship somewhere else, then you might understand. So, can we ask what is what? So, what does the dictionary meaning say that? We can ask so much, we can ask this also. So, if you know this, then you know many other things. So, what is it? So, difficult to define. So, this talk is actually I spent five months or six months creating this talk. So, it includes algebra, it includes formal logic, it includes system dynamics and it includes something else, which I don't remember. So, it's like you want to define something, you need some rules to define that. I mean, I think you have gone through such definitions, wherein you want to prove something, you get some already written axioms for that. But if you want to prove axioms, then you need a higher language. So, then you have new axioms for that language and then the new axiom over here becomes a theorem in that language. So, that's how it is. So, it cannot be understood unless and until. So, the only way to understand is what is it? It's not what it is, right? So, it's not something else and you know what it is. So, particular object or anything. So, it could be a concept, it could be anything. So, the fundamentals of any language, you have six wives and one husband. So, what, why, which, who and these people is to fight all the time, you know. So, the what, why would always ask what question. Likewise, why question, which question. So, all of them would fight and then they would understand something. Anyway, there's no story like that. I try creating it myself. So, what do we finally understand, you know? It's not only asking question, just using what as an interrogative, but other things are also important. So, if you get stuck somewhere, just remember this six wives and one husband. So, you ask questions. So, question has to be, you cannot ask somebody to tell me what could be a question, but you have to generate that question, right? Then only the understanding is there. This is the title and I was scared to put this title on the front. It is universal paradox, circular oscillating self-referential. So, take an example, x square plus 1 equals 0, x equals minus 1 by x. So, if you want to put a value of x as 1 and then you want to know what is the value of x, you get minus 1. So, it's like a paradox. So, it's like what you say I, imaginary. So, it's circular. The never stable, it goes from minus 1 to 1. It keeps on oscillating. You will not find this concept. You will find it in one of the books, which is mentioned in the references. And he considered a Boolean algebra is actually not an algebra, in that, you know, where he tried to give a lot of supporting reasons, saying that Boolean algebra is actually not an algebra. And you have, I mean, this is just included for us to remember. So, if I say I is imaginary, but with respect to real values, right, then phi and not of phi is also imaginary with true knowledge. So, if I have knowledge phi, I can also say not of phi, not of phi, not having that knowledge. It can complete my knowledge, because it's like A versus NP. Does it mean that you know P, then you know NP, or do you feel that if you know NP, you know it is something is P, right. So, that's how we learn. We learn so slowly, we segregate.  Why they are trying to solve that problem? So, if you know something is P, then you know something else is NP, right. Actually, a mother clearing CS, which I mean, it cannot exist, actually. Phi and not of phi cannot exist at the same time. It's just a normal way of the declaration. So, paradox is important. In fact, but you know, when something is contradicting, we don't feel it good, but if you talk about OMS, load of contradiction and through those contradictory sentences, you know, OMS have become so widely world-known that people use them all the time, right. So, you can see this. Our greatest songs are those that tell of saddest thought. If can it happen, I come back to where I have never been. Can it happen, right. So, in that context, actually our mind feels very happy when there is some contradiction, I think, something like that. Are you happy when after reading this, right. So, what we have to do, we have to come out from this happiness, you know. If you have to be happy, then you will not be able to ask questions. You have to come out from that. So, we have not evolved programmatically or whatever. So, we have a lot of paradoxes around. Some people tell us something, some people tell us something. We take it, we change our thinking, we change our definition of a particular concept or keep on changing, you know. So, always evolving. It is not the same. I think five years back, probably what you had thought of programming is not the same now.  So, but then you cannot change so much that programming does not become, I mean, should not become non-programming, right. It is hard to change paradoxes around. So, we have to understand the six W's and one elsewhere. So, how do we do this? I mean, it is not that easy. You know, you have to keep on asking questions. Sometimes we get bored because you are not used to asking questions, right. Sometimes you may find, so, you know, you may have to write something, but you are not used to writing anything. So, but then the simplest way to do it is write some sentences and then keep on asking questions. That is how it becomes bigger and bigger. So, right. So, it is nothing different than doing, you know, un-tech thesis or your PhD thesis because you have to keep on asking questions, right. You have to get answers for that. So, answering is not important. Questioning is important, actually. If you do not get an answer to any, to your question, it should not be said, you know. It should be much happier because you have a chance to find an answer to that question. You keep on asking that question. I mean, I am not saying asking question. Everybody asks to yourself, right. So, that will, there will be curiosity there. So, then you keep on searching, searching answer to that question. So, answering is not important. Otherwise, you are like a machine. Ask the question, give me an answer. Give me the answer. Give me the answer. The IQ is very high. IQ, I mean, very high IQ is also not important, right. You know that. So, the way of keeping says, I keep six or honest serving men. They taught me all I need. Their names are what and why and when and how and where and who. Okay. This is mine. So, you need six couples and manage along with their compliments. Not just the what, but not what or not why. Okay. Six couples. Not correct? Five, isn't it? Five of those. No. The other six there, five of them. No, down there or up. Down there is five. No, no, no. This slide you also mentioned five. One is lost. Six. Which is lost? Oh, you are talking about the Rudyard Kipling. Yes. Oh, I don't know why we not consider one of them. Which? So, which is lost here? Which is lost here? I asked you a question which is lost. I don't know. Which is lost? Question is the answer is the same one. Is it that a paradox? Your question is equal to the answer. So, when paradoxes happen people like it. So happy. Okay. So, this is a, I try to put it as a one day cycle, but it could be two days, three days also. So, this is the knowledge that you are trying to gain. So, everywhere there is, if you work hard you understand and if you understand like it and then you work hard again. People say that some of them they say they, I don't like mathematics. It's because they don't understand mathematics. Because they have not worked hard, right. So, if you work hard you will understand. So, it's like that. The cycle play also almost same. What if you, if you keep on working hard? Very hard, very hard. What happens to this? Your cycle is somehow this should get completed. It has to be balanced, right. Otherwise that will be too large and so you also learn something from here, from here, from here. Right? You have to enjoy. So, these are arrows are important here. You can see the arrows. They are not the quadratic thing. In fact, if you keep on enjoying more, then this cycle, this will not run.  This has to revolve, right. So, arrows are important. This is the typical system dynamics way of representing. So, how to think critically? This is from Polia. Anybody knows Polia? Polia? He was a great Russian mathematician. Great. Why great? Because he had many theorems. Can you name one? Sir, I don't know exactly. He has written a great book. You have read at least? No, I just read the name. How can you assume that he is great? Anyway, he has given something to us and this is, I think 1930s were quite very old and it is still valid. Not only him, I have seen many, many people do consider this as a chart.  But it's not that you have to follow everything. You can read it, you know. So, first thing is you have to understand the problem. There are some steps which are given there, right. But if you read it and if you relate it to what you are trying to solve, you will feel that you are very close to it. But if you miss out something, if you miss out something and just go and read this, you may find that in that. So, just the reference, okay, so that needs to be, you can put it up somewhere, read something. Like a prayer, maybe for some people, not for all. So, you can put it unknown. So, there are, in a problem you have three things. One is unknown, the data and the condition. I would put out this slide somewhere else so that you can read it up later, you know. So, the first is how to understand the problem. Second thing is how to find some sort of connection between the data and unknown. So, suppose I am trying to solve a problem and if you feel that data is not sufficient, then I will try to reduce my data and then make it, you know, or I try to change my unknown in some form wherein I try to put it somewhere, okay. So, some sort of, you know, a little bit high level of, you know, I need to change something somewhere, I need to change. So, you are trying to evolve, you know, all the time and that is why you ask these questions.  So, you better solve something else, no. And why you waste 100 years solving only P versus NP? Just solve something else, there are auxiliary problems available. So, you slowly have some transition in different different domains slowly, right.  So, you have to go back, sometimes you have to iterate in the second, in the second step also, keep on iterating, it is like that, ok. So, these are my views, maybe you can take it or you do not want to take it, you can take some steps, purpose and motivation, you need a purpose, you know, when you want to do something and you just do not do anything, keep on doing just because you feel that people are working, I also want to work. So, there must be a purpose, there are some motivations, some good motivating examples, where you get fascinated, like for example, my friend he has said, he has heard about Polia, but somewhere he has started, you know, like so, if we understand it a bit then consider him as a great mathematician, yeah. Review of the existing literature, so says with proper criticism, of course, because you cannot, you should never believe in anybody, never believe, even if the author says that it is like this, no, I will have to prove it myself, then only I will figure out what is right and what is wrong, so what, it has to fit inside first, otherwise it will be very unhappy person, it will be very difficult to identify, then only the problem is, problem is very crisp problem, not just something called as whatever, big data analytics, it is not a problem, it is a big field, it is not a problem, problem has to be very crisp and solvable actually, putting forward specific and non-missive questions, any of those hypothesis, it is any of those hypothesis, in fact, there is something on my website, so when you can go and read, I have drawn a diagram, you may like it, if you don't like it, don't even look at it, so almost the same, tackling the problem with critical analysis, giving various solutions to the problem I identified, not just one solution, at least 2, 3 if you can give and then you choose, which one case study of course required, you just don't say that this system is going to work, but if there is no case study done, then you don't know whether your system is going to work. So, then you have to demonstrate to somebody, proper documentation is also important, also future, because if you are thinking that you have built up your knowledge on somebody else's shoulders, like reading from somewhere, then it is what future possibilities, so it is your duty to write there, that this is what future possibilities are possible in this area, so it is your duty and it is more than solving a problem, then a clear cut presentation, so that even a layman can understand, so I will ask, no hi-fi terms, analytics, data, they are very difficult to understand. Data sources, journals, non-graphs, renowned books, white papers, technical circumstances, software documentation, even newspapers, you may find something interesting in newspaper and you want to solve it and then it is applicable for everybody including me to do an understanding, that will be it. So, these are my references, in fact the circularity definition is very hard of computer programming, anybody who knows can you tell me now what contribution he has done. He has wrote a great book on algorithms, it has 6 to 7, 3 to 4 volumes out of computer program and he has known as the father of algorithm design he has invented many which algorithms Sir, actually he has means he had made the bigot notations and all that famous for means, analyzing the algorithm. So, you have read that book? Sir, I no, I have not read it. No. I would not get hands on that book, actually I have. Anybody read that book? It is difficult, it was a Bible before when no books were ever done, but he defines what is called a circularity do not surprise how he does it. In the index he defines in his book he writes circularity. So, somebody has to go in the index and find out where is it defined, right. So, there he defines C circularity Sir, talk here Then down somewhere C D C circular definition in the index how beautifully he has put it right. Lower it is mentioned, but in the index it is there. C D C circular definition With that I think we should understand what is circular definition. We will need to study a proper definition to understand certain definitions, right. So, we saw internship projects, but I am not the only person there are other people also. Sir, the big thing I mean the broader perspective it is more about cloud and cloud is also a jug and you should ask me what is cloud? Big data, it is more about data which is structured as well as unstructured or any type of data that exists in the system and what you want to do with that data. There has to be some way to analyze it in some way that you can find some meaningful information from that data. So, that is the whole process. So, there are lots of you know lot of efforts going in different all over the world and people are trying to design some standards. They are trying to design some move DPE or something like that. So, they are trying to say that for this particular platform is a edX platform open edX platform partly we will modify to suit the Indian requirements and so you know efforts are there they want to standardize on say even the schema there is no schema first of all. So, you would like to involve some schema. So, there are so you may consider it or we also have involved some schema. Our schema is different than the mighty schema. So, you can you are free to consider one of them. So, because you need to know which one is the best which one is better not best. So, there are logs which have you know when some students actually navigate on this platform. Logs get generated every event log. So, what do you do with those events? Now, you have to find some way out you have to you need to know something about the student or you need to know how the course is running. There are a lot of pictures you know the teacher would like to know you know the course or for a particular student or a group of students who are they performing. I mean so of you know analysis or analytics or analysis above possible you know you can have a very basic analysis first and then you can build on build on abstract them abstract them all right. So, there is a inside platform I think of in fact a partner covered a section of this in your first site. That eventually it is related to the ID Bombay X because outside the scope right. No, it needs to be given eventually that is why I am asking. Eventually means before June or after after they leave or before they leave? No, I think they can provide some proof of concept for eventually after June. My only concern is I am adding ID Bombay X. I haven't received any request from you. ID Bombay X? No, no, no, no. I will be talking on the platform. I mean just if they are able to show me on open at X code that is fine. I have not any impact on that which is not gone by me. Right. Industry speaking. You have data in the same. Then I will be interested in what you are doing in coming inside. I will never touch your system. I will not touch your system somewhere. Somewhere else. Somewhere else. The rest of the IOS is coming in. That is certain. So there are some basic some information. For example, how many so how many males or how many females. Such type of demographic information or how many students have answered this quiz correctly or that quiz correctly whatever it is. Very basic information there. But there are some interesting questions which we can ask like how many students and how we are learning resource A. So it is also possible to ask. If you want to know how many students have actually viewed that resource it is possible to say the opposite way. So if a student do well on activity B, so they do well on activity C, is it so then you can see if this is true then you say something right and if this is true then you can say something. So we do not know what is that. You can think what is that. If they solve excise the do they also solve excise E, they may be related you know interrelated. So maybe quite very similar excises. So if they solve this they are able to solve that. So these are some questions which you can put forward. There are some 30-40 questions written down also. We will provide you that. See which courses have you learned a lot of learning resources and which of the questions please are not answered correctly at all. So they can tell the teacher in some way that this quiz was very bad so do I need to put up another quiz or maybe the particular concept is not understood properly or set of concepts or there is a dependency between this and this. So if they are not understood to this then they are also not understood this. So it is a way you can identify some relationship between you know entities and then come with an outcome. Then the teacher has to decide what to do with it. So the teacher says you have to give a new quiz. My system says this is my information and my system says you should give a new quiz but that is not correct. So the teacher has to analyse too. The system will tell you some analysis here and there. So these are some other questions. So you can think. The new quiz is Dr. Fartan's special requirement because if he finds the quiz people have failed he dilutes it and gives a new quiz so that people will pass. Yes, another problem that is called make up. Make up, he always has got make up. So everybody makes up towards that. You can also find the time required to solve the particular problem. So many questions should come in your mind. The first day of your internship, I don't know not today, if I find that particular group, that group has to write at least 200 questions. Is it okay? So this is an open space is one thing that we are trying because the whole cloud today that we have, cloud is something the bunch of VMs virtual machines or there is a platform where particular environment is there and in that environment you have bunch of VMs, virtual machines instead of physical machine I can say 10 number of virtual machines in my system. I can have 10 VMs inside this. In this system I can have 10 VMs I should be able to configure it instead of running it on this machine or instead of running on the whole machine I want to run, say for example this software on the one VM So I should be able to configure all of those requirements. I need as much memory for this process. I need as much memory for hard disk space or as much storage should be attached to this VM It is all everything virtual So the cloud open stack is one product which allows you to do that Right? So I made a mistake. I did everything in the morning He says the description is the same of the previous project that we did Oh! It happens sometimes Project is different Project is different The name is there The number is the same The number of intents is 8 That is correct I should be correct somewhere No, no, it should be 4 I think 4, 4, 4 So how the net it is correct 4 is 4 And then I have everything somewhere here Whatever is there on the net is there Whatever is there on the net On this time Just to show you There is also an open stack A lot of other peripheral elements where you can have your separate storage and you can configure it as you want So it is a very large product We have not used it to that extent So we would like present cloud that we have is actually put on VMware in the open stack So we would like to have the whole platform using open stack as a proof of concept we would have to demonstrate either EDX or any of those because we already have specification or platform deployment So those you can consider So some machines would be given to you of course As you can see Swift is one where we are interested because Open edX platform actually connects to S3 services And if you don't want you can always stop the services But what happens is that you don't want the data to exist You would like the data to exist on storage clusters or separate clusters where you can manage separately rather than putting everything at one place So that thing becomes highly scalable because you are not loading a particular machine The cloud will take care of all those So performance is one thing which is sometimes very important because you may feel that you have a product today which is running you may feel that it will support even 5 million students or 10 million students but then it may not scale to that level So what do you do? So you have to actually test out something Is it possible that do we need little better hardware or do we need more clusters So what is the solution? So the solution we need to find out You cannot say that my system will scale to 1 million 2 million just like that There must be some, there should be some test results and saying that I have simulated it in some way or the other At least an experiment is performed and I feel that it can hold to this this much level or whatever So we use the jmeter here the word is also being used You may also need to consider work load behavior These are just an extension to what is, when you talk about performance issues what do you want to really see Is there a uniform access or skew access to certain hardware and certain disks which are there to find some problem first and then you want to solve that So you need to probe into something I am just giving you some very broad overview So is there some variations over time Is it a cluster Suddenly is there an unpredictable load somewhere Is there a problem So where may be the afternoon at 12 to 2 watching movies Do you know when is the peak load So you need to find certain large burst interval load So you may also find what type of workload is it Is it just a video access Or is it just from the data actually you can figure out how much video has been used how much slides have been used or how much problems or quizzes So you will have a fair distribution once you know your data data is there so you can extract this information So there are some benchmarks already available So I put some references there And they have actually tried to use it So this is a VCH type of benchmarks which is generally used for benchmarking So OATB systems No this is a decision support system they really use where in you would like to know how much of processing power can it have So this is my hardware capable of doing that So it has been built on already existing benchmarks and So and even if the reference is there is a book also available it can read a whole lot So if you are talking about Hadoop when you say you have So you use on that architecture in the beginning that big architecture So that architecture whatever if you say the platform the platform does not include what you say Hadoop and all those activities It's not there So the platform is separate And if you want to do analytics and the basic analytics which is there NEDX does not use Hadoop at all There is some elastic search which is there and you know bunch of which I don't know but they don't use Hadoop So if you put Hadoop for this can it do real time or near real time you know analytics and use some information to the teacher like you are surprised suddenly right when you are on Gmail suddenly pops up and tells you that do you like this but how does it know it tries to find some whatever you have searched before I have a question Why real time information required in the MOOC scenario It may not be required It is not required It is not required I don't know Human instance The whole purpose of MOOC is there are so many people all over the world they have been even one week whatever they want Suppose I find that people are answering questions now right now and there is a deadline which stops in another one second or one minute and if I know that I they are answering I would like to extend the time it is possible I would like to extend another one hour for them if they are answering now I would like to do that I don't understand you have given them one week they want to do in the last seven minutes so it happens that people answer towards the end only now the time saying that one hour is there they will do something but if I find they are not able to solve can I estimate how much time they will take no can I estimate there are issues there I have listed where I am now tell me you extended isn't it unfair to people who have answered how would it not unfair I was not sure I did it because the time limit was there now we are extending it it is unfair it is very simple you can't change the rules okay so you get changed based on that let us we will use that system you change the time and then you don't consider them as marks you show them that I have changed but you change it so that you get some information about them that they have taken this much amount to solve this at least they are understanding you will have to know I don't think real time is required so real time is you will not need because unless until the stock market something happening I mean you need otherwise you don't need so even if you have a requirement which is completely different I answer a question in a physical class depending on the answer I should do my explanation there is a real time requirement I agree but not here don't extend don't extend the requirements of the earlier system to a new system it is completely new nothing to do with classroom requirement doesn't going to move two different ways of education so this is our fourth project about replicating I don't know in fact works are part of load days and I just put it up here so did anybody understand this can we have a discussion and I am ready to answer any of the questions no I know what it is this is the this is the this is the same with this blended model oh that is in relation to blended model so I am not not mad so somewhere the relationship is something else the blended model earlier add a lot of servers and the data is missing and the reason why is putting it up is when we did the implementation I told you very flatly that I will not do real time I will do in a batch because I have come from that fundamental thing that real time is not required anymore so whatever I had implemented tried to implement or to implement it completely was a batch yes that's why he wants a real time system implemented for a distributed data base which he can use in the future so there is a main system and there is other systems which are connected the main type was the concept which we are trying to implement was the whole data is put in by an IIT professor some professor who is delivering the book the same copies there in every college same copy of the course the college faculty can add to it his own lectures his own quizzes etc that's why it is a copy now college A whatever is added is college A proprietary I cannot share it with college B nor can I share to this thing so the concept was there are different sets of data available everywhere but the central faculty has to know what is happening in the local colleges the data has to be seen it will pair kind of thing it's not full network because I had implemented it here I said no so something happens on one side you have the instructor maintaining the whole the platform for that course those mini lectures whatever he puts up it has to be seen somewhere and then some discussions happening from the other side maybe you would like to see that was his requirement I said no I hold it this college is charging money for training its students you may have your own water idealistic system but the world works on money this college will not agree that water money it has spent or effort it has spent should be available to its competitors academic the business is there all colleges run on business so the system has to be designed like that but there are people like DBP who does not like business he has a strong know-how to business but he is dealing with businessmen who is selling education he lives in businessmen to get funding he wants to absorb this I have been trying to tell him that this particular model he has to charge the colleges he does not listen I cannot help him I am against all colleges he charged 2 points delivered nothing you made them too much and they do not give anything so the fixed project between Moodle and edX edX edX open edX open edX do it from id compics you should do it id compics after I have delivered the software to the system administration to run after 100 times the production system maintenance is outside my group you can add features to it independently I have no problem do not come to me to add I will give you I will give you I will give you translation these are my two four translation management is included both are related of course that is what she said two activities we are working on translatability of all course content automatic translation automatic translation if you want I will give you automated here I have done it in a project I have no problem that is Google and for this particular purpose we are doing it officially because Google for educational purposes Google gives an API free for my purposes I achieved I thank them they have translated Google and I thank them officially for making their free source available actually in my earlier time I changed I said I have thank you officially for making it available without adding a capture if you want to protect it add a capture everybody knows that if you don't add a capture I will change and Google can't claim they don't know capture technology it's obvious they want me to change to don't leave the door open don't leave the door open you know there is no capture where is that R? somebody found an R somewhere no R is going to come later on R is being called but that is only one project and she will not get too many people okay so R is just a statistical package no I hate you are not doing the R now you are doing QQs stop the R person because of public demand is going to come and talk about R so who is that R? it was an R oh now I got it you don't have to talk I am faster to come here fine so I finished my talk here I feel one you can go through my slides later what it takes will be provided so I will be putting up it here last year we used this for all project description you are the key no no that is brainstorm something you are done that was better we did not have time we finished the project descriptions only very late I told him we were busy with lot of other things okay so that is the link so in fact it has been under construction last year at least I had 400 to find a reference and this time I have less I am coming with your heart


In the meantime, I'll just go ahead and introduce myself, and you all can do the same. Maybe you want to go first, since your name is? Yeah. Hello, everybody. My name is Yohaku Sonen. I work in Nokia Mobile Networks as a software architect. Yeah, hi. I'm Ajay Simha. I'm an NFV architect at Red Hat. I work in a group that produces all the reference architectures, and I focus on NFV reference architectures. All right, today we are going to share with you our experience in DPTK troubleshooting. Although it says troubleshooting, I know some of you at least may not have the kind of architectural background of how DPTK works, so we'll start off with that. And then go on to these are things that we expect for you guys to walk away with. One is to get an overview of DPTK, and what can go wrong in DPTK installation, and how to troubleshoot that. And lastly, the tuning of DPTK, which is very critical for you to get high performance, and what are the things that you can and should do to get high performance, and how to look at stats. Because DPTK, once your interfaces are in DPTK, it's no longer in kernel, so you cannot do IP link show, your links won't show up, and so on. So we're going to show you important and cool commands that you can execute on Thread Hat Linux. So this is the agenda. We're going to walk you through how we got here. What are the telco requirements? Because DPTK is a data plane development kit, is a method for you to get high throughput, which is like most of these requirements came from network function virtualization, and that is kind of the main driver or motivator for developing things like SRIO, VEE, and DPTK. So we're going to talk about the telco requirements at a high level, then talk about journey to DPTK, how we got there all the way from VertIO, talk about the DPTK in the data plane, which is going to be like hardware tuning, what needs to be done there, how do you get the throughput, what kind of throughput can you expect using DPTK, and CPU core allocation, high availability. We're going to talk about high availability just at a high level, but we're going to talk about that because it's really important. And lastly, even though the title says DPTK troubleshooting, the last section focuses on troubleshooting and various aspects of it, which what can go wrong in install time, what you should do and what can go wrong, what does it look like in an HA environment, and lastly, the performance-related stuff. So getting to the telco requirements, there are basically two major pillars. Although telcos, service providers, operators, whatever you may call them, they'll always come back with, we have, I'm sure, a lot of us in that room. I consider myself telco because I've been working in service providers since 1998. So a lot of service providers will say, these are the two major things, which is, first is two pillars, which is high availability, and the other one is performance. But then again, they may come back and say, I also need, by the way, service assurance, I need that, I need billing, other things also. Everything is equally important, but these two are the most important. So we have heard of situations where telcos are talking about, oh, we do not want to set aside node to do control function only because we want to get maximum throughput, meaning the number of CPU cores that we allocate, we want subscriber, equate that to the number of subscribers who are paying for it. So that's how detailed they get trying to get ROI on this. So first of all, maximum subscribers per core, how do you get that? How do you get high availability as close to 5.9s as possible? Because here, remember, we have tiered architecture. It's like you have your data center at the bottom, the hardware, and then on the top of that, you have the network function virtualization infrastructure, which is typically open stack in our case. And then on the top of that, you have the VNF. So the high availability is actually tiered. So you cannot get high availability unless you have some sort of cohesive method from the top to the bottom of the stack. Again, high availability, even though we're talking about troubleshooting and performance, without HA, if you have no high availability, if your node goes down, failure means zero packets per second. So let's remember that. So that's why it's always important to think about high availability. Delco applications and services typically serve millions of subscribers. I mean, we may not realize this, but when we talk about a mobile operator, even the small ones are talking about tier 2, tier 3, are talking about 8 million subscribers and 10 million subscribers, and it can be higher. So what used to happen is the hardware-based solution used to provide them these things because they used to have multiple refrigerator-sized boxes having dedicated ASICs and hardware doing exactly what needs to be done to get the acceleration and the high availability. So now we need to try and get that in a virtualized environment. How do you get that in a virtualized environment as much performance as we can get? And we need to get that, the maximum packets per second without having drops, ideally. With drops, I'll show you a chart where we'll share some testing done by our larger team, which does performance testing. I will share that information with you. And you can see that when you have drops, like even though it's minuscule, you can get much higher throughput. And without drops, zero packet loss, the PPS is going to reduce, obviously. Network equipment providers, people who provide the VNFs, as well as the telcos, have tried to move away from solutions which depend where the VNFs will depend on the hardware, has any dependency. The reason I bring this up here is SRIOV works great. You get fantastic throughput. The only problem with that is really that the VNFs and the drivers for the NIC are tightly coupled. So if the VNF vendor moves forward, or more typically, the operator or service provider goes and changes the hardware, then there has to be a dialogue between these guys saying, like, hey, I changed this. Thing is not working anymore. And not working and those kind of things are not something that is tolerated in a telco environment. Typically, even if you have outages, they have to be on a certain seconds or minutes per year, you have to report it to FCC, at least in North America. So it's a pretty serious affair to have all these things working all the time. So ideally, many of the operators, service providers that I've talked to, want to get as close to the cloud as possible. So they're willing to go for the flexibility while sacrificing a little bit of the performance. Again, this is like a kind of a gotcha, because they may say we are willing to sacrifice some performance, but then you'll realize they want more performance later on. So it's always a game catching up game where technology is trying to catch up and give you more and more packets per second. Yeah, and then let's take a quick overview about the playground where VNFs are living. So this is a picture of Nokia Airframe solution. On the right-hand side, there's Vim layer, which in the Nokia case, also VMware is supported there. And then there's a hypervisory layer, and all of this is operating on top of the data center hardware. So and on top of that is the actual VNFs. And those VNFs may have then certain requirements, all of which are not exactly the same on all the VNFs, but probably in many cases, there are in common that the high throughput in terms of network, as well as latency is required. For example, this is true in case of radio and core cases. So there we see these terms like real-time capability and throughout needs to be optimized. And the DPDK is one approach to meet these challenges. Thank you. So now let's talk about the journey from VertiO all the way to DPDK, how we got here. So when people started thinking about NFV initially, it was all VertiO. And performance was considerably low. You could easily get maybe 600 to 800 Mbps out of a 10 gig link. It won't matter. It's just not acceptable from an NFV telco point of view. So that's what you have two nicks, 10 gig nicks in this whole scenario that I'm going to paint for you. And out of that, first of all, you have all these hypervisor and layering. There was actually one older OpenStack documentation that showed that there were seven-plus layers before, like a packet all the way from the VM. If you think about egress all the way until it makes it out of the physical NIC or the PNIC.  So things started improving. Even OpenStack made modifications. It was not layered so many places, but still the performance of VertiO in its native form is not good enough. So then the next stage was people who in NEP started saying, OK, we need to get as much throughput out of a 10 gig as possible, because you're setting aside a one-hole server to do certain functions. You need to get throughput out of every single 10 gig as possible, so that you can get the 40 gig, 60 gig, 100 gig throughput out of a whole box. So this is PCI pass-through. The PCI pass-through, the issue is if you look at, you have two VNFs, VNF1 and VNF2. VNF1 has got Ethernet 0, and VNF2 has Ethernet 0. But the 10 gig is going to be completely dedicated to the VNF1, because that's how PCI pass-through works. Basically, the hypervisor takes the characteristics of the 10 gig, presents it to the NICs, and one of the NICs, I mean, the Ethernets on the VNF or the virtual machine. And VNF1 uses that, and that's the end of it. So VNF2, Ethernet 0 does not have a place to connect to. That is a problem with PCI pass-through. Again, you could have multiple NICs for all these things, but again, think in terms of monetization. You want your cores and your NICs to be used maximum so that you're getting paying subscribers for those things. Then came SRIOV, which elegantly solved that problem, where so what it did is, again, there's another issue with this previous scenario is that the VNF1 may not really need all the 10 gig. It may want, say, 1.6 gig or something. There's no way to share that 10 gig. It's just like the remaining bandwidth for the 10 gig is just wasted. That is the problem. So SRIOV solved that elegantly by taking the physical NIC and creating physical function and virtual functions out of that. You typically have one physical function which has got all the characteristics of all the registers, all the characteristics of the hardware, whereas the virtual functions, the VFs, which you see, VF1 through N, those have typically only the receive and the transmit queues. And this is all kind of built. You can think of it like a switch within the NIC card itself. This is not really a part of the OS. That's why it's in its own box there. So with this, you can still have VNF1, Ethernet0 connecting to virtual function 1. And VNF2, Ethernet0 connecting to virtual function 2, and both of them sharing this 10 gig bandwidth and getting better utilization out of this. One issue with this I already alluded to this earlier is that the VNFs, VNF1 and VNF2 have to have a driver support for the exact family of NICs. If it's a X540, whatever it is, it needs that. So if the private cloud changes, they got a new generation of NICs, VNFs have to be reprocessed so that they will have that support, which is a problem, which is what we want to go away from. This is what DPDK typically looks like. You have a couple of fundamental concepts that you need to understand in DPDK is that DPDK uses something known as pole mode driver or PMD. So what it does is there's a polling mechanism. And another equally important concept is that OVS DPDK actually is going to reside in user space. Sitting in the user space, it directly is able to, because of other optimization, such as V host user and stuff like that, it's directly able to communicate from the VM and bypass the kernel. That is where you get the optimization and the speed. And the pole mode drivers. The pole mode drivers, the way it works is you will need to dedicate one core to do the, in my case, in my lab, I have 48 CPUs, 0 through 47. So I will need to dedicate one of them saying you shall only be serving this polling function. So it's very important to understand how this thing works, because it's all about in DPDK, the flexibility comes from complexity. And the complexity is, if you tune it and you go through the extra effort of doing that, then you get a pretty good solution that you can use where it's more cloud ready than SRIV. So in the DPDK data plane, these are the things to consider. One is the hardware tuning, what kind of throughput that you can get, what you should expect, the CPU core allocation and the high availability. We'll go through those things now. So from a hardware tuning point of view, first of all, we talked about zero packet loss. That is fairly important that we have this high throughput with no packet loss. For that, you will need some later hardware generations. You cannot have a older one. It's not supported. There are a lot of tweaks that are done. If you use Red Hat Director for installing the OpenStack, a lot of it is done for you. And more and more things, as we go from 10 to 11, 11 to 12 and so on, more and more of these things will be done for you. It will be completely transparent. But you will need to have a hardware, like a CPU, which supports those things, the generation of CPU. For example, if you're using the NFE lab that I work in uses X540 and some of the other labs have ordered X710, these kind of CPUs are required. Secondly, it's also important that you go into BIOS and kind of disable things like the C3 power state and take all these extra steps that are required. The reason they are required is if you don't do that, it will cause a lot of these local interrupts. If you look at the interrupts, you can see I think it's like slash proc slash interrupts. If you look at that, it'll show you all the CPUs and all the interrupts that are associated with that CPU. And the local interrupts are pretty bad. I will show you some charts, what it looks like when you have a tuned situation, tuned profile, versus a non-tuned profile. I'll show you what it looks like. So this is something that you'll find even on the OVS web page. These are things that they recommend saying that you should do these things to get a high performance. So now talking about, I know you have a nice picture on the right side, but I'd like you to focus on the left side, which is talking about some important things. For the test that was done here to produce these results, two Vortio net interfaces were used in the VM. And then two 10 gig interfaces were used. And then test PMD was the test tool used. Although we have kind of developed a new tool now, our group has developed a new tool called Pbench, which is also available for public consumption. I will show you that and share the link with you. Bidirectional traffic, as well as the various profiles. You can see on the left column, you have the packet sizes. If you talk to any service provider, if it's RFI, RFQ, always people will bring up the discussion about 64, the smallest packet size. But in a true iMix, whether it's a typical mobile network, everybody uses mobile now, like Netflix, whatever. In a true internet mix, or whatever you want to call it these days, it's a mix of all these packet sizes, 64, 256, 1024, and 1,500. In fact, there's a very cool command that I came across, which I've taken the output of, which will show you for these packet sizes, what were the counters. So you can actually look at that in DPDK. Again, if you look at column two, you will see with 0.002% loss, I'll just take the 64-byte packet, since it's very popular and discussed all the time by mobile operators and telcos. So with that, you could get almost 12 million packets per second if you tolerated that kind of loss. But remember, not all traffic is TCP. Not everything is going to retransmit. There's a lot of applications are built on UDP. It's just a lot of overhead. You don't want to throw a package if you can avoid it. And then you compare that with 0% packet loss. Again, remember, this is for two cores, not for one core. And for two cores, if you now look at column number five, which shows 7.34, that is realistically what you can get approximately about 3.7 million packets per second with per core. That's what you can realistically expect. So which is a huge improvement over VertIO and getting closer to the cloud ready model. With SRIOV, you can get close to line ready. You can get 9.6, 9.3 gigs per second on a 10 gig interface. Again, we don't like to talk in terms of GVPS because it really package size, frame size comes into play. Because as you can see, just smaller the frame size, the lower the GVPS.   And large packet sizes, you can get more GVPS, less packets per second. All right. So this is a diagram which kind of shows. On the bottom, you have the physical cores 0 through 3. And core 0 is being allocated. We do the CPU isolation and say, these are the ones which the host can use. Host can use for multiple purposes. One is for nothing to do with housekeeping, some other functions that it's doing. Nothing to do with NFV or DP DK for things like that. You can set aside CPUs for that. You need to set aside CPU for the pole mode driver which runs on behalf of the host to use. And then the physical CPUs 1 through 3, 1, 2, and 3 are given to the VM to use. I'll show you the configuration parameters in the director which you would set to achieve this. I guess no questions so far should be pretty clear and easy to understand, I guess. So talking about, yeah, we should wait till the end. OK, let's wait till the end. All right, so DP DK HA, the way we have kind of designed this for the lab and the upcoming reference architecture document is that what I don't show here is like NIC 3 and NIC 4 are also bonded. That's my network isolation for all other traffic like internal API storage management, all of that. And if you're using VXLAN, the tenant traffic which goes from the controller to the compute nodes for the VXLAN traffic, all of that rides on the bond which is on NIC 3 and NIC 4. I don't show that here because it's not out of scope for this. But just I wanted to mention that NIC 5 and NIC 6 are dedicated for data plane in the NFE lab that I work in. What we do here is we bond those two using OVS DP DK bond, the DP DK bond. And in this example, we show that VM 1 is also running DP DK and it's got a pole mode driver and it poles. That's the other thing that you have to consider is that it has to, if the DP DK and the user space outside the VM delivers a packet, unless the pole mode driver is active and goes and fetches that packet, you could have very poor throughput. So both things have to be in sync for this to work well. So in this example, if you have the way we do it as ethernet 0 connects to the DP DK bond, NIC 5 connects to switch 3, NIC 6 connects to switch 4. So either if switch 3 or switch 4 fails, you still have at least half the throughput. That is the whole criteria of creating these bonds. Finally, we'll start looking at the troubleshooting part. One is there are three aspects to this. How to install? What are the things to set for installation? And what can go wrong during install? How do you can look at that? Again, from what we said, HA is important and we want to use DP DK bonds. And we'll show you a command how to look at that. Lastly, we'll look at performance related stuff, how to get it on the node with show commands, how to look at counters, and what are the important things to look for. Hey, how are you, man? Do you need a Polemode driver DP DK to enable SRIOV as well? That you sort of made them sound mutually exclusive? OK, that is thanks, Arnais. So you're talking about a scenario. There's only one scenario where you would use a Polemode driver in the VM and SRIOV on the host. Is that the one you're talking about? Possibly. Yeah, that's the only scenario. Otherwise, you don't really need a Polemode driver for SRIOV. So there is a scenario, but I don't know of any NEPs or telcos wanting to use that. Yeah, they are. They are. OK. We should talk later. I'll talk at the end. OK, yeah, please pull me in. I'll be happy to discuss that. I've mentioned that in my. Do you think it will exist? Yeah, yeah, yeah. I know it will work, but whether you. That's again, like going back to the. That doesn't take away the problem of using SRIOV, right? You still need to use SRIOV there. It'll just increase the performance, I guess, from a VM point of view. Yes, do pull me in. I would like to learn what you guys have done, and we'll take it offline. Thank you. So from an installation point of view, again, in the lab, we have used recently Red Hat OpenStack director with the Red Hat OpenStack platform 10. And these are the parameters that are set in network environment.yaml file, which are kind of related to DPDK. Some of them are used for other things, as well as you'll find similar configurations for SRIOV and other things. But these are the things that I've kind of listed here, saying that you have to kind of configure and touch it for it to work. And the ones that I've highlighted here, if you look at the host CPU list, host CPU list can be specified as like an enumeration like you've done here, or you can give it a range, saying, for example, you can say, I could have just said 1 through 47, and just go ahead and use all of them. Or I can select these, which means that other CPUs, like one or something else, is not really set aside for this purpose. And out of the host CPU list, again, some of these, you might find some changes as these things are progressing and evolving, how these things are being interpreted. And already, I'm hearing through my developers at Red Hat that these things are going to change. How are we going to set this and interpret that to establish the tuning may change in the upcoming releases. But for right now, what it means is you have an entire list specified under host CPU list. So you say, this is the entire list. Out of that, you say, Neutron DPDK core list can use 4, 6, 20, and 22 out of that list for PMD, pole mode driver. You can use that. And then you set aside for the VCPU pin is the one which is these are the CPUs which are set aside for the VMs to use, the 8, 10, 12, and so on. So together, if you think of it, the host CPU list is a superset of the Neutron DPDK core list and the Nova VCPU pin set. And that can be some of this information is available. In fact, examples, a complete sample file of network environment.yaml. And each of these configuration parameters I explained on the customer portal, Red Hat portal. The link is provided here. If somebody wants a PDF copy, I can try to take names and send you a PDF of this presentation later. So during installation, if your install went well and this is what the DPDK setup, OVS DPDK should look like, in our case, we have a bond. So you will see the port mentioned as DPDK bond 0 and it's got two interfaces, DPDK 1 and DPDK 0. This is what a healthy installation should look like. There's some interesting things which you make mistakes in the ML file and all that. It could lead to some interesting situations, like having extra space between commas. I thought that it'll be ignored, but it's fed into the command line which fails silently and you'll never know about it. So those kind of things can happen to you. But if it's healthy, everything went well, this is what it looks like. The type DPDK is important because the non-DPDK 1, by the way, this is referred to as OVS user bridge versus OVS bridge. OVS bridge is the one which you would normally use outside the DPDK environment. If you want to check what OVS DPDK options are being set, you can go cat this file. After the install is done, you can go cat this file, etsy, sysconfig, open vSwitch, and you will see all the DPDK options there. There's also like this file, whether it gets used or not, also depends on which OVS version that you're using. 2.5 uses this and restarts OVS, and 2.6 reads out of some sort of a database is what I've stored. So anyway, I mean, you can go cat this file and you'll find that information there. This is kind of what it looks like when DPDK was not set up and you had some sort of error, like this one scenario, there's another scenario that we ran into, but I've not captured that. So this is like, what happened was there's kind of a race condition and because of which the DPDK, like I said, I fat fingered and put a space between two parameters which are fed to OVS on a command line. So obviously, in a command line, you can't have spaces. It barfed and silently did not install DPDK, but that's what the output looked like. You want to talk about this? Yeah, this DPDK dev bind is a pretty handy tool to bind and unbind devices and also checks the status, which means that if something goes wrong even in a very beginning when you are taking the DPDK into use, it's worthwhile checking that whether you have the interface even created or not. Thank you. We talked about how we have used the bonding for DPDK and the way to look at the bonds is you do a command OVS app CTL bond slash show and the actual bond name in this case is DPDK bond zero. From that, you can see all the details. The most important being that the DPDK one is the active slave. And so when we did the failover test, we actually went and failed the active, the switch side of the active link and tried to see whether it fails over to the standby and stuff like that. The other thing to notice is the bond mode right now is balanced TCP. LACP is going to be supported shortly, I think, in OpenStack 11. I forget the details, but I can easily find that out for you if someone emails me. Later on, normal LACP will be supported. But right now, it's not supported with this one. So you have to use, if you want LACP, you have to use balanced TCP as the mode. Sorry, when I say LACP is supported, it will be 802.3AD, which will be supported. Then comes the final part of this presentation, which is the troubleshooting, things to look for in performance. And there's two aspects of it, things which you can look at on the node and measurements that would be interesting, both. It's not just from a troubleshooting perspective. Of course, making sure your RX and TX counters are going up, drops are not going up, things like that are obviously useful. But also, if there's some way you can use it to maybe craft something or use it for stats, that's also important. Here's some of the commands that I got to use in the lab. And so you can grip tune D in the etsy tune D boot command line. It will show you the host CPU list, which was used. So this is the one that, because of tuning, this is what is provided so that at boot time, these CPUs are set aside for the host. Second one is, in the OpenStack director installation, it creates this file called CPU partitioning variables.conf. And that will also have what CPUs are being set aside for the host. And lastly, you can grip a vCPU on the etsy nova.conf, and it will show you the vCPU pen list. These are the things that are useful to look at. So this example is taken from an output of a lab, which is not there's a dynamic lab where we use it for performance. This is not from my lab, but this is very interesting data. So I thought I'd share this with you. So if you take a look at this is graphing the local interrupts.  And you'll see it's a huge, very long file. It's like one line runs into one single roll, runs into many lines. And it's kind of hard to view it visually. But what they've done is they built a tool, that P bench tool that I talked about. And that tool is available on GitHub. And I've provided the link at the bottom. So you can go grab that and this readme files and stuff like that. They use that for a lot of testing as well as tuning stuff. So here, if you notice, I've highlighted CPU 17 in that like radio box or whatever you want to call it. And that's showing that the average local interrupts are approximately 1,000 per second, which is very high. That means it's doing something. You should only get two interrupts. One for the pole mode driver, one for the VM if you have tuned your CPU. If you don't tune your CPU, this is what it'll look like. So you may have run some installation, thinking everything is great. And then you use your traffic generator and you're not getting the performance. These are the kind of things that you need to look at. And here's the example of CPU which has been tuned. Now I've selected CPU 1 and CPU 1 from that radio box. And you can see that the average number of local timer interrupts is close to 2. Again, this is not the only thing. You can also look at var log. I think I showed that var log tuned.log. And you will see that it has actually attempted to do the partitioning and things like that. You can look at that.  First of all, this one is the one which displays the packet counters and the drops. I've highlighted receive, transmit, and drops. These are things which you can quickly look at. You make sure you receive RX and TX as constantly incrementing and drops are not. Sometimes when there are transitions, you can expect drops, even in hardware-based solutions. You will see certain drops. There are transient drops. We tend to ignore them. The more important thing to pay attention is drops should not continuously go up. And there should not be large numbers. That's very important. Again, I mean, this is a sake of reference. I've provided this. But this is not something that, at least, I have not used this a lot. But it gives you kind of correlation between the port numbers and the port names so you can make the connection. This is a very interesting command. I alluded to this earlier. This gives you, based on packet sizes, like packet ranges 128 to 255. That's the RX queue. And 1 through 64 small packet sizes is in that received queue and so on. It gives you counters per packet size, which is very cool because you can kind of extrapolate and figure out like the traffic mix also. Pretty cool command. And lastly, this gives you like displays the PMD port allocation to show you that NUMA node 1 is where the cores 14 and 15 are being used. I think that's all I had. If you have any questions, we can take them now. So TCP dump is not available with DPDK because it works at kernel mode. But there is a utility that replaces TCP dump. And that's all I know. Can you explain a bit about how you do TCP dump like troubleshooting in a DPDK environment? OK. First of all, for sake of transparency, I ran into the exact same problem. We ran into the exact same problem. We wanted to see packets. I love to see packets from the source all the way to the destination. Every hop of the way, I like to see that. And I quickly found out that TCP dump doesn't work. In fact, if you do an IP link show, it will not even the links don't even show up in that output. So there is actually an OBS TCP dump I'm being told by Flavio, I think, and Red Hat, who's like kind of a guru in this area. He said that you can use that. And that will show you all the output. But if you do a normal deploy using Red Hat OpenStack Director, you don't get that. What I understand is you may have to kind of build your own DPDK. And there's like a tools directory in that. From that, you can start using that. I think that OBS TCP dump will be available. I've not had time in the last month to dig into this. But if you leave me your email address, when I find out, I can share it with you. OK. Thank you. To add to that, you might consider also trying a Portmare within, again, within that. The key there is that anything that DPDK pole mode driver software is utilizing in an interface that's been assigned to it, no longer uses a normal Linux kernel mode driver. So TCP dump, of course, deals with kernel interfaces. Excellent slides. I recommend everybody in the room do get these guys' email addresses and ask for these slides. This is golden information. Can you go back to the install slide with all the vCPU variables? There are some bits here I want to point out to everybody that is absolutely critical in understanding high-performance VNF compute nodes, this. So this is a hyperthreaded system. You can tell by the numbering. The other thing, there's another session that was earlier today. Find the slides and recording. NUMA node alignment, CPU pinning, 1GIG huge pages. All of these things work alongside DPDK to enable not only DPDK if you're deploying it at your host level, which is what we've been covering in this session. But for any other use case for VNF as well, having 1GIG huge pages, nice, neat blocks of RAM, having NUMA node alignment is critical. And this is on a NUMA node system. The short version is if you have a multi-socket system, you have two motherboards that are connected via a QPI interconnect, like a bus. So every time you map a port, you want to map your CPUs and your RAM to the same side of the motherboard, the same NUMA node that whatever your transport mechanism is. So excellent stuff here. This slide alone is worth everybody reaching out to you for the slides. The other thing on performance that is always alarming the first time people run into it is the apparent CPU utilization of the pull mode driver.  And all of a sudden, my utilization is 100% on these cores. No, really, it's a pull mode driver, which is doing what it says. It's pulling constantly for packets to be sent or received out of port. So that's a false positive, maybe way to describe it, until you knock people to settle down. But yeah, if you're still running classic OVS, you are failing your company, in my opinion. Thank you. Thank you for the comments. Just to follow up to what you said, does it make sense to actually track or measure the processing cycles for DPDK and the polling cycles for DPDK separately? So it gives the sense that, OK, how much of the CPU cycles are being used for processing packets? Great question. No, do not do that. Those CPUs that you're allocating to anything running DPDK, whether it's the many, many vendor-provided VNFs that employ DPDK within their own data planes, or whether it's DPDK and OpenVswitch, those cores are dead to you. They are gone. They are allocated for forwarding. You will not be using them for any other purpose. You should not want to use them for any other purpose. They are giving you much greater functionality in your environments. So don't even bother tracking. Those cores are gone. I understand. What I meant was that is there any way to find out that the CPU core allocation or CPU cores allocated to a VNF are pegging to the threshold? And that is the reason why the performance is limited, or that's the bottleneck for your performance. How do you do that? OK, so let's say we were comparing classic OVS to OVS DPDK. That's a good context for your question. Total number of packets per second throughput is what you should look for, because if you're running OVS, on a good day, you're getting 1.1 million packets per second, and you're done for the host. The rest of your host is sitting idle. Core is doing nothing. So packets per second is the threshold that you're looking for. As soon as you go to here, his chart's already covered all of that. I mean, even lightly tuned, DPDK-enabled OVS is going to decimate that. Regarding coexistence, shortest possible version of this, using open stacks scheduler filters, groups of compute nodes that have ports allocated for SROV, and groups of compute nodes that have, well, anything running OVS should have DPDK-enabled OVS. But those hosts coexist in the scheduler filters, do a very nice job of picking where your workloads go.  They benefit more if you can afford to give them Newman node CPU pin and one gig cube pages to those SROV-enabled interfaces. So the rest of your commodity oversubscribed compute load should be on top of these nodes without any question whatsoever. So coexistence works great. You guys choose what your business needs are to determine where those workloads go. I'm being signaled that we are run out of time, so can we take your question offline? Or maybe last question. Make it quick. Maybe it's the last question. Thank you so much for attending. There's also a reference architecture that we put out. There are two of them, one called deploying mobile networks using NFV, something like that. And the second one is going to be using OpenStack 10. So look forward to that on access.redact.com under reference architectures.


A chevlin woman is dead after an ATV accident near Bemidji. The Beltrami County Sheriff's Department says 42 year old Rosalie R. Miles died yesterday morning at the scene of the crash nine miles northwest of Bemidji. The investigation of the crash revealed Miles was traveling north on Nature Road when the ATV left the roadway and entered the ditch and then struck a barbed wire fence and then overturned in the field adjacent to the roadway. The crash was reported just after eight in the morning at the intersection of Nature Road northwest and Mayflower Road northwest. Miles was not wearing a helmet.