So I don't have comments or discussion on it, what I do have is questions there. Okay. So basically, let me see the big picture here is we have three happy outcomes, let's say. We have growth. We have an improvement in labor market indicators and we have an important reduction in poverty. But then I would like to know what's going on with inequality because we see inequality has declined, but this decline is not as important as the poverty rate decline, right? So basically what I would like to know is what is causing the poverty rate to fall but not inequality? So I was going to this UNDP report that says that 10 out of the 15 most unequal countries in the world are still in Latin America. So basically that was my question for you. And I know that you are also looking at labor as the channel for this improvement in a reduction in poverty, but I would also like to have your input on non-labor income and the importance of social spending because most of Latin American countries we have a lot of cash transfer programs. So yeah, basically that. Thank you, Sam. With the purposes of discussion, do we need to go over this or can we sit here? So let's do this then. Do you need the audience if you have to be mic'd? Can we have the mic? Yes, okay. How about if we do this then? Name a comment. People ask questions using one of the mics. And I'll write the questions down, then we'll get a mic and we'll all turn around so that we're not sitting over like this. And head of side of the stage. Yeah, let's do it. Let's make you make it three at the time. All right, let's start with the three in front. One, two, three. I like that. Anyway, great work. Congratulations to the whole team. It's gonna be very useful to have this information. And I have some questions and some suggestions. The, and probably Guillermo knows about some of the curiosity that we still have from previous. No se escucha? Y que hago? Bueno, me paro. Trata tratando de ser modesta. Ahora? Yes? Okay. So the question I wanted to ask is, what didn't you look at education? The story that in the book that we, Philippe and I edited several years ago, sort of found, and that was in 2010. So there's been much more data since that one of the key factors that seemed to drive a lot of the results in labor market inequality had to do with the expansion of education. And especially access to tertiary education. And then I became skeptical and I started to look at the demand side. We're still working on this with many people in different parts of the world. And yes, you have demand in some countries, especially those that experience the commodity boom, but not in others. I still think that the common denominator that you will find in the story is what happened with the expansion of education because it was a push in the 1990s. You have all that information. So you could actually integrate it to your story and have that as part of the question. So that's, then, did you do any checking? Because you kept telling me about this is household surveys. I mean, that's okay. But some countries have a peculiar, no, but even the average, the Dominican Republic didn't actually experience growth if you look at GDP per capita. And with it, so it would be good also to identify contradictory stories because in your analysis that you use GDP per capita, not the per capita household income from, you have both, you have both. But it'd be interesting to isolate those countries in which you have different macro and micro stories to see what's going on because then otherwise, you'll have, and the Dominican Republic is outstanding because anything, GDP, I think was growing at between six and 8% per capita and here you have negative growth. Sorry. Shout. Who should I give it to now because I don't want to. Okay, okay, one, two, three. Who should I give it to? Who should I give it to? Thanks. I'm Per Honos from the Swedish Development Agency. Well, thanks first of all and congratulations to looking or revisiting this growth employment of the nexus. I think employment as the main intermediary between economic growth and improved welfare is obvious to everybody, yet this using it analytically is done that often. I have three, well, yes, two or three things I'd like to ask. One is, did you look at the supply side of labor? For instance, changes in labor force participation rates, changes in dependency ratios, and changes in labor force growth, not least. Because if these may vary from one country to another and they would probably explain some of the residuals in your analysis. Secondly, you look at shifts from low earning occupations or sectors to high earning sectors. And that means I suppose you capture structural change through shift of labor from one sector to another. Say from agriculture to industry. But did you look at all at productivity and labor income improvements within sectors? For instance, in agriculture, you would do capture that at all because that would be one way to increase or reduce poverty would be through productivity growth within agriculture, for instance. And lastly, you keep measuring things in terms of how many labor market indicators are improved or do not improve. But some of these surely are very highly correlated. Did you take that into account at all? Because it might be that if some instance is a, for instance, occupation sector, typically very highly correlated or sector and self-employment, you're very highly correlated. That for some of these are clusters. So if one is affected, all the others are also affected. Hi, my name is Beatriz Muriel. I'm from Bolivia, from the Inesat Foundation. And I have more comments and questions. I find it very interesting about the relation relating poverty with employment. In the case of Bolivia, for example, labor earnings represent the 90% of household income. So that is why it is very important. But I think that also it is very interesting the correlation that you show. And what is behind the correlations? I think it's very different in countries. In the case of Bolivia, for instance, what we see is that a high dynamics of the informal sector. The informal sector is impressive how was the dynamics with the, we have almost 60% of the economic is controlled informality is informal economic. So it is a reason what the point of view of poverty have been reduced. In particular, for instance, in formal employment, the real wage of informal employment growth, the right more than 3% by year. And on the contrary, and for the formal sector, for the private formal sector, it decreased over year. So you have to see, and why is the reason and many reasons among them also it is important in Bolivia remittances was very important for the consumption, internal aggregate demand and so on. But I think the reasons why the indicators of related with employment have improved are quite different, I think, between countries. Thank you. So thank you for the questions and comments. So in terms of the first question by Carla, in terms of the changes in poverty and changes in the equality, that's something that could be looked at a bit more systematically with Boojinyon's triangle. But it's not something that we didn't hear, especially because we wanted to concentrate on the labor, but definitely that's something that can be done. And regarding non-labor income, we try to, in the country case studies, we try to highlight the role of new programs in each of the countries, and we try to highlight the role these programs had in terms of moderating the impact of the Great Recession. But I think that this is something worthwhile that is worth another project, in that sense, not as commitment to equity, CEQ, Commitment to Equity Institute at Tulane, has been very useful. I think it would be duplicating this work. There's also been some work on the data side of the World Bank, but I think that part is there, or it could be another wider project, but I'm not sure about that. Regarding, so what I'll do is I'll go through these questions and if there are some bits and pieces that you want to jump in, let me know. So not a definitely education. So we look at this in this paper, we haven't highlighted it in this presentation, but we look at the changes in the composition of the labor force, and this is broadly consistent with what your work with Louis-Philippe Fan and what we are working with Gagliani, but he also found in terms of a story of supply that is constantly growing, and so that leaves more of a role for changes in demand, for the changes in relative wages. But of course, our conclusion to that was that we needed to know more about the story in each individual country, and that's what you're doing. Okay, no, it's not constantly, and it varies by decade, but it's not, we still find or believe that the most interesting part is the demand part and is the demand part, as you correctly said, in terms of countries and the story of terms of trade. On the Dominican Republic, there's definitely something we need to look at in more detail, thank you very much, we will check on that. Employment as the main, I shouldn't have recorded that, I didn't write all, but you started saying employment as the main intermediary between growth and welfare is obvious, but seldom not always done anonymically, that should be on the blurb for the book or something. So thank you. So we do look in the paper at the supply side, perhaps not as much as in detail, as you mentioned. We do not look at changes in productivity within sector, so I agree that this is a partial change story in changes between sectors and not within sectors, and we had a lot of discussions about this, how many labor market indicators change, and so these indices, we didn't want to have an index, we didn't give it a name, that was a step forward, I think, we don't have a name for the index, it's just Z, there are different ways to weight the components, et cetera, we went for one over K, which was the easiest. The story doesn't change much if we do weights with correlations, et cetera, and you could have started by saying why is those 16 not others or more or less, it was just a way of organizing the unwildly amount of information from all these countries all these years, but if you have a proposal for an index, we could have a website where you create your own index with your own weights and see what happens. Sorry, and so that is yes, definitely in what is behind the correlations is different from country to country, and again, formality or registration, which also security, et cetera, is something that is very important in the region that played a very big role here, and it's not something we understand completely what happened on average or for each country, and so we need to look a bit more into that in further papers that definitely formality in the region during this decade is something that will be worthwhile researching on itself. Sorry, and you wanted to add something. And I was wanting to add to this problem with the index, we are not sure also about the index to doing just the percentage of the labor market indicators that improve. That is why we include the analysis of what happened with all the labor market indicators considered one by one. So the indicator is just a summary, but the important part is what happened with analyzing each one separately. I wanna add a couple of comments to each one of the questions. So the first one, respect to the role of education. So we have the share of workers with high education, with median education and with low education, and we found that countries with larger increase in the share of high educated workers, for example, experienced larger reductions on poverty, but the correlation was high, but was not as high as other correlation, for example, the share of high earning sectors or the mean labor earnings. With inequality is, yeah, that's true. So maybe it is affecting through the channel of inequality. That's definitely true and it's something that we can look at. It's a very interesting point. Okay, and yeah, and the Dominican Republic story is quite a puzzle because we found a large rate of economic growth, but we also found a decrease in labor, in mean labor earnings, and we found a small increase in inequality. And when I look at the data on Dominican Republic and because I was striking by this fact, and what you find is that the share of labor in Dominican Republic has decreased. So it may be a story similar to Piketty one in which the capital has increased the participation on the GDP and we are not observing the results of economic growth in the household service because they are going to the top 1% or they are going to some other part of the distribution that is not present in the household service. That may be a hypothesis, we should look at the data on taxes for that. Finally, the labor force participation, we have that in the country papers. We didn't include this here, that here because here we put all the labor market indicators that we put a direction on the welfare, a welfare direction, and increasing labor for participation is not clear if that's welfare improvement or welfare worsening because it depends on the reason why people is getting into work. So that's why we didn't include it in the 16 labor market indicators. But we have that in the country studies and it's something that can be looked at. And that's all I'm going to say. I just want to add something too and then we'll go around with another round of questions. And that is that the country papers themselves are quite detailed and they have lots of disaggregations. So you were asking about changes in earnings in different sectors of the economy. We actually disaggregated earnings growth not only by sector but by occupation, occupational position, gender, youth and adult and there may be one other one, I'm not sure. But anyhow, there's much more material there in the country paper, each one of them is about 50 pages long text and accompanying tables. And just now they're being put up as wider working papers and so everybody is invited to look at whichever one or ones you're interested in. Okay, all right, now another round of questions. One, two and three and then if we have more time we'll have more questions. Hi, such is Lovada from the University of Illinois at Chicago and so my question comes from a non-economist so I wanna make a disclaimer and it happens to be a sociologist and I am very curious to understand or to, yeah, it's a question, like why is it that if you're so interested in labor force participation and considering that after 1986 there was this big flow of immigrants from Latin America who created up to 40 million now living in just one country which is the United States. You are not considered in either remittances or labor exports as a variable in your labor force participation that will make a very interesting story at least for El Salvador, Dominican Republic, Haiti and Mexico. And so I am curious why like economists are not like very much into considering remittances for the region of Latin America seeing that it's not negligible when you aggregate them into how much they ended up in only one country in a very short period of time, post 1986 that will really be a great addition for a 2000 and 2010 data set analysis. And so it will be very much appreciated by sociologists and political scientists and others who are always looking for econometric data that to measure remittances and there is not much out there. Developmental economics I feel like it's behind in understanding that migration and development nexus. So I wonder if you have like theoretical insights as of why this happens in your data set and maybe beyond your data set. Thank you. I'm Susan. I have a number of questions which are out of curiosity. The first one is on regarding the sectors that were driving the increase in employment in the different countries, particularly Bolivia that is mainly a natural resource country. We always have that belief that natural resources are not labor intensive. So was the growth in employment in the mineral resources or they were diversifying the sectors and such that it was other sectors that were driving the increase in employment. Another concern is what was happening to population, say population size because I think it also has a bearing on the employment rate and unemployment. So I would also like to know what was happening with employment during the period of analysis. The other concern is the relationship between employment and poverty. You told us the story that is mainly because of increase in employment. Is it the complete story or they could be that the social sector, social protection was a contributory to the reduction in poverty. And lastly, it would also be good to know what was happening to labor productivity. We've seen that there was an increase in main earnings over time. Was it as a result of increases in labor productivity or they were just efficiency wages? Thank you. I'm Andrea Cornia from the University of Florence. First of all, a question about, was Germo already answered a little bit on the motivation. Now, if you are in an economy that grows substantially for most of the period, unless you have priors suggesting that you will have a jobless growth, then probably you should imagine that the labor market will improve welfare. And so do you have a motivation thinking that this was a jobless growth? Probably not. And so then, okay, you provide more details but you'll find what was to be expected exactly. Now, the second question is in our volume, Germo, to which you contributed and also La Jefa del Commitment to Equity Project. La Jefa. La Jefa, no, La Jefa in general. Well, I mean, I think that what we learned is that the part of the success of Latin America in reducing human equality was related to what happened to labor market policies. And these were, I mean, I repeat them, minimum wages. All the regions increased except for three countries. Second, formalization of employment like through institutions like labor inspections. In Argentina, what Saul Kaifman and Roxana Maurizio were telling us that before the changes, there were three inspectors for all Argentina to check whether the firms were complying with the labor regulation. And then with Conel Marido and La Mujer, Kirchner, they grew to 400, 500, so they're more serious. Thirdly, wage bargaining. So the wage setting has become more negotiated between trade unions more than during the neoliberal period. Now, then there has been as part of the safety net, some public works, not in many countries, but in some. And then finally, there is the issues that Nora La Jefa mentioned, that there has been a large increase in the supply of educated labor force. So these have been all favorable factors. So you could have said, aha, I think that these policies have been basically revoked, and therefore I'm expecting some possible negative changes. As far as I know, they've not been abandoned. And so the only doubt you may legitimately have, and then Mariana mentioned that during 1998, Bolivia, the crisis was basically not due to much because of stabilization funds. And Latin America is number one in that. So unless you have a huge contraction of the world economy, exactly you would expect that the labor market outcomes with no changes in policies would be, I mean, with all the variations you have shown, there should be a positive, you know? I mean, this is not the part to the world of jobless growth, you know? We don't have all this labor market rigidity that should have in Europe. So then the first one is motivation, but where are your fears? Why do you do this? And second is why do you think that in 2014 and 15, the continuing crisis in the West, because there might be a contraction induced by what happens in the world economy. And so we'll, I mean, how long will, because I have to write a paper with the same question asked, is it structural or is it transient? And my argument is to say, so far, I've seen no changes in institutions, this and then the redistribution. So they will find it more difficult to finance it. But I don't see any reason why Latin America will become less redistributive if not less money in the budget. Let me do this as the chair of the session. It's now 3.32 and there's a coffee break beginning now. We will definitely answer the questions and we would be glad to go on answering questions for anybody who would like to stay. Anybody who wants to go to the coffee break now is a good time to get up and do it before we answer these last questions and take another round. Okay. Your excuse. Okay, who wants to? Well, regarding the migration question, the thing is that we are analyzing the local labor markets, right? So the way the remittances from abroad can affect the labor market indicators we are considering is in the importance they have in the household income. So the way they appear in our analysis is not so much, in fact, they do not appear in the cross-country analysis but in the country papers they appear when we explain the evolution of the poverty rates and the inequality of household per capita income because it is part of the household income is a source of income for the families. That is the way we are considering the remittances from abroad. That is the role they play. Then for the case of Bolivia, what we have is that most of the improvements were related to the mineral and hydrocarbon sectors and there was also an increase in public servants, so people in the public administration. We didn't consider the changes in the population size in the analysis. Maybe a way as the factories enter or this variable is in the changes of the labor force participation rate that we consider it didn't appear here in the presentation but it appeared in each of the country papers. So thank you for the question. Definitely I'm going to link one of the population growth and labor exports, which is minus population growth, could be a factor which we did analyze and it's something worthwhile. Now, I think you've hit all the right buttons. When I, at the end, we mentioned in the presentation, well, we need to understand a bit more what is going on, the mechanisms. For instance, natural resources are definitely not labor intensive, okay? So they're a bit more than we think but as Maniana just mentioned, many countries for instance have export taxes and so the countries that benefited with higher exports and higher terms of trade also had higher revenue collection which implied an increase in public employment but also, for instance, in Argentina, there was a construction boom that some attribute to the increase in wealth from the owners of land that appreciated a lot in value in terms of exports, et cetera but it's exactly those mechanisms and as you mentioned, productivity that need to be understood more fully to see what happened because definitely there's a lot of mining in Chile and there are huge mines and amazing mines, et cetera but there are drivers and there are people running the mines, et cetera. There are hotels for the workers but it's not necessarily that intensive. How that trickles down into more employment besides public employment is one of the key unanswered questions I think and that we need to know a bit more about. In terms of social protection, your question is very similar to the comment that Calla made, that's definitely another chapter and we could include remittances in all the other non-labor earnings there. Commendatore Cornia, did I have? So as we said at the beginning of the presentation, definitely if you want the motivation was understanding what role employment growth and changes had here because we have this paper where we show changes in labor supply and labor demand relative in terms of skills, et cetera but we don't look at relative terms, inequality, et cetera. We don't look at what happened with role employment numbers, what happened with trends besides it and in terms of labor market policies, so on the one hand we definitely agree and we mentioned that at least in the last slide, there was a more labor friendly, there were more labor friendly policies and ambience if you want in most countries in the region. Now, but these changes happened, okay I always give the same example, Argentina and Paraguay experience about the same changes in inequality, et cetera, Argentina has 30% export tax, Paraguay has 0% export tax and so it's very difficult to find a silver bullet and as you say, well formalization policies, okay, Argentina went from three to 300 inspectors so that's a 10,000, I don't think I can divide anymore but it's a huge, a hundred fold increase, it's still 10 inspectors every 10 million, every million people, okay, so a million wages increased a lot in the region and there's certainly something to look at but they increased at the same time as wages increased so I'm not saying it's not there and I'm sure they played a role, I'm not sure we can attribute like quantifying the role exactly at least at the aggregate level as wage bargaining is the same so there was definitely, there were changes in institutions, et cetera, I think each country's own story will be able to attribute how much of the changes we observe can be attributed to those changes in policies. The policies are still there, I'm not at as optimistic, I think if the economy remains flat in the world, even if you had some of these great minimum wage policies, I'm not sure they will increase a lot much more further in the near future if the word they're coming in on me doesn't pick up, so it's good that they're there and there's certainly a new floor but I'm not sure how much they will support. Just to add a little bit more about the motivation of the project, I think that we all have some expectations that we will find some positive stories in Latin America but we were surprised to find that positive story in 15 out of 16 countries very generalized improvements in labor market indicators. I think that we were expecting to find some stories, some positive story, but what we found was kind of striking how generalized the improvements in labor market conditions were. And I think that the motivation is not only to think about Latin America but to think about other regions of the world how labor market grow works through labor markets to reduce poverty. So in this case, we applied the analysis to Latin America but it's something that can be used for performing the same kind of analysis in other regions and see what's the link between growing employment and poverty. So, Alain, you have a question and Roberto, you did, are there others? Okay, and Andy, so those will be then the last three. Just a quick point. Number one, it would be good to do a more disaggregation especially by youth. There's clearly a big question as to how much unemployment and how much participation is created for the youth. The second is to look at kind of trajectories of wage. For example, employment in the Dominican Republic tends to be free duty zones and there's very little labor mobility so it's good to reduce poverty but as a one-shot thing, it does not lead to wage progression. And the last point I would like to make is this is great as a diagnostic but it really lacks identification and at some stage you have to worry that if you want to make causal statements about this relationship in growth, employment indicators and poverty, you have to look for identification strategies. I mean we are in a situation where diagnostics are good but it's not enough in our profession. Yeah. I'm happy to see that with a complete different methodology, the results are similar of which in the paper that Damila and I wrote for the project that Commandatore Cornia coordinated few years ago. Engineer, Commandatore is better. And so happy to see the macro analysis of the relationship with the activity levels, growth and employment and poverty. So in the case of, we desegregated in that paper between the Caribbean and north and Mexico and the Caribbean and the south of the Columbia to the south and we show that in this case there is some additional factor. For instance, terms of trade have a dire effect on employment and poverty reduction independent of the effect of growth. Even taking into account growth, you're taking into account. And in the north, in the Caribbean, I'm Central America, I'm Mexico, is the rate of growth of the United States that have a dire additional impact on employment and poverty reduction. They may be related to these premises in the case. But okay, this is the command. I have a question, maybe it's a mistake or something that you mentioned Guillermo, the 2008 great recession. But 2008 was, the crisis impacted the region in 2009, not 2008. So talk about 2009 because it's confusing because 2008 was the highest terms of trade of the region. Yeah, so the Cleveland brothers cried, it was in September. So it's the last quarter of 2008 where the prices of commodity fell, 35% was the falling nominal. Do you buy eight slash nine or you prefer nine? We can reflect. No, no, no. The recession was in 2009. In the region, the recession was. Even in Germany, it was in 2008. In Germany, the recession was in 2009 because it was the impact after the failure of Lehman Brothers, for instance. And you show in the graph that the only in 2000, you show the per capita rate of growth and you show that in 2009, the per capita rate of growth, not 2008. No, it's a mistake. Okay, just another disaggregation point about gender. I wonder to what extent, Gary mentioned gender. I wonder to what extent there is a gender story here. Increased female level for forced participation, women getting better jobs, getting better paid. Those could be really important drivers of poverty reduction. And I wonder if that's true in this case. I've got a research project on this in other regions of the world, so I'd be interested if you do have a perspective on that. Yes, I'm Paul DeHunen and I'm working in the Committee for the Future in the Finnish Parliament. I'm not a parliamentarian, but worked 25 years there. But anyway, Committee for the Future will travel to Latin America. That's why I'm here. And suddenly I will bring to the delegation. There is the delegation from my committee. 11 parliamentarians are traveling in October to Latin America. Very many of them first time. And this is the, I really thank for the material I can bring them. But my question is we have been discussing the Committee for the Future a lot about jobs, the employment. Technology and science has brought a lot of jobs, a lot of growth, a lot of, so the poverty has went down in Finland. But unfortunately we have noticed we are in the situation that so many parliamentarians are afraid that in the long future, science and technology with digitalization with intellectual robots, high technology, unfortunately they are not any more jobs. So for ordinary people, not even for the academic ones. So the intellectual, high educated Finns. So did you see any remarks, the role of science and technology? First, doing great job for poverty. Going down poverty, but in the future it's a risk abyss. Yeah, thank you, Moabu from Kenya. You show that the hand-county ratio fell from 40 to 20%, which is a 50% reduction of a 10 year period. Actually I think that is too much a reduction. And the growth elasticity of minus two also looks too high. Actually that eliminates poverty completely of a 10 year period if the annual growth rate is 5%. Okay, so the growth elasticity of poverty is too high and also the fall of a 10 year period is also an abbreviably too high, thank you. So this is a shameless self-promotion our friends, Nandaro Haparin and Maria Marconi, we just finished a book in fact on female labor force participation in Latin America, which I can, I just received the print proof, so. And there's a story, there's an amazing story. So for Latin America for the last 25 years, male, prime age, et cetera, has been 90, 95. I mean it's up there. Female labor force participation has been increasing steadily. Now something happens in 2002 when inequality starts to fall, et cetera. So there's definitely something happening. Every thing I look at in Latin America, there's a break in 2002, so we need to understand what happens there. So female labor force participation was growing on average 1% each point a year and after 2002 it starts growing at 0.3, okay. So there's a break there and it stops. Now that is not necessarily bad news. So this is more quality, I'm sorry, but I was doing these interviews with people and there was this woman who was the beneficiary of a new social protection program in Castranford program in Argentina. I asked her, did you stop working when you started receiving this? Yes, what were you working? What was your job? Well, I was selling band-aids at a red light with my baby in my hands in England. Is that a decrease in that kind of labor force participation? It's not necessarily bad news, but there's something going on there and we have a little bit of, we have this aggregation by gender and we look at it, but there is another book on this if you're interested that we can send you. So Roberto, 2008, Lehman was 2008, we can call it 2009. If we don't want to confuse people, we don't want to reinvent the crisis, definitely. And so youth, we have a bit more degradation in the country papers, but perhaps not as much as you would like. The story of trajectories is certainly very interesting and it has to do with the flexibility of the labor market, but we don't have much on that. In terms of diagnostic and identification, yes, but we still believe there's a... So all these 16 country studies follow exactly the same structure, have the same data, comparable data, et cetera. So we hope that some people will find massive amounts of comparable information useful, even without identification and we're very careful when one gets carried away and says poverty causes, or increases in earnings causes, et cetera. But that is only when we speak and in some slides, we had an identification police getting rid of those statements or most of those in the papers. Just to add a couple of things. So in the data for the recession, we use the changes between 2009 and 2008. So all the data that we show about the recession is the recession, independently of how we call if it is 2008 or 2009 is the effect of the recession. But it's 2009, I agree. And then to answer the point about the elasticity. So the elasticity is a percentage change. It was not a change in percentual point. So even if you find a 2.0 elasticity, it doesn't mean that it's going to end in 10 years because it's decreasing by 2% by each increase in GDP of 1%. So it's just a relationship. It doesn't mean that it's going to end. And in fact, it will converge to zero, but it will not end at any period if you made the graph. And the reductions in poverty is huge in Latin America into 2000, but that's what the data said. That's what the World Bank study says that what another papers has found is amazing how much progress Latin America has done in terms of reducing poverty. Before I land these, I think that there were some hypotheses. So for example, there was a hypothesis of jobless growth in many countries, including Latin American countries. We didn't find any evidence of jobless growth. There's also a hypothesis of growth of jobs without growth of wages, right? Okay, which is the United States, month after month. And we didn't find that either. Okay, we found that, so that's where we were looking for trade-offs, for example, where something good in a welfare improving direction was happening perhaps at the expense of something else. Higher minimum wages, but less employment. Didn't find that. So there's a sense in which we're not assigning any causality other than to say some of the obvious causal interpretations, like moving up a downward sloping labor demand curve, we didn't get a few other things like that. But we have a long way to go before this would be any sort of causal story, country do this. But the main thing that we were trying to say is that there's this widespread improvement in 16 Latin American countries, 15 Latin American countries, was one didn't have it. And this was to me and to all of us, quite remarkable, quite a robust and strong finding, and disaggregated by all kinds of things. And that's what we wanted to share with everybody and put on the table. So thank you all for not only for coming, but for your patience and missing the coffee break. And we thank you all.


Okay, traders, that is 1pm, Greenwich, meantime, and we are about to get going here with this week's live market and trade analysis session with me, Patrick Munnally. Before we jump into today's presentation, as always, I want to adhere to the risk disclaimer. Most pertinent today's presentation is the views and opinions expressed by me are solely mine. They're not indicative or representative of those held by Tick Mill UK or Tick Mill Europe Limited. Those of you here for the first time, brief introduction to myself. After I graduated from university, I joined a city PLC consulting firm. I left with some colleagues and went on to successfully co-found and exit a consulting startup focused on C-suite executive search for technology businesses. Essentially, I had a front row seat to the dot-com bubble, witnessing people make and lose a fortune in the market sometimes quite literally overnight.  I started day trading the S&P 500, or probably more appropriate at that stage, day gambling. And after some early beginner's luck, I actually racked up some pretty solid gains. However, as is often the case, my beginner's luck ran out. And as the market phase changed, I began to average down into losing positions, giving back all my gains and ultimately experiencing a significant six-figure hit to my personal capital. So this was a gut-wrenching and sobering experience is an understatement. So I really had to stand back and figure out if it was feasible for me to make a living from the market. So I decided to get serious about trading and I sought out a mentor with an excellent trading track record. Working with this mentor for a period of 18 months was a time during which I had not just my technical game in terms of researching, developing, sensing back and forward testing strategies that crucially suited my personality and all of which were underpinned by a rigorous risk management approach. Most importantly though, during this period of mentorship, I significantly developed my mental game.  So what does that mean? Well, it means I had to stop focusing on what I could make for the markets and start focusing solely on managing my mindset to allow me to consistently execute my trading strategy oftentimes in the face of negative feedback from the markets in the form of losing trades. But once you become process-orientated and you have a professional trading mindset and you understand the true nature of trading being a numbers game in which we're simply playing the probabilities, you lose that emotional investment and that hellish emotional rollercoaster of living and dying by the outcomes of individual trades. So I'm no longer concerned with the outcomes of individual trades or even a small string of trades. My focus is on the next 100 trades because I know if I focus on excellence in execution, my edge will demonstrate itself over an extended series of outcomes. My multi-strategy approach has delivered profitable annual returns since 2008. Since 2013, I've also been managing investor capital through a managed account service, again delivering annual positive returns. I'm currently responsible for managing a multi-million dollar portfolio. Since 2010, I've also mentored hundreds of private traders of all experience levels from complete novices to former CME floor traders in developing the technical and mental skills to reach consistent returns from the markets. In addition to fund management and mentoring, I'm a resident market expert, exclusively providing market and trade analysis to Tick-Mill clients. I provide an in-depth daily market outlook, breaking down fundamental and technical drivers for the day ahead. I also provide daily technical trade setups through the Tick-Mill TradingView account and I'll post a link for that at the end of this presentation. I also run Tick-Mill's E-mini strategy Facebook group where I post a daily trade plan outlining my pre-market thoughts for the cash trading session ahead. I give my bias for the day and my specific action areas where I'm looking to engage the market. These pre-market plans have delivered over 5,000 points of profit since we started the group in April 2021. The second Tick-Mill strategy group I run is for traders who really want to take their trading to the next level. The Tick-Mill Futures Telegram group is a real-time environment where on a daily basis I share in-depth insights, analysis and real-time trades. I also provide live commentary during the opening hour of the cash trading session in New York where traders can essentially see in real-time how I dissect the markets and identify asymmetric trading opportunities. These sessions act as a platform helping traders to develop a professional, consistent approach to navigating the markets and the mental mind games that must be mastered to make it as a profitable market operator. That gives you a flavour of where it is. I'm coming from now. Let's jump into the charts. What I would say before I get going with the charts here is I'm going to run through the charts that I'm tracking at the moment and set up some I'm watching. If you have any questions just drop them into the chat box and at the end of the session I'll come back to those. If you have an instrument you'd like me to take a look at, I don't cover in my slide deck here. You can also put that into the chat and I'll give you a view on that. I would say today I'm a little bit time constrained because I want to be off at least 10 minutes before the CPI release as I'm obviously going to be looking to trade that myself today and trade the market reaction. I want to get prepped for that. We're going to try and keep this to 20 minutes today. Starting with the SM500, two competing setups here. The bullish thesis is versus this 3700 level. We're sitting right at trend line support here at the 3750s. If we can hold here and get a breakthrough back through 3840s then that's going to give a bullish impetus to the trade for a break back through 3870s to retarget the prior highs here 3925s. The ultimate objective versus the current setup and I'll just draw this so you understand what I'm talking about I'll repeat this throughout the presentation. I'm looking for an equality objective versus this swing structure which gives us a target level now of 4128s. Now if we don't, if the CPI print comes out on the hot slide today and that gives further cover to the Fed obviously to ramp up the rates we will see a negative reaction. I've been looking for any break of the key level for me on the downside today is 3996. Keep that level in mind. If we get a breakthrough there then the next downside objective versus the swing structure here and equality objective will be 3642. So any break of 3696 today I will be looking to be sure and I'll be targeting initially 3642 as the next downside objective. Moving to the NASDAQ. So we're sitting again NASDAQ is sitting at some pivotal support. NASDAQ has been the weakest of all the equity indexes obviously it being a duration asset so you the investment in a lot of these tech firms is that the hope is down the line that we're delivering revenues etc and with obviously fiscal conditions tightening they don't have access to that capital and that's been weighing on the stock prices so the NASDAQ has been the weakest of all the equity indexes. So from a technical perspective at the moment versus the swing level here at 10641 we have an equality objective above us at 11878. For me I'd really want to see a close back through this high volume node 11128 to get a confirmation signal that we're likely to trade up into this target zone. If today we again it's going to be pretty much dependent upon this inflation print but if certainly if we start to take out the invalidation level here 10641 my focus then would be on the downside and we would be thinking about a move towards 10,000 on the downside. So that's those the key levels in terms of the NASDAQ. Dow Jones has been the outperformer in terms of US equity indexes and the reason for that as I mentioned last week is this is where the value plays are so growth is in tech and the value safety, utilities, health care, financials etc are in the downs that's why it's been a relative outperformer, relative strength we're seeing in the down. What I'm looking for now is any pull back into the trend channel support here 32,000 watch for bullish reversal patterns there to engage on the long side. I've got a target on the upside 33,800 equally any close back through this trend channel resistance this internal trend channel resistance would be a bullish confirmation for me as well. So close back through 33,000 again same target on the upside 33,800 at this stage it would take a close back through 31,745 which would be a bearish development and then we would be starting to think about downside objectives but for now the impetus here whilst we hold this support zone 32,000 is for 33,800 test. Russell small caps again relative weakness we're seeing here in terms of the Russell so versus the resistance at 1796 I'm looking for a test of 1721 if we take that out on a closing basis the next downside objective for the Russell here on the weekly versus the swing structure and the quality objective is 1575 no worthy the Russell is the only index out of the US indices not to have tested it's a quality objective all the others have moving to Germany. Dax has been a relative out performer and what we are seeing here at the moment is the potential now for any move that tests and holds 13,474 I'm looking for the next leg to the upside to target 14,000 from there I'd anticipate we get another pullback equal to let me just draw this in here so you can see what I'm talking about in terms of the wave structure we have wave one two potential wave three here into 14,000 I've been looking for wave four back into 13,300 and then we get a final fifth wave extension towards 14,200 and then from there I would anticipate we see a deeper corrective move so for now the focus is really on this 13,453 if we take that out then I would anticipate that this is actually our wave three high and we have a wave four equality objective to the downside at 13,000 when I talk about the equality objective I'm referencing this symmetry swing here in terms of our the scope and scale of our wave two overlay versus our potential wave three high so that's what we're tracking in terms of the Dax there the Nikkei traded up into the target zone we're now getting a pullback and it is sitting at pivotal support so the Nikkei you can get a close back through 27,500 I've abolished the Nikkei looking for an upside objective now into 28,200 it would really take a loss of the high volume node here 27,000 to suggest that we're done on the upside for now and we'd be thinking about a test down into trend challenge support 26,400 the nifty the indeed the nifty here this has got an interesting technical setup here this is one of the few equity indexes to actually make new highs this year so with the Nikkei with the nifty here any pullback into the 17,848 I've been watching for bullish reversal patterns there and we have a five equals one objective and a one two seven extension versus our last corrective leg which gives us an upside objective just below 18,500 so 17,840 area what's bullish reversal patterns gives us a target there 18,500 TLT the iShares 20 year treasury bond ETF got a setup here that's developing obviously it's going to be very dependent upon the inflation data today but if the inflation data comes in in line or a little bit weaker than expected we could see an upside extension here we've got a target versus the swing low the invalidation level is 93,30 and that gives us a 99,19 target on the upside if inflation comes in hotter money's going to be coming out of the out of the bonds and obviously raising yields so we'd anticipate then that our next downside objective is 90,26 on the downside again I've watched for bullish reversal patterns there as long as we maintain momentum divergence and we can see another corrective leg to the upside moving to the FX phase dollar index has it tested or came just shy of last week's equality objective which was the 109 level now seeing a corrective move here I'm looking for a rejection into the 111,30 111,50 area to set up the next leg to the downside we have a new equality objective versus the swing setup which gives us a test of 107,72 this stage any close back through or above 113 would be a bullish development we would then be looking for an upside extension into the next targets on the upside are as high now as 118 but we have so far managed to maintain that that resistance at 113,85 so that's a key level on the upside to bear in mind but again this is going to be very dependent today on the inflation data we can expect some volatility there but we have our invalidation level set up and we have our target levels euro we've got a new upside objective in terms of the euro versus the swing low here at 97,29 I'm looking for a test of 102,84 we've got a little internal equality objective that's been tested here 99,40 we're finding a little bit of demand coming in again though this one's going to be very dependent on the on the inflation data but certainly any move back through the 10,040 would be bullish and we'll encourage a test up into this 102,80s target zone sterling similar setup here we have this trend line support coming in so I'm looking for selling to ultimately test and hold just below 113 and if we get that the next upside objective is going to be this ascending trend line resistance that comes in 117,65 we have an ultimate target level up at 120 versus the swing low here at the 109,20s Dolly Yen got the breakdown through that 146 area we've now snapped back to test resistance here the 147 as long as 147 holds we have a downside objective at 143,26 which is the equality objective here versus the swing structure after the last leg of intervention from the BOJ this stage any close backs above this internal trend channel resistance so 147,60s would suggest that we're in a range trade I think then and we could think about a move certainly back up into the 148,85s and then potentially into the invalidation level 149,70s Euro Yen Pay close attention to 145,20s if we hold there and get a bullish reversal pattern as the potential that would extend higher again and take out this trend channel resistance 146,70s however if we don't find demand here at 145,20s currently have a downside objective in terms of the Euro Yen and this swing structure here the equality objective gives us a downside target of 143,16s so watch for any break of that 145,20, 145 area as confirmation there so we can see the daily setup is looking pretty perilous as well so any daily close back through that 145 want to be short Euro Yen looking for downside objectives let's move this on dollar CAD got a nice setup developing here so I've been looking for any break back through the 134,90s 134,60s we have two equality objectives sorry not an equality we have a 161 extension from the swing high here at 138,50s but we have the equality objective 138 this would be considered WXY pattern for earlier waivers so that we have the confidence down here at 133,20s, 133,30s so any loss there back through this 135 we have downside objectives 133,20s, 133,30s moving to the Aussie so we we've achieved our first target on the upside which was that 65,20 test we then got the pullback held and we now have a new equality objective which versus the swing low here at 62,70s we are looking for a test of 65,90s so I've been looking for any close back through this internal trend channel resistance as as confirmation for that next leg to the upside or we get a deeper pull back here into the trend channel support again at 63,20s watch the bullish reversal patterns there break through the trend channel resistance same target on the upside is is what we're looking for there 65,90s this stage only a loss of the 62,70s the embarrassed development and we're going to be opening downside targets again the Kiwi similar setup to the Aussie with a nice trend channel here so we are looking to find support into 58,18s the 58 level let's bring in our internal trend channel resistance so I've been looking for any close through this trend channel on the upside as a bullish development and we've gotten a quality objective new quality objective on the upside at 61,60s at this stage it took a close back through 57,39s suggests this upside correction is complete and then we'll be thinking about downside targets again Euro-Soyce this one's got an interesting setup so if we test here if we test and hold the 98 to 98,10 level watch for bullish reversal patterns there engage on the long side next target to the upside will be the 99,30s but ultimately we have this equality objective still in play which is 100,30s on the upside moving to the metals at gold gold nice bullish structure here so I'm looking for any break back through the 1720 to target 1750 on the upside in terms of gold if we if obviously if the inflation data is hustling expected I think we get a deeper pullback and we likely retest prior trend line resistance then to actors support before we try and make this next move to the upside relatively I'm looking for 1740 1750 as the as the target there in terms of goals crude oil in a box trade at the moment really I think we're just in this range 93,70s to 81,20s big range but versus the swing low that we have at the 81 level as long as that holds that 81 handle we still have an upside objective here at 98,70s in terms of crude oil I'd be waiting for a break of range resistance before going in on the long side and we'll wrap things up with Bitcoin obviously it's been a torrid week for for crypto balls with the FTX collapse and for those who have been here week in week out we've still got about 12,185 level if we get down there if we see some force selling some margin calls coming in and we liquidate down into that area that's going to be a key area for me I'm going to pay very close attention for longer term position trades in terms of crypto and Bitcoin any move into that area I'm going to be certainly watching for daily reversal patterns to build longer term swing positions in terms of in terms of Bitcoin there and the the crypto space in general so that's the whistle-stop tour of the setups I'm looking at obviously paying very close attention to the CPI data coming up in 10 minutes and and that's going to have a big impact today in terms of dollar risk sentiment in general so so that's going to really define the next I think directional drive in terms of these these markets are there any questions post quickly just post links into the chat here for those who want to get the daily trade plan for the S&P 500 or E-mini S&P all you have to do is request access and you can and you can get that daily trade plan and I post some other interesting information institution insights etc in that group okay I can't see any questions coming through so like I said I'm going to wrap this one up a little earlier today as I want to to prep for the CPI trade as always traders plan the trade trade the plan and most importantly manage all risk until next week thanks very much


So, welcome. I think we can start now. My name is Maxime Chevalier. I work at Smile. I'm based in France, in the south of France. And I work in embedded Linux development. And so, in my work, I've worked on multiple projects that use the preempt RTPatch. And today I'm going to present you what it's like to use the RTPatch with more of a user perspective. I'm not a real-time patch developer. So, does any of you use the RTPatch in the project? All right, quite a few. So, let me just present what kind of projects I've worked on. So, the first one was a simulation platform. It was in the aerospace industry. It was one of these big workstations with two Intel Xeons inside, loads of RAM, and we needed to run software that would simulate real code that was supposed to run on a real-time operating system. So, the main constraint here was to be able to respond to an event that would come on the network interface in a given time. And so, we also run cyclic tests on this machine and our target latency was 200 microseconds. That's the first project. The second one was pretty similar. It was a test bench that was communicating with real-time software and it was performing acquisitions on various input ports. So, you have this kind of rack with a lot of input IOCards. On each card, you have maybe 128 inputs, so that's a lot of things to acquire. And so, when receiving a packet on the network interface, we would have one second to perform all the acquisitions, be it digital or analog, and respond in one second. So, that might seem like a large time to respond, but when the system is on the load, we experience some spikes that would go over one second. The third one was a project I've worked on for two years, which was a more embedded product. So, this time we were using an ARM-based platform, so an IMX6 from FreeScale. So, it was a product that was embedded into vehicles, like tractors, buses, trucks, and that was performing acquisition on all the vehicle bus, that's the CAN bus on this kind of vehicle. And so, in order to gather all of this data and never lose any incoming packet, we needed to use the real-time patch, because we also needed to run some customer load on the IMX6. We didn't have any idea of the very nature of these loads, but the system could be under heavy CPU load and still needed to acquire each and every one of these packets and never lose any. But here was more for a deterministic behavior rather than real-time, although it's quite the same as I'm going to show you. And finally, I worked on it on an image processing solution for the medical industry, and we were doing some real-time processing, real-time not as in RTOS, but real-time as the things are happening in real life. So, we have this video stream, we need to do some processing, and this processing is fed into a robotic laser, I think. So, that's for medical imaging, so you are filming something and you point to laser to what you are filming in real life. And so, if you have too big of a jitter or of a delay between the time you acquire the frame and the time you point to laser, you're not pointing at the right thing. So, that was pretty critical in the medical industry. Just to give you some feedback of what is some ground for us to be on the same ground on what is a real-time operating system. So, pretty much all of you know what that is with the quick poll, but RTOS is all about determinism, having a deterministic behavior, especially regarding timing constraints. So, when you have an event that occurs, you want the response to that event to happen in a known timeframe. That's what you want primarily on the RTOS. You also want to run multiple tasks, and to do so you need to have a real-time scheduler. This scheduler will decide which tasks need to run given a priority that is assigned to this task. So, a critical task is assigned a high priority so that it runs always before the lower priority tasks. And finally, on an RTOS, you want to handle some specific cases.  And so, with absolute priorities on your scheduler and lockings happening on the side, you can have situations where, due to this interlocking mechanism, you have a lower priority task that prevents a high priority task to run. That's called a priority inversion. So, we need to solve all of these problems to have a real-time behavior. What do we have in Linux? In Linux, we have pretty much everything we need to have a real-time operating system. Thanks to the effort of the real-time patch, the programmed RT patch, a lot of the things that were missing are mainlined. So, this is based on 4.9, but yesterday there was a pretty good talk about where things are nowadays. So, you have a real-time scheduler. You have three of them, SCADE 5.0, which is a pretty simple one. The first task that needs to run will run, according to its priority. SCADE RR, which is for run robin. So, this is the same as SCADE 5.0, but with a notion of time-sharing and time-slicing. And you have SCADE deadline, which is the new one, which basically allows you to tell, I need to have this work done by this deadline, no matter when it will happen, but I need it done by the next 200 milliseconds, for example. We have support for the priority inheritance problem. So, that was the tricky use cases that we can have. So, we use priority inheritance to solve the priority inversion problem. We have a preemptable kernel, almost, which means that a task can stop the kernel from running to do its job, but the kernel still has a lot of critical sections in it. And so, that's nowadays what the RTPatch is all about. It's removing these critical sections. We have high-resolution timers for quite a while. So, we lack full kernel preemption and also some worst-case scenario optimizations. So, what you have to understand is that between Linux and a real-time operating system, you don't have basically the same approach in designing these OSs. And when you design a real-time operating system, what you want to do is to be 100% sure that you will have your deterministic behavior. And in order to do so, you have to check every execution pass that your con can take, measure how long it will take, and prove that you will never have unbounded latencies. This is not feasible with Linux. It's impossible to do because Linux is too complex and it's moving really too fast to have this kind of verification and formal proof. So, what the RTPatch does, it's kind of a best effort to make everything in the kernel nice to everything else. So, every time we take a look, well, will we do so that, instead of having non-preemptible sections with interruption disabled, we try to do so that the lock is sleepable, is interoptible, you have still interruption that can occur. So, you make sure that every piece of code is nice to the rest of the kernel. To do so, they introduce the threaded interrupts. So, the threaded interrupts is also a mainline thing. It's just enforced in the preempt RTPatch. Basically, when you have an interruption that occurs, instead of running the interruption handler in a specific context, that would preempt everything that was previously going on, even if it was a real-time task, a hybrid task, the interruption handlers are now run into dedicated threads. So, you still have a tiny interrupt handler and all it does is wake up this task that is pending. The advantage of that is that you can assign a priority to this given task and make it so that it is scheduled after your critical task, if you don't really care about this interruption. The locks are sleepable inside the kernel, so that's where all the magic happens. You can take a look at some talks I give at the end to really understand what is going on. But you have to have that in mind when designing a user application, is that the kernel itself, in the ways it handles the resource sharing, behaves differently, and this has an impact on your performances until it removes the critical section, as I said. Just a quick look at how can we determine this from the user space. I don't know if you know the tool Perf. There has been a lot of talk about Ftrace. You can do that with Ftrace, but with Perf, we can really see what is happening to your task that is running on user space. For example, here I am running ping-floating from my machine, and I monitor what is happening using Perf. So Perf will every now and then see what current function is running inside the kernel and give you the percentage of how many times did you spend in that function when you run your specific load. So this is the result on vanilla Linux without the preemptRT patch.  So that's not good for real time, but that's kind of good for performances. On the contrary, on preemptRT, this lock is replaced with an RT spin lock which is sleepable, which means that when you cannot acquire the lock, you will go to sleep and let something else run. So you have a context switch here, but you are nicer to the rest of the system. So that's really just to show you that real time patch changes the things inside the kernel. So what about everything that is not real time related? Because the kernel is changed, but the ABI is the same between the kernel and the user space. So you can still run everything that runs on vanilla Linux. If you have your JVM with your custom load on it, you can still run it on preemptRT and have your critical task that runs on the side. What you have to be aware of is that your non real time task will have what is left of the resources. So what is left of the CPU time, what is left of the memory that you've locked for your critical task, I'll get into that later. So when you've applied the real time patch, what do you see exactly? So the first thing you do, you do uname-a, you see that, I didn't show the result of the comment, sorry. You have preempt and RT in your uname string. Also you have a file that is created in this kernel real time. If you look at what inside, it says one. You cannot go zero to disable the RT patch. It's just here to say I have the RT patch applied. Another thing that you can notice and that can be confusing is what happens when you do htop. I'm sorry I wanted to do a demo, but I have to reinstall the machine and I don't have the RT patch running on it. But I can explain it to you. What happens is that you have some more tasks that are running on your system. And this is because you have threaded interrupts now. Because of that, every interruption will now show in htop as if it was a process or a task running. This can be confusing because if you have something from the external, from outside of your system that is triggering interruptions, for example, the ping-flood example, if you are pinging your system from the outside, now the response to the ping that happens in interrupt context shows into htop. And you can have your load getting higher in htop, but your system is still doing the same thing. So actually, when you apply the RT patch, you can visualize better what your system is doing.  I don't know if you're familiar with the load of rage. It's not really an indicator that you should rely on, but it basically tells you how many tasks are waiting to run on your system. So if it's inferior to your number of CPUs, you're fine. And if it's over the number of CPUs, you're having preemption on your system and you have more than 100% of a CPU that is used. So that will be impacted too by the preempt RT patch, just in the way that things are computed. A good set of tools that I used, well, to debug, to analyze this kind of things because I had to prove to my customer that the preempt RT patch was having an impact, but it was not as big as we thought. So I used the stat tools, so PID stat, VM stat, and MP stat, which I found very useful, which allows you to monitor the events that are ongoing on your system. So it allows you to analyze the context-switching that happens, the interruption, the cache misses, page faults, and at some different levels. With VM stat, you have a global level of what is happening on your system. I will show you an example later. With MP stat, you can monitor, for example, the interruption on a per-core basis. So how many interrupts are handled by the Core Zero, Core One, how many context-switches, for example. And PID stat allows you to do that on a per-task basis. So here is an example of the output you can get from VM stat, the first one. VM stat, so the one after the common line means that I do an acquisition every second and I compare what's happening to the previous second. So R means how many tasks are running, IN means how many interruptions happen in the last second, and CS, how many context-switches happen in the last second. And I was running VM stat, and in parallel I was running stressNG with a 5.4 run. So stressNG is a tool that can allow you to stress some specific parts of your system.  And so while I am stressing my system with stressNG, I can see that I have about 700,000 context-switches per second. That is pretty huge. This is the same behavior on real-time and vanilla Linux. It's just to show you the tools. PID stat will allow you to have the same information but on a per-task basis. So this is one per thread. You have the PID. I didn't show it in the slide. So each of these processes triggered 70,000 context-switches, and NV is for non-voluntary context-switches. So it's the time that you get preempted by the kernel. So the kernel says you've run for enough time. It's time to let the other processes run. So obviously this is using SCAD other. This is not some kind of real-time task. Just doing what you would do on your system regardless of vanilla of preempt-arty. An interesting thing to show is what I talked to you about with the Htop is using these tools to monitor what's happening when you have a ping-flood happening. So that was the problem that I had to solve. So why, when I have a ping-flood with a real-time patch, my load of rage explodes and my Htop is going crazy? And what we realize is that the difference between RT and vanilla Linux is just only in the way things are monitored. But also the same thing in the real-time patch triggered more context-switches. So you have kind of a tiny performance loss in that regard. You can confirm that with PIDstat that shows that you have roughly the same amount of context-switches as this particular IRQ than you have interruption incoming. So this is really the culprit, the guilty part for this performance loss. So this is an effect of the thread interrupts. But if that bothers you using the real-time patch, you can still assign a different priority to this particular task. You can also use IPerf to compare the bandwidths available on the network. And IPerf showed no difference at all between vanilla Linux and pre-empt RT, although you might think there is an impact here. But this should be tested more with tiny packages because the more interruptions you get, the more context-switches you get, and so the more time you spend switching from one task to another. I also use StressNG a lot. So StressNG should not be used as a benchmark tool, but I did it anyway. It's a rough benchmarking tool. So if you are running something on vanilla Linux and you want to know how it is going to behave on real-time Linux, and you know exactly what kind of load it is, so is it doing CPU-intensive work like doing math calculations, or is it using a FIFO, or is it accessing the disk very quickly, or the network, you can try to use StressNG to get an idea of how this will behave on real-time. So I've used five stressors that I compared, so stressor is the thing that StressNG will run into a loop as quick as possible. I think you have like 70 available, 90? 180? Yeah. But I limited myself to the stressors when you can set the number of operations that you will run, because you have to compare the execution time. That's what I did anyway. So for the CPU, for example, I've run a load that took 11 seconds, and in RT it took exactly 11 seconds also. It's logical because you do not go through the kernel. Vault triggers a page vault, and here you can see that you have a significant difference. The highest difference is the FIFO stressor, which takes eight seconds to run on my system with vanilla Linux, and 70 seconds with preempt RT. So you can see that there are areas in your kernel that will change their behavior and that might be slower, or faster, like the few texas that is the fast user mutexes, and that actually gets faster when you have preempt RT applied. So you can use StressNG. That's what I did. It gives you a pretty good idea. But so to recap the performance issues, you can expect an overhead when you run into the kernel. So when you are doing a Cisco or you rely on a path that goes through the kernel, you will get an overhead. And the thing is it's very dependent on your platform, so you should test it very thoroughly as soon as possible. My point is that when you use preempt RT, it's only the first step and you want to have a real-time operating system. So you must first make sure that everything you want to run on your system will still run as you expect, but you also have to take into account the fact that you are building a real-time operating system. And so preempt RT is the first step to build a real-time operating system.  Why is that? It's because you have tons of other things than the kernel that can impact your latencies. So for example, the CPU idle state, and I don't know if you know what that is, the CPU idle state is basically what your CPU is doing when it has nothing to do. So for example, my laptop is probably 99% of the time asleep and the rest of it showing my slides. So what is my CPU doing in that time? It's not running in a loop waiting to have something to do because my CPU would catch on fire. Instead, it goes to sleep, and the sleep states are depending on your architecture, on the SOC you are using or the CPU you are using. On my system, I have 10 different sleep states that I can go into, each one having a different latency, different times that it takes for it to wake up. So when you want to build a real-time operating system, you might tweak your CPU idle state and disable the deep sleep ones in order to guarantee your latencies. It's also true with DVFS. So CPU idle states, you don't have many on ARM. You basically have, I think, one or two available, which is either pulled in a loop or wait for an interrupt. On X86, you have tens of possible states. But DVFS is also something that you are starting to encounter on embedded systems. It's basically adjusting the frequency at which you run depending on the load you have on your system. And this can affect your real-time behavior because if you're not running always at the same speed, how can you guarantee that you will respond to an event in a bounded time? Well, in a known time. So what you want to do also is to fix the frequency at which you run. You can also have hyperthreading on your system and my recommendation is to disable it if you want to have a real-time behavior. That was one of the problems on the big Xeons system that I used. You had a lot of CPUs and you had hyperthreading and hyperthreading and what's really the thing that blew our latencies. So we disabled that and obviously when you do that, you have very much less processing power. So you should also take that into account as soon as possible when you are designing your system. The key thing about all of this is really knowing your system. So both into the software side, so your resource-based applications, what's going on in the kernel. So you don't have to necessarily know everything about the real-time patch. You can just use some benchmarking tools or measuring tools to see what's happening. You also have to know your hardware because there are things that you cannot disable. For example, if you have DMA, well, you might be able to deal with it, but DMA can cause latencies on the buses of your SoC, for example. System management mode on Intel processor or x86 processor can be really helpful for your latencies. So if you don't know what SMM is, it was originally used on the laptops to do some thermal management. So every once in a while, you have an interrupt that fires that is not maskable and the code that runs into that interrupt is not under control of the Linux kernel. It's inside the firmware, the BIOS, I think. And so you cannot mask it and you don't know how much time it will take to process that particular interruption. You have tools like hardware latency detector that will help you see how much time you spend in SMM. But since it's doing thermal management, if you disable it, you can sometime in the BIOS, your CPU will catch on fire again. So that's not recommended. And you should also know if you have resources that will be shared in your hardware. I have a pretty good example that's not mine. That's one from a former talk at ELC. Someone did a talk about running real-time Linux on the multicore system and one of his problems was the single instruction multiple data the vector unit in its system was shared amongst several cores. So I think he had four of them and eight cores and so when he tried to run five critical tasks concurrently it would see he would see some really important jitter on his latencies and because that was happening on a hardware level. So things like that, you should be aware of that when you are designing a system. Just a quick refresh on CPU idle there is actually a way to know the latencies that it will take to wake up your CPU from the sleep state. In the CFS you can check out you have this pass and you can see how many time it will take for each state. So for example this is a quick view of the ones that I have on this laptop so I have tens of them and so for the first one which is the pole state so in that state you don't go sleep at all you just run in a loop waiting for something to do and you have zero latency, wake-up latency and the deeper you go the more time it takes to wake up so this is in microseconds and the real residency time is the time that the scheduler will try to approximate how many time you will stay asleep and it will use this residency time to choose the sleep state that it will go in. So you can disable CPU idle from the kernel command line so you can choose the max C state that you will use you can disable it sometime in the BIOS so on some platforms for example the test bench one we could not disable it from the kernel and the decision of the sleep state was made in a BIOS level so you have to disable that in the BIOS so that's pretty much it my hope was that we could have some feedback from you users of the real-time patch if you also have anything to share if you also are more interested in the real-time patch there are a lot of good talks about it a lot of them were gave by Steve Rastet I think he also gave one on schedule deadline but I cannot quote it on each and every line the real-time Linux on embedded multiple processor was the one that talked to you about resource sharing also a good talk about IRQs if you want to understand how they are endowed and vanilla in RT Linux and also you have the talk from yesterday about the real-time patch that had a lot of good informations so thank you now if you have some feedback or questions no feedback yes, let me give you the mic thanks for using stress and G it's a tool I've been writing to measure systems like this it helped me a lot the latest version has now a cyclic test built in so you can actually get a full histogram of latencies and all sorts of stuff related to real-time performance measurement that's right, I didn't know about it just to let you know that that's very useful anyway to benchmark everything in your system and also to stress your system besides any other questions or feedback well, thank you for coming