So good afternoon. We are gathered here for a first session on the future of macro and macro finance theory and models. Let me just say, as a matter of introduction, not as a matter of introducing the speakers, which I will not introduce, because they are very well known and respected, but as a matter of introducing the topics, that it is very proper that we start a discussion on the future of central banking with a discussion on the future of macro and a discussion on theory and models. And it is even more proper that we do it here when we're all gathered to contribute to our vice president, Vitor. It is very proper for central bankers to care about the future of macro and the future of modeling. We care a lot about models, theoretical models, and empirical models to quote, or maybe to misquote, max labor. As central bankers, we both have an ethics of responsibility and an ethics of conviction. The ethics of responsibility is about delivering on our mandate, which is bringing inflation close to 2%. I'm sure of it, or you will agree with that starting point. And the ethics of conviction, the Gizinox Ethic is about doing policy based on first principles informed by facts and framed by sound economic thinking, which is about having a sound macro framework and which is about having good models. And we're doing it seriously. And Vitor has been leading the effort here in the ECB to upgrade our models, revamp our macro thinking in the light of facts, as the president was saying, in the light of the lessons we've learned from the great financial crisis in particular, introducing financial frictions in our models, introducing home to mouth agents, and all kinds of imperfections that we've seen were important in the crisis. And it's not enough, and what we're going to discuss now is how to make it even better. And I always like to start a discussion on models, theoretical models, or empirical models by quoting General Eisenhower, who famously said that planning plans are useless, but planning is of the essence. So being on the receiving side of that discussion, I very much believe that models are useless, but modeling is of the essence. And I hope I will be maybe proven wrong on the first account and confirmed on the second account. So let me open the floor to John Mulbauer for the presentation of the first paper. Yes. And there should be slides. There are slides. See them here. Well, I greatly appreciate this opportunity to be here. And Mario Draghi said beautifully what I think about Vitor, just beautifully. Vitor has been very patient with me, has given me lots of time. We've had some wonderful conversations, not just about economics, but also music. And it's a real pleasure to be able to contribute to this occasion. Right. There is a paper. I'm going to talk about parts of the paper, just briefly the introduction. I'll skip the DSG New Keynesian debate most of it, but just touch on it. And then I'll talk about the positive aspect of what I want to talk about, which is financial stability and how evidence-based work that's a bit more open-minded can help improve our understanding of financial stability and integrate better with stress testing. And then, again, talk about the other big issue that central banks are concerned with, especially at the moment, which is the future of inflation and what drives inflation, is the ruff-lips curve and so on, and talk about some very recent work on modeling US inflation. So let me pay homage to be in with two intellectual giants who have influenced me since the 1970s. One is Joe Stiglitz. The other one is David Hendry. Joe's work on information economics had a profound effect on my thinking really throughout my professional life. He writes in his Nobel lecture that the attempts made to construct a new macro based on traditional microeconomics with its assumptions of well functioning markets was doomed to failure. And he particularly highlights the issue of people who lose income and aren't able to maintain their consumption because they don't have access to capital markets, at least in our good terms. And a second thing he says in his lecture is that information economics has alerted us to the fact that history matters. The path dependence is really important, something that Roger Farmer and I very much agree on. And dynamics are better described by evolutionary processes than by equilibrium. Now David Hendry is my other big influence and just give you two quotes from him. The false belief that database model selection is a subterfuge of scoundrels rather than the key to understanding the complexities of macroeconomies.  And then on economic forecasting where he's done really the leading work of anyone, understanding structural breaks, what you do when you're faced with a structural break, what you do afterwards is really, really important. So my brief comment about the New Canes in DSGE is well, it would have been nice. It wasn't new based on ideas made redundant by the asymmetric information revolution of Stiglitz and his colleagues. It wasn't Keynesian, it ignored coordination failures particularly between the real economy and finance and so useless for understanding financial stability. It wasn't dynamic enough because the leg structures implied by these models really don't fit the world at all. And it was hard to stochastic because stochastic is about statistical distributions. Radical uncertainty means in the time dimension the distribution is really quite different and cross-sectionary heterogeneity is missing. And hardly gender-equilibrium because it missed most of the system feedbacks that are important. And then the assumptions of rational expectations into separate optimization in the world that is subject to radical uncertainty and structural breaks has to be reformulated. So obviously repairs are necessary and many people agree and the ECB among others is leading in this work. Evidence to me is crucial. So there's lots of empirical evidence of course against the New Keynesian DSG model. The failure of aggregate consumption, the oral equation, something that Larry Christiana himself says is the most rejected equation in economics. The mountains of new micro-evidence on heterogeneity, credit constraints, buffer stock behavior, the influence of house prices on consumption in credit liberal economies, and of course the evidence against the New Keynesian Phillips curve both micro and macro. And then sadly in macro, evidence is seldom allowed to speak in the top journals. You have to go to the sub-journals, the specialist journals to allow macro evidence to show up. I think this is because of the pincer movement of the Lucas critique and the Sims critique, the incredible restrictions critique, which argue that we don't know what the theory is. Let's formulate a very, very general model and let the data speak that way. But of course, Bayesian BARs, which is what you have to, well, what the profession resorted to when faced with the curse of dimensionality are subject to compromise by strong priors and Bayesian restrictions. So let me turn to the positive side of my talk, the first positive side. So I want to talk about financial stability and think about it in a very general way and think about how empirical work can help appreciate the risks and understand how to integrate with stress testing. So I'd like to divide risks into three. So first of all, exogenous negative shocks, which for a small level of economy, really outside of the control of the policy makers. There could be a deterioration of the sense of trade. There could be a rising global interest rates. I mean, we haven't yet fully experienced the Trump shock, but who knows what that might do to global interest rates. There are all shocks, external credit supply shocks, physical risk, climate change, and a lot of things that small countries can't do anything about. A second thing they could do something about, which is the so-called fundamentals, things that look as part of the long-run solution, maybe fragile, I think of duration mismatch and credit supply, currency mismatch of debt, unsustainably weak financial regulation. These are all things that could have been avoided if the policy makers and the politicians had been on the wall. And then finally, the endogenous feedback loops, which can amplify risks, and the financial accelerator is what I'll talk about next. So let's think about positive and negative feedbacks. Let me start with the negative feedbacks, because obviously on the downside in a crisis, this is what really matters. So the buildup of debt, especially if the quality of lending has deteriorated ways on spending, a large expansion of the housing stock, which later weighs on prices. Well, this is something that varies across countries in the same way as the quality of lending is something that varies across countries and across time. A third negative feedback possibility is the increased saving for a down payment when house prices are income-wise. And it's really interesting that this negative feedback loop is quite high in Germany and France, but low in the UK and the US. And so that has implications for the stability of those economies. Now, quite a lot depends on the timing of negative feedbacks relative to the positive ones. So let me turn to the positive ones. First of all, extrapolated expectations of capital gains matter more when high leverage is possible. And that makes sense in first principles. It's also something my empirical work shows. Leverage amplifies gains. There's a leverage cycle. I expect the gains cause lenders to relax conditions because higher equity cushions make the loans look safer. Profitable end increase of the capital base. The third positive feedback loop occurs through residential investment, something we certainly saw in Ireland and Spain and the US. Booms that boost employment and income raising aggregate demand and feeding back. And then the fourth, the consumption channel. In countries with easy equity withdrawal, large collateral effects on consumption can occur, you know, think of the UK and the US. But time varying. In countries where equity withdrawal was not possible, that amplified mechanism is absent. So thinking about hedging net across countries and across time is really important. Of course, at the timing of the way these loops work. So here's a picture that my colleague, John Duca, designed showing how the household financial accelerator worked in the US. So we start off with the falling asset prices, the Norwegian housing crisis at the top, feeding into lower residential construction on the far left, weaker consumption second from the left. And then, of course, feeding back on the financial sector with bad loans rising, the ability of the financial sector to extend credit is very much reduced, spreads in the credit markets tighten, and this all feeds into lower GDP growth and feeds back onto lower asset prices, lower capital of financial firms and so on. I think most people recognize that story. So what are the implications for econometric models? Well, the FRB US model, which is still around, is not a DSG model and its example is being followed in many places where banks are now thinking about non-DSG models. It's good on expectations. I think it's very good on expectations, but I think it fails in other ways and there's some important lessons there. It imposes the net worth constraint on consumption. In other words, consumption depends on the aggregate of assets minus debt, lumping all assets together. So liquidity and credit shocks can affect consumption only through net worth, given income. So that means that debt necessarily has a trivial role relative to housing and stock market wealth. It missed the amplifying of feedback loops via the financial systems ability to extend credit. And in 2007 at Jackson Hole, Michigan gave a presentation showing the simulations of the Fed model, which showed that I think at 0.25% falling consumption relative to the baseline, so hardly a blip. It's got unstable parameters, the speed of adjustment in the consumption equation, which is the most important equation in the system, has almost halved in 10 years. And it's slow speed and inconsistent with focus and among central bankers about the real economy effects of monetary policy. So it's claimed in microfounded, but in my view it's not a structural equation in the more fundamental sense of the Cowles Commission. And the Fed model also lacks a decent residential investment equation. So how do we make progress? I think the encompassing principle is a really important principle for thinking about how to learn from data. So for applied work, we need to formulate models that encompass alternative theories. But allow the possibility that by imposing restrictions on this general model, a particular theory might be supported by the data. Something that David Henry and his co-authors proposed in the 1970s. Let me take the example of the Lifecycle Permanent Income Model and show how you can generalize it and learn from examining the implied restrictions, relaxing the implied restrictions. So here we have log of consumption is equal to a constant plus the log ratio of permanent income to current income plus the asset to income ratio plus log current income. That's the best approximation, the best log approximation of the story. Now you can generalize that. You can break net worth into different elements, liquid assets, debt, housing wealth, financial wealth. Most people think that cash is more spendable than pension wealth. And we know from theory that housing wealth is different from other wealth because housing is an assumption good as well. We can allow the coefficient on permanent income plus the current income different from one. Textbook says it's one. But if some consumers are myopic, it could be less. Moreover, the discount rate used to construct the discounted present value of future income should be far higher. That's what the microeconomics in the buffer stock theory says. Far higher than the conventional real interest rates that's used in textbook models. And then the intercept should be time varying. If down payment is required for mortgage, fall, saving for a down payment declines so that the consumption of income ratio can rise. And then because of shifting access to home equity, the coefficient on the house price effect or the housing collateral effect should be time varying. So we find for example, the new S and the UK was pretty much zero in the 1970s. And then with credit market liberalization, it rose very strongly. So what we need is a household equation system because of course, once you condition on these assets and debt and credit conditions, you need to estimate a system of equations so you can in terms of general equilibrium solve forward by working through the portfolio effects and then doginizing those. And in our work we extract credit conditions using a latent variable method. Credit conditions are latent variables from the same system. So in a ECB wooden paper on Germany and a new paper that's being published with Valerie Schodau, we estimate six equations for consumption, unsecured debt, mortgage debt, liquid assets, house prices and family income. And for France and Germany, we find very clear evidence of this dampening mechanism that I was talking about for the US. On the other hand, we find powerful evidence of this amplifying mechanism, time varying, which is physically strong in the upswing to the financial crisis and then very strong in the downswing. And just to show you that latent variables really do what they're supposed to do. The latent variable picks up everything that is not being explained by the rest of the model jointly in the set of equations. So in the house price equation, the mortgage stock equation, the consumption equation, it's a latent variable in all three equations. Now estimated from the French data, it turns out that it's incredibly close to the ratio, negative eight that is, to the ratio of non-performing loans relative to total bank loans. So the contraction of credit in the early 90s in France was very much connected with what was happening to the asset base of the banks and then the subsequent improvement and the new contraction later on. So, and of course that also then gives you the connection with the financial sector. You can connect what's happened in the banking sector with what's happened to loan conditions for the households and that's an important part of the whole story. Right, let me talk about the other topic that central banks are particularly concerned with which is inflation. Now in a paper that Janine R and I published in 2013, we forecast PCE inflation in the US. We looked it out of sample and performed it to different models and our key insights are that inflation is partly a process of relative price adjustment. Now that's not a new idea. Dennis Sargan actually, in his famous 1964 paper proposed exactly that and there's a great paper by David Hendry on the history of inflation in the UK with the same idea. What are the drivers? Well, unit labor costs, international prices and exchange rate and house prices are key elements of the modern solution. We find that including union density, a measure of labor market power, greatly improves the relevance of the unemployment rate. Now, because of the cost of dimensionality, such a problem in VARs, introducing a poor trade-off between the number of variables you can control for and the number of lags. We used a technique called posseminous longer lags, PLL, to give a better trade-off. The key intuition behind this is that the impulse response function becomes fuzzier as lags rise. In other words, the precise timing of shocks, a lag of 11 months or 13 months is not only precise, but there is an effect. So rather than ignore it, which is what the profession does, let's introduce it. And the idea here is to replace 24 unrestricted lag parameters by six. And that's shown in the slide here. We showed that every information set considered, PLL beats the Bayesian information criterion in terms of performance. And in our latest work, which is still ongoing, we find a fifth insight, which is that the pricing power affirms matters for price setting, along with the union power for wage setting. There's a great paper by Roulin and his colleagues showing what happened to the concentration ratio in the US industry back to the early 70s. And that measure is highly significant in our forecasting models, improves in-sample parameter stability and out-of-sample forecasting performance. So our model, the long-run drivers of core US inflation are union density, the unemployment rate, foreign prices, house prices, and the health and dollar concentration index. And under the compilation of union density, the unemployment seemed to do a good job of picking up unit labor costs. So the implication is that there is a, let's call it the Sargon Phillips Curve, where relative prices and equilibrium correction play an important role, is stable. And it turns out after three years of crisis, we're pretty much back to the model that there was before. Now here's a picture of what the model says about the impact of the concentration ratio, which is the red line, union density and the employment rate. You can see the explanation for one of the things that's puzzled central bankers, which is in the big, in the great recession, why did inflation not fall more? Well, our story is in the US. Part of it is because the concentration ratio rise rose. Most firms had more pricing power to offset the weakening power in the labor market. Well, stability is a big question. In our model, we can do recursive parameter estimates. And meaning from the top, we have the constant term, we have the integration term for import prices, then we have house prices, then the Huffendorf index and the unemployment rate, sorry, the unemployment rate and the union density. You can see recursively estimated from about 98, the parameters are really surprisingly constant. I mean, for an econometric model on inflation, I think that's rather good. So we conclude that the New Canes and Phillips curve is dead, but the Sargon Phillips curve is alive and well, given the right long-run controls, which include rise of price adjustment and including market power relationships. There are a few other insights that come from this as well. For example, the dynamics, I mean, you look at the details of the adjustment process, very consistent with what we know about from the exchange rate path through literature about pricing and local dollar markets. And for forecasting, once again, averaging of the best model with a relatively simple or progressive univariate model is improved robustness. So we can get a considerable improvement in the forecasting performance. Now here's a list of questions that come out of Janet Yellen's marvelous survey of issues and uncertainties connected with understanding inflation. Is there a stable relationship between unemployment and inflation? If it's stable, what's the lag? What measured labor market slack works best? Is the nairy or the natural rate a useful concept? Does the inclusion of private sector inflation forecast to improve the model? And is there a, well, and other questions, which I think we can all provide some answers to. So finally, to conclude, in macro heterogeneity rules, unemployment, for example, we know, is incredibly heterogeneous. It's one of the big inequalities in our society. But price-setting behavior is heterogeneous as well. And yet between these two heterogeneous objects, aggregations of heterogeneous objects, there is a relatively stable relationship. That's part of what macro is about. I made the point that profession often neglects longer lags. And our technique is a very simple method. Anybody can use anybody with a regression package can use to improve the trade-off between the range of variables considered and the range of lags. And then finally, I think the information economics revolution has highlighted the importance of credit and other liquidity constraints. And once you think about that, I mean, we know that the banking system has gone through a huge revolution. It's much more collateral-based than it was before. So once you acknowledge that credit is important, and that shifts in credit have taken place, surely you want to take into account the shifts in credit availability in the macro models, the policy models that you design to understand financial stability better. And this latent variable technique that we've been working with for some years, and it's quite helpful in making the link between the banking system, credit, and household behavior. Thank you. Thank you, John, for being exactly on time, which sets a very high standard for all next speakers. Where does the floor is yours? Thank you very much. So I'd like to echo John's thoughts about Vito and to thank him for including me. Like John, I've had many discussions or several discussions with Vito in which I would turn up in his office at four o'clock for a 15-minute discussion, and then an hour and a half we'd still be talking about economics. If I'd known that we could have also had a discussion about music, I'm sure we would still be there now. So let me say, to start out with, in 10 minutes, it's difficult to do justice to everything that's in John's very interesting paper. I'm gonna draw on what I take to be three themes. The first one that John did not say a huge amount about is that clearly some of the DSGE models we've been working with have not been particularly successful. Secondly, John draws and mentioned a couple of people that he's found very insightful. One is David Hendry and one is Joe Stiglitz, and I echo that. In fact, in my first job at the University of Toronto, I didn't really have a thesis, and then I went to see Joe Stiglitz give a talk, and my entire thesis ended up being inspired by that work, so I share the notion that there were some very important insights in what we call the information revolution in economics, and one of the things I'd like to talk about in this discussion is exactly what we can learn from that. My view is that what we learn is maybe even a little more radical than some of the things that John drew attention to. And finally, John provided some insights from his own work, his empirical work, and I'm going to, I hope, compliment that, agree with some of it and provide what I think are some important thoughts, particularly for policymaking, when we think about the relationship between inflation and unemployment. There are a couple of things, or three themes that I'll talk about. The first one I've alluded to, so exactly how should we think about introducing information theory. And there are really two ways, I think, in which it's important, and one's connected with the asset markets, and the other is connected with the labor market. And if you start thinking deeply about what we've learned about market failures, my view is that the lessons we should be taking away from information theory are a great deal beyond the idea that shocks become amplified. And in my own work, I've really tried to go back to what I take to be an important idea that was in Keynes's general theory, and which became forgotten. And that is that market economies are not self-correcting and can get stuck with unemployment rates, which could be 20% for decades or 5% for decades. And the way that we ought to think about that in the language of modern general equilibrium theory is that there are not just multiple equilibria, there's potentially a continuum of equilibria. And if you go away with that idea and you think about the progress we've made about the empirics of Phillips Curves, I'm gonna make the argument that there's a really important question we must ask ourselves because there's a huge amount of persistence in unemployment. In fact, I'll show you some evidence which suggests that it's difficult to distinguish the unemployment rate in the US from around and walk. And when you take that view, the question then becomes, is it on the supply side because some object that's useful called the natural rate of unemployment is moving around? Or is it on the demand side, in which case there's the potential for monetary policy not just to get us back to a better place more quickly, but to permanently influence the place that we're at? I'm gonna show you a little bit of evidence here from my own work on the connection between financial markets and labor markets. And what you're looking at here, the blue line is a measure of the real value of stock market wealth in the 1920s in the United States. The red line is the unemployment rate measured on the right axis on an inverted scale. So we're moving from zero to 30. And it's difficult to see an eyeballed chart which is as suggestive as that about a potential causal connection between the asset markets and the labor market. If you shoot ahead and you look at what happened in the most recent recession, the picture is similar. The magnitudes are not as great. We're moving from 4% to 12% rather than from zero to 30. But the notion of a causal mechanism between falls in wealth and increases in unemployment is I think a relatively easy one to take away from that chart. If you then ask, well, maybe those are special, this is a picture from 1945 up through 2011 showing you the connection in those same variables over that period of time. Again, the stock market is measured in real terms. The unemployment rate is, again, that's a transformation of the unemployment rate. And if you look at the time series properties of those two variables, you find that even though the unemployment rate is bounded, you can take a transformation of it that maps it into the real line. That transformation can be a random walk and looks like a random walk. The real value of asset markets looks like a random walk. Those two objects are co-integrated and the co-integrating relationship between them has been very, very stable over the entire post-war period.  And now, I'm gonna show you some toy models to help you think about what that ought to entail. So this is the kind of model that all of modern DSGE theory works with. It's what I call a rocking horse model. And it has the property that after a shock, the economy would return back to its growth path. And the dynamics of a rocking horse model are a vector auto-aggression in which there's a stable point or a stable growth path which the economy is converging back to. The alternative is what I call elsewhere a windy boat model. So the economy is like a boat on the ocean with a broken rudder. And if the economy is like that, this is a description of what is called hysteresis. So if there's a shock, as there is in this picture here, instead of returning back to the same growth path, this economy returns back to a different growth path. So the interesting question is, is the economy more like the rocking horse or is it more like the windy boat? In the windy boat example, the dynamics of hysteresis are that instead of there being a point that the economy returns to, there's a set. So think of that as many, many potential equilibrium unemployment rates. And this is what happened in the US data after the Great Recession. And in my view, that picture is a lot closer to the windy boat view than the rocking horse model. And there's a saying I learned to pick up in my adopted country in the United States.  The conclusion I take away from that is that, yes, there's a lot wrong with the kind of DSG models we've been looking at. Yes, we can learn from the information revolution, but the kinds of things we need to learn from the information revolution are likely to have more profound effects than the work of simple DSGE models. And it leads to a key question. And this is where John's work on the dynamics of unemployment and inflation come in. I personally, being quite critical of Phillips Curves, I've argued that they haven't really existed in data for really since Phillips wrote the first paper in 1958. On the other hand, on the other side, there are people like Bob Gordon. I'm not quite sure where John comes down, whether he comes down completely on the side of Gordon. Bob Gordon has been arguing that the Phillips Curve is alive and well, that you can estimate a stable Phillips Curve over almost all of the post-war period. But the way that he does that is by including the assumption that the natural rate of unemployment itself is a random walk. So the data has this non-stationary element in unemployment. The question we need to ask ourselves is that non-stationary movement, that very persistent movement in unemployment happening because of the structure of labor markets, which is causing natural rates of unemployment to go up, or is it due to some demand side variables that we can potentially influence through monetary policy? And they obviously have very different consequences for the way that we should think about operating, not just monetary, but also fiscal policy. So again, I enjoyed reading John's paper enormously. I've tried to give some complimentary ideas that I've talked about elsewhere, and I'm gonna end with a simple plug for where you might find them. And thank you.


Rhaid i'n gweithio i ymddangos ar y cyfnod y tîm yma o'r llwyto ar y rŵl y cwntain. Fy fyddwn i'n Llywodraeth Cymru, ac rwy'n gweithio ar gyfer y tîm sy'n gweithio'r cwntain ar gyfer Clav Andrew. Felly, fel ydych chi'n gweithio, rwy'n gweithio i'n gweithio'r ystod. Felly, rwy'n gweithio ar gyfer llwyto ar gyfer Llywodraeth Cymru, ac rwy'n gweithio ar gyfer Llywodraeth Cymru. Ond dweud â bydd wedi eu cael ymgwrthoedd gyda Llywodraeth Cymru yn aeth, mae'n gweithio'r prgynod lleonill yn ei dweud. Felly, mae'n gweithio, roeddwn i'n meddwl, chwy adnwyddoedd y neud hynny. Rwy'n gweithio'r teimlo'r wneud. N-rwt, rwy'n gweithio'r roedd, mae'n gweithio, n-rwyth, doedd yn ein teimlo. Sgwyddwch! Felly, rwy'n gweithio'r cwntain. Ychwanegon o'r gweithio ziw yw'r wych Poes HelCA oherwydd rydw i gyd yn hynau. Yna ceisio, dwi'n gweithio'n gweithio a'r bobl? Yn ni'n gweithio'n gweithio fel rhoi ar gyfer gweithio'n gweithio, rydych yn gymryd y cyffredinol ar y cyffredinol yn dweud. Felly mae'n gweithio'n gweithio fel roeddiol o cookl there. Yn eu cyffredinol yn gweithio'n gweithio, yyn ni'n gweithio'n gweithio'n gweithio gyffredinol. i'r ffair o'n rhoi ar scrapwyrn bach y cyflod. Byddwn ni wnaethaf â'n gwirionedd hon. Rwy'n meddwl o'n gwirionedd. Mae'n bach so fawr, na wnaethaf panndwch astu'r iawn.isticallynydd yr ysgolter i'r cwestiynau. Yn Ffair yw'r ffair, mae'n gyfly saidd o gyflyneddau o'r blwyddyn sy'n gwirionedd gyda fi Klawfandri. Actually, that's exactly why, just before we start talking about rootless containers, we first need to talk a little bit about containers. And then before we talk about containers, we first need to talk a little bit about Cloud Foundry. this talk we're mainly focusing on the cloud foundry seeks application runtime as opposed to the container runtime. But actually like a lot of the topics they still relevant across both. It's just that if we're referring to cloud foundry we're kind of specifically talking about the application runtime here. So what is cloud foundry and we're assuming that we're at least partially familiar with what this is given. Where we are but just to sort of set the scene cloud foundry is an open source platform as a service, so it allows a chymlu'n gwybod i ddim yn y gwleidydd, ac mae'n gwleidydd yn y ffordd i'r gennydd yw'r ffordd. A oherwydd mae'n gwybod i'r gwleidydd a chymlu'n gwybod i'r gwleidydd yn fawr, mae'n gwleidydd yn bwysig yn lleiwyddoedd, a oes i fynd i wneud yn gwleidydd. Mae'n gwleidydd yn gwybod i'r gwleidydd. Ac mae'n gwleidydd yn fawr, mae'n ddim yn nhw'n meddwl ffawr o'r cyfrifio. Dwi'n gwybod i'r ffordd i'r gwleidydd, nad yna gweledd i'r gwleidydd. platform, oherwydd bobl sydd wedi cael ei fod yn oed â omlwch. Y dywed y ddweud o gyffredd gyfroedd y hollredd a'r rem ychydigion y cyffredd Platform, sy'n oed o fewn adeilad ac yn gallu Yn'r Ddenbyll Cymru, Pan-Famol I, i gyffredd. Y dywed y ddweud o gyffredd gyffredd cyffredd dywed o bobl ar gyfer cynllun a'r dŵel mwrdd. Mae'r ddweud eisiau bod chi'n ceisio ar gyfer gweithio ymddiol, oherwydd o gweld a'r grifelau ac yn grifelau, into the cloud and away you go. And so we look at this and we think, like, that's pretty concerning, actually. It's actually like a worst case scenario, if you think about it, completely public, completely multi-tenant, and we allow you to push whatever you want. That's pretty terrifying. And so it means that we have to make sure we are on the absolute forefront and the bleeding edge of container security. We have to be prepared for the worst. So how do containers fit into this? What even is a container? Let's start with that. And the first thing to know about containers is that there's no such thing as containers.  What do I mean by this? Of course, containers are real. All of our applications are running in containers. So what I really mean is that containers are just Linux processes. There's nothing magic about them. They're just processes running on a VM somewhere. And the only thing that differentiates them from any normal process is the fact that these containers are isolated. They have layers of isolation around them so that they are isolated from each other as well as any other processes running on the VM. And it's that isolation that helps us here with the security aspect of this. So how does this work? Well, we use some kernel technologies, namely the Linux namespaces, which provide isolation of global system resources, so things like PIDs, mount points, et cetera, et cetera, as well as C groups, which provide limitation of physical resources, so things like CPU and memory. And there is a third one that we have to especially think about given Cloud Foundry's multi-tenant nature, which are disk quotas. So we need to make sure that applications aren't able to just write to disk and fill that up and provide an attack surface like that. But we're going to come back to that a bit later on. So that's kind of the intro. That's why we care about this. This is why we think it's important. What's coming up? So to begin with, we're going to take a bit of a deeper dive into exactly what makes a container a container. So we're going to look at namespaces, C groups, as well as dependencies. Then we're going to talk a little bit about some extra security layers that we apply on top of that to make sure our containers are as secure as they possibly can be. And then, finally, we'll get to rootless containers, what they are and how they fit into all of this. So, Claudia, what makes a container a container? Thank you, Ed. That's a very good question. Let's start with namespaces. So namespaces are a Linux kernel feature which allow you to isolate global system resources for a process. So these are things like mount points network devices and process IDs, et cetera, as Etta said. It's basically about creating illusion. When you put a process inside a namespace, you can make it think it has exclusive access and sole control of those resources. So there are seven namespaces in Linux right now. These are Mount IPC, PID, NET, UTS, C group and user. And this last one, the username space, is particularly relevant when it comes to rootless containers. So we're going to come back to that in much more detail later. But for now, let's just pick a couple which are used to create just containers in general. And those are the PID and mount namespaces. So let's have a look at how they work. Let me move my mouse. There we go. So it's up with the PID namespace. So with the PID namespace, two processes running on the same VM can have the same process ID in different namespaces. It's what it appears to be for the process inside the namespace when in reality that process is then mapped to a different PID outside these namespaces.  So as you know, when you boot up your machine PID one, it is the first process that runs and does everything and all the other processes are then children, grandchildren, et cetera, of that process. So it's the big dog, it's God. It's the God process. But the parent namespace knows that these are mapped to other processes inside the host. The child namespaces have no idea they have a parent, none at all, because PID namespaces are top down. The parent has clear view down into its children and its grandchildren and everything. The child namespaces can see its children, yes, but it doesn't know it has a parent. It doesn't know it has siblings. It just has only visible control over its own given world now. All right, so the next one we're going to have a look at is the mountain namespace. So processes which have been put in a new mount space have a different view of the mount table. When we use pivot root at the same time, which I'm going to come back to in a minute or two, that process will see something different mounted as slash. So mountain is root fs. You'll notice here that I've done df minus m print type to see the file system mounts. I just did that because it gives you a pretty output for slides. In reality, you probably do like at prog PID mount info or mount or anything like that to see the mount table. If you want to see more about what namespace your PID is running in, you would do your ls minus l, prog PID ns, and those links would give you the namespace IDs for the namespaces that your process is running in. All right. So if namespaces alter a process's view of the system and its resources, then C groups can actually be used to enforce certain power over physical resources. As I already mentioned, those are things like memory and CPU. Control groups are again the next kernel feature. And so every time we create a container, we will put that container process inside a C group and then apply limits to that C group so that it can't access more CPU than it should to ruin the experience of other people. As I've said, the other thing we care about limiting is discreta. And this caused us a lot of problems. So we're going to come back to that again in more detail in a minute. So the last thing that makes a container thing, we're all very familiar with and we love, and different from a standard Linux container, is the encapsulation of dependencies. So a few years ago, Docker figured out a way to encapsulate dependencies into shippable units within things called images. And they figured out how to do this efficiently. These images could then be moved around between machines and this meant that we could run many different containers based on these identical blueprints, which underneath we're using the same Linux primitives and mostly we're sharing dependencies. And this is the key difference between Linux containers, which have existed for a while, and the containers which we talk about today.  Container. So let's look at how dependencies and encapsulation works. To do this, we use a layered file system and pivot root, which is what I mentioned earlier when we were talking about the mountaining space. Let's start with pivot root. So pivot root is a system called, which changes what a process sees when it looks at its root directory. So instead of seeing the host file hierarchy, we can tell the container process to see somewhere else as its root file system. So say for example, you had a program called run.show. It asks a question, what's in my root of s? And what comes back is whatever is on the host. Something boring, because it's not a container, it contains a call. But then the next time it runs, what's in slash, sorry, everyone has to talk, slash we can do pivot root with a path to somewhere else. And this time what comes back is something much more exciting. Busy box. It's a container root file system, much more exciting. But we can't just point to any old path. Actually we could, but it wouldn't be very useful unless it actually does what a root file system is supposed to do. So we have to give it an actual root file system. And we do that with layered file systems. So in the past. Could you speak a little bit slower? Yes, of course, very sorry. This is your job. Team members, this is your job. Shout, you know this. I'm very sorry. I do, sorry. All right. All right. So it's fine. I'm trying now to count seconds in my head. So this is where layered file systems come in. So in the past, the way that we, by we I mean the community as a whole, used to handled dependencies was we would package everything up into an AMI or a VMDK or whatever and that was fine. If you didn't care about things like boot speed and filling up your disk with junk. So imagine you had a container which needed Ubuntu as part of its dependency package. So that container would come along, you would download Ubuntu, cool. Another container comes along and they're like, oh, hi, I also want Ubuntu. So you download that Ubuntu again and then a third container comes along and you kind of see how this spiles out of control because Ubuntu is not small. A lot of times dependencies are quite large. And this is where Docker really saved us a lot of pain. They realized this thing called layered file systems which is not a new concept. Could be used to get containers, their dependencies. So how does this work? Another awesome diagram. If you picture standard Docker files, the first line begins with from something we've taken Ubuntu. When a container's orchestrator reads that line, it will go ahead and download the contents of Ubuntu into a read only directory somewhere on the host file system. The next container comes along and it also wants Ubuntu. But this time, because the orchestrator already knows that we have one of those, it doesn't do it. It doesn't need to download it again. It can just point to it and say use this one too. So now they're sharing. We've already saved time and space. Doctor Who reference. And the next line, they run at update. Again, only one of them needs to do this. Whichever one gets there first will download the things that are required in that run at update. And whoever gets there second will just use the one that's already been downloaded. This third line, now they're doing something different. But this is still the first time that that second container created called has actually had to download anything. Until then, we've been sharing, we've been saving space on our disk. Finally, these are all then mounted into a read-write layer which becomes the container's root file system visible thanks to Pivot Root. So here we can see how layered file systems and Pivot Root allow us to share dependencies and stop container processes from interfering with each other's root file systems while they are running. Alrighty. All right. So at this point, we might be thinking, okay, we've got some pretty awesome isolation in place. We've got our C groups in place. We've got our dependencies sorted. We should be pretty good, right? Like, what else do we need to do? And of course, like, nothing's ever that easy. And so now we kind of need to think about, okay, well, we've got our isolation, but what happens like if or more likely when somebody breaks out of that isolation? What happens then? And what can we do to protect ourselves in that instance? And so to that, I say security onion. So of course, this is about applying layers. So there's some additional layers of security that we can apply to these containerized processes in order to protect ourselves here. So namely, I'm talking about capability dropping, set comp, and app armor. Let's just quickly run through those now. So capabilities. So historically on Linux systems, privileges tend to be divided into two. You had the all-powerful root user, UID0, who had the privileges to do absolutely everything. And then there was every other user on the system, so all of the unprivileged users, and they weren't allowed to do anything at all. And if an unprivileged user wanted to do some privileged operation, like do amount, do a pivot root, they had to sudo su up to the root user, at which point then they've gained every single privilege available. So it's not great. These days, those privileges have been split up into little chunks known as capabilities. Here are some examples. So we've got, for example, CapChone. So historically, if you wanted to chone a file to another user, you had to be the root user. But now that these capabilities have been split up, it means that actually any process is now has the privileges to do this as long as it has that CapChone capability. Similarly for CapSetUID. And there's about 40 of these in total. And you can see that that gives us a much nicer separation of concerns there. But there is one that we need to talk about, which is CapSysAdmin. And this has kind of become like a catch-all for any of the privileges that don't fit into any of the other capabilities that we have today. And so as a result, it's really overpowered and really to the point where if you have CapSysAdmin, you are still essentially the root user. And this is going to be really important later on when we start to talk about the rootless containers. So keep that in mind. But the reason we're mentioning this now is because it's actually possible to remove capabilities from our processes. So what this means is that when we've got our containerized processes that are running, we can actually remove CapSysAdmin as well as all the other capabilities that we want to. And it means that if that containerized process is able to break out of the isolation, we've actually limited the amount of damage it can do because it doesn't have the capability to do anything. So next up, we have SecComp. SecComp stands for Secure Computing Mode. And this is pretty awesome. SecComp basically allows us to limit the number of SysCools that our processes can make. So this is great. So if you think about it, like all of the security here, it's kind of relying on the surface area of the kernel. And the kernel's a pretty big thing. And so by giving, well, the fact that we are able to remove some of the SysCools from our containers or the containerized processes, that's awesome because it means that we are dramatically reducing the surface area for an attack there. And it means that, yeah, any vulnerabilities in any of those SysCools, that's no longer a concern for us because the containers don't have the permission to run that SysCool anyway. Finally then, we have AppArmer. So AppArmer allows us to restrict these containerized processes even further. So AppArmer is a form of mandatory access control, kind of like SE Linux. And the way this works is you create yourself an AppArmer profile, which contains a bunch of rules. So there's an example rule there. Deny at proc slash star w, which means that when applied, this process is not allowed to write to any file inside the proc file system. For example. And so we build up these profiles and we apply it to the containerized processes. And again, even then if they're able to break out of the isolation, we're limiting the damage that can actually be done in that situation. And so we have the security onion, capability dropping, set comp, and AppArmer. It's probably worth mentioning that in order to apply all this lot, you do typically need to have like Capsis admin and those root level privileges, which is going to be important later on. But the other thing, just want to quickly mention is we've actually covered loads of stuff, right? We've seen like namespaces, C groups, set comp, dependencies, AppArmer, like all this stuff. And it's actually really, really tricky stuff. It's quite low level sort of working around the kernel. And historically that fell to the garden team to program around that and to be responsible for that. But fortunately with the introduction of standards a couple of years ago, so particularly the OCI standard, this is now no longer such an issue for us. And in particular a tool called RunSea, which is just a very small binary tool that's published by the OCI. It basically abstracts away all of that hard work for us. And it means that we as garden can just cool down to the binary and it does all of that complexity for us. So it's been absolutely invaluable. I think it would be hard to overstate how good this has been for security in general. So thank you to RunSea, cheers. Okay, so at this point, as I said, we've covered the namespaces, the C groups, isolations, set comp, AppArmer, all this stuff like surely, surely, surely now we must be secure enough, right? So yeah, sure. So far we have suffered no major exploits. There is no word here, I can't touch word, I'm really sorry, sorry Molly. But it doesn't mean that we're invulnerable, not yet. If you picture the various exit points from a container as a door, then yes, we've made a really solid door. But how likely are the attackers to stop there? Pretty unlikely. Once you've sealed off the door, they're gonna step back, they're gonna look for the windows, which means that while we think we've done this, if we step back, what we've actually got is that. So we've secured the containers, but now we need to make sure that we secure everything else. We need to secure the things that are managing and running those containers. We need to secure garden itself. We've garden to not run as root. Right now, garden and all the other components, we're all running with mad privilege right now because we've had to be in the past. But things have changed, and now there's all sorts of primatism tricks that we can use to get to this. Yeah, so this is where rootless containers finally come into play. So the nice thing about rootless containers is it means we can actually create all of those containers without needing to be root ourselves. And the key trick here is username spaces. So we kind of touched on these earlier. So in the same way as we saw with the PID namespace, which allowed the PID of a process inside a container to be different from the perspective inside the container as compared to outside the container, such as the case also with the username space. But with the username space, we're talking about UIDs rather than PIDs. And this is really great because it means that inside the container, a process can appear to be running with a UID of zero or root. But actually from the perspective of the host and all of the other containers, it's just running as some completely random unprivileged user. And what's even more awesome about this is that actually any unprivileged user has the privileges to create a new username space. And so I'm going to put this up here in sort of giant letters because this is kind of the key point of the whole talk here. Any non-root user can create a new username space in which they are the root user. And once you are then the root user within that username space, you have capsis admin, you have the privileges that you need to go and set up the rest of the container. So that's pretty awesome. But username spaces can be a bit confusing. So I've got an analogy here. Hopefully this will help you. So let's imagine that in the host, we are an average frustrated user. We have no privileges. Our UID does not equal to zero. But in the container, hey, we're Brad Pitt. We are the root user. We have all privileges. Our UID is zero. And yeah, so this means that we've got that capsis admin. It means we can go and set up and create the rest of the container with all of those layers of security that we've just been through exactly the same as before, which is great. But the key thing here is that it's only, that those root level privileges are only applicable with inside that username space. So outside the username space, i.e. in the host and in the other containers, you're just an unprivileged user. Which is pretty awesome. So how does this work? Well, there's a map, a UID map. So as we can see there, we are basically mapping UID zero inside the container to UID 429 blah blah blah inside the host with a length of one. And that UID there, that's basically the maximum UID that's available on the Linux systems. We chose this because it's the one least likely to have any privileges or to clash with any other sort of UIDs. But there's a problem. And the problem is that by default, you're only allowed to map one single UID. So let's take a look at what this might look like. So let's assume that in the host, let's say we are unprivileged user UID 1000, Alice. And we say, okay, I'm going to create a new user namespace and I'm going to map myself to UID zero, the root user. That's great. But what about all of the other UIDs inside the container? We haven't applied a mapping for those. And so it means that if you were to look in it, look inside it, they would just be unknown and they wouldn't be allowed to do anything. And obviously, users expect UIDs to work. So how do we get around this? Well, fortunately, there is a solution, new UID map. And this is basically a small set UID binary. So set UID is basically, it allows you to run a binary with the privileges of the owner of the binary rather than the currently running user. And so if that binary is shown as the root user, it gives us a way to temporarily elevate our privileges, write the entire UID map, and then drop that down. So it's kind of cheating. It's not like 100% rule-less, but we kind of think it's okay. New UID map is a very standard binary that you'll find on most distributions. What about rid of s mounts? Yeah, so earlier I talked about how encapsulation is the thing which makes containers nice and portable and stops their dependencies from flooding your disk. And this is done through file system layering. Well, unfortunately, doing a root-first mount often requires root. So that was our next target when we wanted to become completely rule-less. Just quickly clarify by what we mean when I talk about our file system. There are two meanings associated with this word. One is the structure, just the files and directories, what you see when you go on and do LS and move around. The second meaning is the way that data is managed within that structure, and that is done by a file system type. So today I'm going to be talking about the second one, how that data is managed. So let's start in the beginning, back in the day when Garden was young and we weren't even called Garden. We used AUFS to create those root file system mounts for our containers. And we used AUFS for ages, and it was a headache. But we stuck with it because nothing else was mature enough for our needs. Well, either that or it was under proprietary licence and we couldn't use it. Our main problems with AUFS were that it was not in the mainline kernel and compiling that module went wrong more often than it went right. It was maintained by precisely one person, which meant asking for patches. It was not quick. And you have to be root to do an AUFS mount. But we were stuck with it until two years ago. Another option became very attractive. And because there was this huge push for us to get all of our components rootless, we created an entire team around our new root file system creator. And we called it GrootFS, which is a terrible name because it implies we created a new type of file system which we did not. We created something which downloaded the stuff and then gave you a directory at the end of it, which is still cool, a lot of cool stuff going on, but we did not make a file system anyway. We chose BetterFS to actually buy our new file system type. And it looked good to us because it had this quota tooling built in, which, as we know for Cloud Foundry, is really essential, multi-talented. And aside from needing a bit of privilege for setup, which can be done before we start our container server, it could snapshot our root file systems as an unprivileged user. And so we were very excited about this. But it turned out that that quota tooling we were so excited about ended up not really being very good in production. It could not stand up under load and the whole system ground to a very unpleasant halt. So we had to quickly turn it around and we chose our next favourite file system, which was OverlayFS, which is very similar to AUFS in that it performs a union mount, which basically means it takes the contents of several directories and make it appear as if it's under just one directory. The big difference is between AUFS and Overlay is that you can do mounts as an unprivileged user on Ubuntu. So Kononky Ubuntu had a chat with the OverlayFS maintainers and decided it was not a security risk to perform mounts as an unprivileged user, because not many, I don't think any, apart from is it ShiftFS does or soon does mounts as an unprivileged user. But Ubuntu now compiles a patch version of its kernel with the OverlayFS system white listed for that action. So it's very cool. It does tie us down a little bit, but it does get us by on our quest for rootless containers. We still use Grutifes to download all those layers and to chone all the directories to be owned by our Maximus user. But Grutifes will not do that mount. Instead, it will return all the information needed to do that mount. To RunC, in the form of the, is it the OCI spec, then RunC will perform those mounts in order. So we just need to make sure that the Grutifes mount is at the top of that list, that array, and RunC will get that to those details. And inside the username space, where it has all the correct powers to do everything it needs to do, it will mount our root file system for us as an unprivileged user. So it's very cool. So to sum up, we are running Garden rootless. We've got user NS and mappings. We have got Grutifes running rootless, which is creating a rootlessly owned of Grutifes. Are we now? Good. Not entirely. So there's still a couple of roadblocks left. I'm going to quickly run through these. Just aware of the time. So the first one is disk quotas. So we mentioned this earlier. We need to apply disk quotas to our applications to stop them from filling out the disk. And the way that we do this is with XFS. Fundamentally, XFS disk quotas require root-level privileges to apply. So the way that we worked around this was to just extract the tiny piece of code that was actually responsible for applying these quotas into a tiny binary. And then again, we applied that setUID bit to it. So very similar to how newUID map works. Next up was networking. This is kind of like elephant in the room. We haven't talked at all about this. And that's because, again, it requires root-level privileges. So what do we do? Well, we extracted the piece of code that does the networking and we applied the setUID binary to it. SetUID bit, sorry. You may be noticing the theme. Just quickly mentioned there is some work on going here. That's Slurp for NetoNest tool, which has got a great name. This is like user space networking. You can do that entirely unprivileged, but it's really slow so we couldn't use it for production. And then finally, there's C-group churning. So by default, C-groups are exposed as a virtual file system. And if you want to apply a limit, you have to echo some values into the C-group files. And the problem is that those files are owned by the host-root. So our container root doesn't have permission to do that. So to get around this, we actually have like a very small privileged setup phase that goes and tones the C-groups to our container root user. And this is sort of okay because that happens before any container is ever created. So there's no risk of a user interfering with that process at all. Are we production ready? Well, Garden has been sort of production ready for a while now, but of course, Cloud Foundry is a big thing. There are lots of other teams that need to align before we can get out there. So no, we're not production yet. We really hoped that by the time we got up here, we'd be able to say that we are so close. We can see it. It's painful to think how close we were being able to say that we were in production, but we're not yet. But hopefully this time next year, we'll be back or someone on the team will be back to give you the latest updates on all the fun new bugs that have come out from running rootless in production. It's going to happen. So where time is going to do this all at once.  If you want to try rootless yourself, if you're deploying by SCF, just turn on the experimental rootless mode on your garden job, and then all your application containers will be running rootless. If you want to run the binary, just on Ubuntu stem cell, just to play around, you can create some rootless containers there. And the last one there is a link to the rootless containers roadmap, which is maintained by Alexa Sarai, who is a contributor to Run C that component we talked very highly of earlier. So that's just how rootless is going, generally in the community, which is very important for us because we work alongside the community. And I just actually do want to shout out to people in the community who have done so much work with rootless containers. Without them, we would probably have nothing to even talk about here today. So Jess Frizel, Lexus Rai, Acahyrus Suda, and many, many others, thank you so much for your work. And that's done, just under the wire. Any questions? Thank you. Do you have a 30-minute break after this talk, I believe? So we can take some questions if people want to stick around. Not 30 minutes of questions, hopefully. But I'll pass you the mic if you have any questions. Going once, okay? I think I'll have one. Thank you for the presentation. And did you consider ZFS instead? Yes, we did. My understanding is that the licensing around ZFS is a little complicated. And so we kind of got scared off from it. But you're right, I think you're right, actually. I think ZFS, you are able to do the disquadring unprivileged as well. But yeah, we can really consider it for production in the Cloud Foundry case because of that licensing. Anybody else? This will be a quick one. I'm curious about how to test these kinds of changes. So it sounds like a lot of these implementations come from like a theoretical kind of observation of what it would mean to be rootless. But are there any tests or automation suites to try to break out or try to really exercise and be really mischievous to your apps? Yeah, so I think there's a couple of parts to that. So first of all, the vast majority of the work here is actually contained inside RunC itself. So again, we get a lot of this for free. We've actually been helping out with some of the rootless stuff. So a year or so ago, we were the ones, well, we were helping out to get the features that we needed to make it work for Cloud Foundry. And as part of that, there was some testing stuff that we added into there. So I think RunC has a set of tests. Garden also has a pretty awesome set of security tests, integration tests as well. And yeah, we spent a lot of time talking about this and just any idea on the wall, like what we wanted to make sure was that we weren't making the security worse accidentally. Which was a possibility, right? But yeah, I think we got the RunC tests, the garden tests, and we're pretty sure it's pretty secure. If anyone could spot anything that we haven't noticed, please come and talk to us. But yeah, I think we've pretty much got it covered. The username spacing seems to provide a way to inside containers of different users and maybe add some restrictions. Can you clarify whether it's already possible for Cloud Foundry developers to set those restrictions to create new users and to set those? And if not, is it something that could be envisioned as a future? So I think in Cloud Foundry, I didn't quite hear the question, but I think I'm going to answer it. Let me know if it's not. So I think when you're pushing apps to Cloud Foundry, like we are running them as a very specific user inside that container. And so we have ensured that the UID mapping for that user exists. We also, with regards to the UID range, so we've got the first mapping, which is the root user back to Maximus, but then for every other UID, we actually start the range at like 65,000 or something in the host. Above nobody. Yeah, above nobody basically. So the idea there is that hopefully, it's not going to clash with any of the more common users, UIDs that you might be running on a system. So a common use case is, for example, in PHP to run the PHP interpreter with low privilege, so that if ever the app gets compromised, it has limited capability. For example, it would not be able to make network calls or restrict the system calls it would be making. And so currently, I understand the Diego is creating VCAP user. So you're saying I could have some user provided script, such as a provided script to create new users and to restrict the way my app is running. Would that seems possible? So I think it depends. If you're just running a buildparks-based application, there's no chance of customising that.   I'm not sure I'm totally hearing the question. Maybe we could chat afterwards and now. Thank you so much. I'll try to clarify. Thank you. Anyone else? Cool, thank you very much.