So, I've got a question to Luke because I'm really in your hands. Can I start earlier or do I have to wait until 4.45 sharp? Okay, so let me open this panel on exiting non-standard monetary policy measures. That's a fascinating topic. It's a very timely discussion, so I will be listening very carefully and taking notes. One point I wanted to make before we jump in the discussion is about the gender balance on this panel. This is an all-male panel. We've tried hard with Luke and with the colleagues. But for some reasons I'm not going to explain here, it just didn't work out. But it's something that we certainly have to improve in the future because it's certainly not our policy to have an all-male for that matter all female panels here at the ECB. So I just wanted to make the point. Second point, which is housekeeping. We have one hour and a half, which is plenty of time. As suggested by Luke, we can end a little bit earlier if needed, so don't feel any pressure either way. And I'm still going to keep you to the timeline because we want to have enough time for the discussion with the audience. So with that, let me start and thank very much the speakers for being here. I'm not going to do the introductions because they are all very well known. And we're going to have a variety of angles into that discussion. Jeremy, as I understand, will discuss issues related to the liability side of the balance sheet of the central banks. Hewn will discuss some of the more market-related issues related to the impact of large-scale asset purchases in the market and investor behavior. And Olivier will wrap it up and come back, I guess, to the core monetary policy discussion, which is how should we exit? Also, drawing on what Hewn and Jeremy would have said. So without due delay, Jeremy is off the floor. Okay. Well, thanks very much. Thanks, Benoît. Thanks to Luke for having us here. So as Benoît said, I thought I would try to talk about the liability side and I'm going to focus particularly on the Fed and focus on that. And you might think that that's a little bit curious because obviously most of the focus in the QE era has been on the asset side of central bank balance sheets. And in particular, you know, how using the asset side can act as a substitute for monetary accommodation when you're stuck at the zero lower bound.  And so the Fed and other central banks, to the extent that they've been able to, have tried to buy longer duration assets, for example, because you get more push on the term premium that way, the Fed decided to buy mortgages because they wanted to get just a little bit more bang for their buck altogether. And there was some thought that, you know, working on mortgage spreads was going to be helpful. So that was kind of the mindset was really on that.  What I want to do is point out what I think is essentially an important, I don't know if it's a coincidence, it's quite the right word. But it's, yeah, it's kind of a coincidence, which is that the QE era, you know, it was a monetary policy effort. But it happened to coincide with fairly dramatic changes in financial markets and in regulation in particular. Okay. And the point I want to make is that some of these regulations, and I'm thinking particularly of things like the liquidity coverage ratio, margin requirements, central clearing have, were essentially creating a much increased demand for various safe assets. Okay. And then you might have wondered, well, geez, wouldn't that have put a strain on the supply of safe assets? And why didn't we see sort of scarcity? And why didn't we see the system creaking in various ways? Well, it might be. And this is just my conjecture that coincidentally, the Fed, by massively increasing reserves, was creating at least a certain kind of safe asset that was actually very helpful. So things went smoothly basically because of this almost inadvertent supply effect that was just the sort of balance sheet mirror image of QE. So my point is now, as we start to unwind, one ball that you want to keep your eye on is what's happening there, and will we see some of the strains that were somehow papered over finally starting to emerge?  On the asset side, very simply, think of that as being about two and a half trillion of treasuries, 1.8 trillion of mortgage-backed securities, and some dogs and cats. Less attention, as I said, to the liability side. That's about 1.6 trillion of currency, 2.3 trillion of reserves, and again, some other stuff like treasury deposits and a smaller repo program. Okay, but again, 2.3 trillion of reserves, and to a first approximation, that number is going to change by a lot. So interestingly, and I think actually very intelligently, while the Fed has been quite precise about the near-term trajectory, they have not called out the end point yet. I think that's smart for reasons I'll sort of get to. But, you know, the kind of numbers that people talk about are, you know, it could shrink by 1.5 to 2 trillion, the balance sheet. So, and that would be essentially those 2.3 trillion of reserves coming down by that very substantial amount. Okay, so why would that matter? Why would that matter apart from what's going on in the asset side? To be clear, I think it only matters, if you want to convince yourself that it might matter, you have to start out by believing that there's not only a well-defined, you know, there's a lot of work, a lot of research on safe asset demand. You have to go a little further. You have to believe that there's a sort of segmented safe asset demand so that there is a particular demand for short-term safe assets. Because if you think about what's going on, the aggregate supply of safe assets available to the non-fed public is not going to go down because as the reserves are going down, the, you know, treasuries in the hands of the public are going up. Okay, so total sort of government-provided safe assets are not going down, but of course the short maturity stuff, the reserves are going to go down. So, you know, premise number one or sort of friction number one that you have to buy into is that there's a special demand for this stuff. Empirically, that seems to be well-founded. We've done some earlier work and Arvind has done some earlier work. And if you just look at, you know, variation in things like the supply of T-bills to GDP, when there are fewer T-bills in the system, the price, if you will, goes up. That is to say the slope at the very front end of the yield curve gets steeper. So first observation I want you to keep in mind, the yield curve unconditionally, pre-crisis, so think of sort of 1980 to 2009, is astonishingly steep at the front end. So the average difference between a one-week T-bill and a six-month T-bill is somewhere between 40 and 60 basis points. That may not sound like a lot. That's a lot relative to the risk. Okay, so that's the sort of unconditional number and it moves around in response to variation in the supply of T-bills. So when there are more T-bills, when there are more T-bills, it flattens out. When there are fewer, it steepens, okay?  You can be a little clever and you can instrument, for example, with the tax calendar. So there's seasonal variation in T-bills outstanding because the government needs to borrow more before they get the tax receipts and then when they get the tax receipts, they borrow less. And that traces out also this variation in the slope of the yield curve. So there's some evidence to suggest that there's a demand not only for safe stuff but for short-term safe stuff. Think about money market funds, for example. What do they want? They don't want to hold long-term government bonds. They want to hold short-term stuff. If you believe that, if you believe that that's sort of a somewhat segmented market, then the magnitude of the shrinkage that we're going to get is quantitatively a big deal. So as I said, reserves are going to go down by one and a half to two trillion if they go all the way. To benchmark that, the quantity of what are other sort of short-term safe things, T-bills of maturities less than 30 days, it varies. But think of like 400 billion as being the number, okay? So if you think of the aggregate supply of short-term government safe stuff, you're going to get a huge impact or potentially huge impact in the shrinkage. All right. So let me be a little bit more concrete now because I'm sort of very loosely talking about T-bills and reserves like it's kind of all the same thing. That's not quite right. I mean, now that there's interest on reserves, it's sort of kind of the same thing. But of course, reserves are kind of like restricted T-bills because only banks can earn the interest. So let me be a little bit more specific. Who's holding those 2.3 trillion of reserves right now? Obviously, it's all banks because they're the only ones who can get the interest. What kind of banks it seems like the best data I can get, the first several hundred billion are probably in foreign banks who are doing what's called the IOR arbitrage. They're basically borrowing in the funds market and turning around and investing it with the Fed and earning a spread that right now is about nine or 10 basis points. They're just doing it because they're sort of advantaged in doing that arbitrage. And presumably, they're pretty elastic. So as the supply of reserves goes down, rates will only have to move a little bit for that to be un-tracted for them to do. So my best guess is the first several hundred, first let's call it 500 billion will roll off without any real big deal. So the first year or so of the balance sheet shrink, I think, basically these foreign banks just do a little bit less, no big deal. At that point, I think it at least gets interesting. You know, I don't know what's going to happen, but here's where the regulatory induced demand, I think, is going to be potentially relevant. So the other big category of if you just look at who holds reserves are not foreign banks who are just doing arbitrage, but they're the big U.S. banks, think of J.P. Morgan in particular, who do this to comply with this new regulation, and the liquidity coverage ratio.  Reserves count, Treasury securities count, some other stuff kind of sort of counts. Okay, the overall demand on the part of the big G-SIBs, on the part of the big banks, estimated to be about 2.7 trillion. Of that, about one trillion of what counts as HQLA seems to be held by the big U.S. banks in the form of reserves. So they are using reserves quite heavily to make up this demand. Okay, so they play a substantial role. Now, here's the tricky question. This is the elasticity of substitution that I do not know what it is. Okay, so in other words, one theory of the world would be, well, reserves count as high-quality liquid assets, so do Treasuries. When the relative supplies of these two switch, the JP mortars say, that's fine, I don't really need to hold reserves, I'm happy to hold 10-year Treasuries. Okay, in which case, the whole thing is like a non-event. Or, there could be a reason why they're holding it all as reserves, that they actually have a reasonably well-defined preference for holding it as reserves rather than as 10-year Treasuries. They don't want the interest rate risk. They want the intraday liquidity, essentially. I mean, it's very related actually to some of the stuff that Monica was talking about, that reserves provide a function, especially in this sort of HQLA world, that is not perfectly substituted by bonds. So, if that is the case, they may be a little bit more reluctant to give it up, and then you've got something of an effect. Okay, and again, a little bit more broadly, even if the banks say they only have a preference, not over Monica's sort of liquidity thing, but just over maturity. So, suppose a bank says, fine, I lose my reserves, I'm totally happy to hold like 30-day T-bills. Okay, but then those 30-day T-bills have to come from somebody. They tend to take it away from the money market funds, and then all you need to believe is that there's a segmented market between sort of short-term bills and long-term stuff, and again, you get something. So, again, I don't pretend to know, but I think just based on sort of previous evidence, I wouldn't be surprised if you got something of a price effect. Okay, so what's the big deal? So, you get a price effect. You know, again, what would it manifest as it would be something like the yield curve, which is currently quite flat over the first, you know, six months or so, even if it just reverts back to its sort of normal slope, or it might get even a little bit steeper than that. So, again, think of there being, you know, 60 basis points of slope between one week and six months. So what, the yield curve's always kind of looked a little bit like that. Now, one simple point, I think the less interesting thing, is it makes the regulation more expensive for the banks to comply with, because now they're going to be holding all the short-term stuff that has a very low yield, so they may feel, well, that's kind of not nice and unfair, and they'll, you know, they hadn't complained as much about the liquidity coverage ratio. They may complain about it more, you know, that's one thing. To my mind, the much more important thing is think about what this does. If the yield curve does slope up like this, you know, 60 basis points over the first six months, think about what this does to the incentives for private sector firms to do maturity transformation. So I think, you know, and this is work I have with Rob and Greenwood and Sam Hansen, we think sort of an underappreciated aspect of the pre-crisis period, you know. We know that the story is too much maturity transformation by the private sector, right? All these guys, Lehman and Morgan Stanley, all funding themselves short-term, we've passed a lot of rules telling them, don't do that anymore, it's bad. It has externalities and you shouldn't do it. But, you know, you've got to think about the economic incentive. That's a very high-sharp ratio. Okay? Again, 60 basis points, but it's 60 basis points against not a whole lot of duration risk. So if you're a private sector firm and you're choosing between borrowing at the six-month point and at the one-week point, you save a lot of money. And so one reason to not want to let that reemerge is I think then it makes it much harder. In other words, we've passed a bunch of regulations since the crisis and guys have apparently not been funding as aggressively at the short end since the crisis.  I think one reason it may be working is it hasn't had to fight in the teeth of an incentive that was as strong as it was at some time prior to the crisis. So, you know, I think that's something that should be monitored and it's not to say that you shouldn't be doing the best you can with regulation, but if you make it too hard for regulation, if you make the economic incentive sort of overwhelmingly strong, I think it's tougher. So, you know, what should the, what can you do? I mean, I think the Fed has wisely not committed to an endpoint and I think a sensible thing is, you know, let's watch, let's see, this could be wrong. Maybe it's not a big deal, but if you start seeing signs of either the yield curve getting steep or various spreads that are suggestive like T-bills to OIS, you know, widening, you know, you might want to either be willing to slow down the rate of shrinkage or adapt in some other way. With that, you know, a bit of communication, I think the Fed might want to start socializing the idea that what matters when you're thinking about the size of the balance sheet is not literally how many dollars of assets they have, but in a sense, the total amount of duration that they're, you know, if you think about it from any, you know, reason you would want to have a small balance sheet. It's because you want the Fed to have a smaller fiscal footprint. There's two ways to get a smaller fiscal footprint. One is to just pro-rata shrink everything. The other is to take some of those eight or nine-year bonds that you've got and start holding three-year bonds on average. So they could try to communicate the idea that we will be shrinking our fiscal footprint, but we may be doing some of it by just holding shorter maturity stuff that would allow them all else equal to keep more kind of short-term claims out there. Of course, if, you know, if the Fed doesn't do it, the Treasury can always step up, okay? The Treasury, you know, so this is just a matter, you know, the Treasury can decide, well, we'll issue less than the way of long-term bonds and more in the way of short-term bills. That would work as well. I mean, again, the Treasury historically has been reluctant to do, you know, they're sort of stopped out at about four or 500 billion. The idea that they're gonna add a trillion to the very short end is probably not likely, but I think this is a case where, in spite of all the sort of taboos against it, it makes sense for the Fed and the Treasury to talk to one another. Last thing, I've been sort of talking about this all in the context of the balance sheet shrinking in isolation, but I think this interacts in an interesting way with rates going up in the following, and I think it takes on a little bit more urgency in a world where rates are going up, because what do we know about banks? When rates go up, what do they do? They basically don't raise the interest rate that they pay on their sort of core deposits, you know, savings and transactions deposits. As a result, you know, some people just stay there and get stuck with a low interest rate, but the more sophisticated people like everybody in this room, basically takes their money out of those accounts, puts them in a money market fund, and then it gets recycled back to the banks in the form of more wholesale funding. In other words, the bank's funding mix goes from being more retail deposits and less wholesale funding to relatively more wholesale funding, which just makes all this stuff more urgent, because in other words, the wholesale funding tends to be runnier, flightier, and so at the point when they're doing more wholesale funding, I think it's precisely when you would like to have them doing it six month wholesale funding rather than one week wholesale funding. So again, I think all these issues will come a little bit more to the fore then. So again, my best guess as a forecast, nothing too exciting will happen for the next year or so, but then after that, I think it's at least an open question and it would be good to kind of keep a little bit of optionality available. Thank you very much, Jeremy. Let's move to Hyun and from the short end to the long end of the curve, if I understand it well. And then we'll have a round of discussion when we can connect the dots, not yet. I prepared some slides. So thank you, Benoit. You've given me the task of explaining why long rates seem to be so resilient in the face of monetary policy normalization and I think this is something that's familiar to a lot of you and that certainly is making the rounds in market commentary. In the long rates in all the major jurisdictions, they're pretty much where they were last year, this time last year. Certainly got a lot lower in the middle of the summer last year after the Brexit referendum in the UK when long rates in some jurisdictions went negative. But we are one year since this time last year and yet long rates haven't really budged that much. And one line of argument is, well, markets are far sighted. We do personify markets and endow them with foresight and so we tend to take market prices as signals and so if prices are not reflecting monetary normalization, well, do the markets know something that we don't? In fact, if you run a standard term structure model on current set of prices, the risk premiums, the term premiums actually come out to be negative in many cases. And so there is a bit of a puzzle but I think another way of thinking about this is to take a slightly more skeptical stance because the market is not a single individual with foresight. It is a place where lots of different actors interact and market prices are the outcome of that interaction rather than the beliefs of any one person. And so if we look into some of the details of how the market participants behave, perhaps we can gain some insights into what is driving some of the market action. So what I thought I'd do is to give you a glimpse of one small example of this by looking at the very long part of the yield curve, so the ultra long bonds, by which I mean sorry, in bonds with remaining maturity greater than 20 years. And we have some ongoing work that looks at the micro database at the Bundesbank for the German residents. And we've taken the ISEN level, the security by security level data, but of course we don't have access to the detailed data but we go there and aggregate, we get the aggregate output. And what I'm gonna show you is the result of some summary statistics of around 4,200 ISENs. It's held by German residents and it's gonna be aggregated at the sector level but this is only going to be looking at the seven Euro area sovereigns, so it's if you like the core Euro area countries, Austria, Belgium, Germany, Finland, France, Luxembourg and the Netherlands. And the focus is gonna be very much at the ultra long end but let me just show you some summary statistics just to set the stage. So here's a snapshot of the distribution of the insurance sector in Germany and it's the histogram of the holdings of the sovereign bonds of those seven Euro area countries arranged according to the maturities and into these maturity buckets. You see that the insurers hold relatively long maturity bonds between 10 and 30 years. And this is a snapshot as of the end of March 2013 and if we turn the clock forward and look at the situation as of March 2016, you see this rightward shift. And I should emphasize that these are notional values rather than market value, so these do reflect the underlying shift in the holdings rather than the changes in the valuations. Now, the other question might be, well, what's happened to the overall outstanding amounts of these bonds? Is this just a reflection that there are just many more longer term bonds outstanding? And the answer is no. In fact, if anything, at the ultra long end, so the last two categories, you see that the total amounts outstanding has, if anything, fallen and yet there is this shift towards the holding of these very long maturity bonds. Now, we can arrange this holding of the ultra long bonds as a time series and here is a chart that shows how the holdings, again the notional amounts have evolved since 2005. And the message here is that there's been, as you would expect, there's been a very big shift upwards in the holding of these ultra long bonds. So to give you a sense, compared to 2008, compared to just before the crisis, the holding by German insurers of these ultra long bonds of this subset of Euro-Area sovereigns has more than quadrupled. So there's been a huge increase in the holdings. And that's coincided with the decline in the long-term interest rate. Here is just a plot of the 20-year swap rate. And if we just put that into context, if we plot the other sectors in Germany, what you see is that there has been a shift. The yellow is the investment fund sector. A big chunk of that will also be held by the insurers, but we don't actually have the detail breakdown. The blue is the interesting part. The blue is the banking sector. And for the banking sector, that's really shrunk. So there's been a shift away from the banks towards the insurers and also to the investment funds. If we just look at the proportions, you can see that much clearer. The blue is the banking sector, where that's really shrunk. It goes from around 10 billion euros notional to less than five. Whereas as I said earlier, we have this huge increase in the holdings by the insurance sector. Now for those of you who are familiar with long-term investors, I think you can guess what was going on. I think one story here is that this shift in the holding of long-dated bonds reflects risk management motives, where if you have a balance sheet where the liabilities are of a longer duration than the assets, and there is an imperative to match the duration for interest risk management purposes, then you may see dynamics which give rise to behavior which may seem at first sight somewhat perverse. And the story here is that life insurance firms in Germany, as well as in many other European countries, serve a very important purpose in providing long-term savings products. And the liabilities have very bond-like attributes because they have obligations to policy holders which are often in nominal terms and which have bond-like flows. And those nominal liabilities tend to be of a much longer maturity than the assets they hold in order to back those liabilities. And here's a snapshot from the stress test report from EIOPA, the European Insurance and Occupational Pensions Authority as of 2013. And the way to read this is on the horizontal axis, we're measuring the duration of the assets. And on the vertical axis, the duration of the liabilities. And the dots represent the combination of the asset duration and liability duration of a number of European insurance sectors arranged by country. And you see that Germany is this observation up here where the asset duration is around 10 years and the liability duration is around 20 years, meaning that the liabilities are of a much longer maturity than are the assets that are held to meet them. The fortified realign represents the set of points where the assets and liability duration are equal. And you can see that most of the observations lie above the fortified realign, meaning that for most countries, what's true of Germany is also true, but perhaps to a lesser extent. But the key here is that as well as this duration mismatch, there is what's called negative convexity, which means that as interest rates fall, of course, the duration of both sides are extending, but liability's duration, because it's longer, is actually increasing faster. So there's a convexity here. If you imagine a pair of scales, imagine that you have the asset side of the scales and the liabilities side of the scales. And the balance means that you have the interest rate risk roughly balanced. But as interest rates fall, because of the longer liability duration, the liability side of the scales is tipping down. And so the response is that you have to add more long dated bonds to the asset side in order to even out. So what's just happened? As interest rates fall, you're buying more long dated bonds. So as prices rise, you're actually buying more. So it's a kind of a perverse demand response, but it's a perfectly understandable response because it is arising from the imperative to manage risk. And it may be reinforced by regulation, but certainly prudent risk management is gonna be enough to induce that kind of behavior. But the question is, what will this do to market dynamics if a sufficiently large segment of the market is behaving in this way? Could there be the possibility of a feedback loop where as interest rates fall, there's a greater demand for long dated bonds? And if that demand is big enough and the residual supply is not going to be large enough to offset it, you may see a price dynamic which pushes long dated yields even further. And just to give you a sense of what these demand curves look like, I've just here plotted the holdings of the insurance sector against the 20 year swap rate. And you do see this on the left hand panel, these are just in levels. You do see this very striking downward sloping relationship where the lower the interest rates get, the more the insurance sectors is holding. So on the surface, it seems that there is a downward sloping in this space, demand curve. In price space, of course, it will be the reverse. It will be an upward sloping demand curve. Of course, this is in levels, you might say, well, that doesn't tell us very much. What about in one quarter changes? And the right hand panel gives you a sense of what's happening in terms of the one quarter changes. And it's, of course, much noisier, but there is this negative relationship which seems to be there. And what we're trying to do is to dig deeper. Because we have this icing level data, we can actually do some pretty cool time series analysis. And it does seem that there is this element of a feedback loop that you see the impact going in both directions. When the insurance sector buys more, then it does seem to impact long rates in the near term. And as long rates fall, that seems to induce a greater buying. Now, the question is, are we likely to see something like this, but exactly in playing out in reverse when we see a normalization of monetary policy? And I think that is a really important question for us to ponder. But I think before going to that question, I think there are some, I think much less controversial questions, much less controversial lessons we can learn from just even these very simple charts, which is that first of all, we may take heed of nominal rates much more for their own sake than purely as an input into real rates. Because typically as economists, we care about real rates and when we talk about our star, it's very much about the real rate. But to the extent that asset holdings, to the extent that portfolios depend on the asset liability mix, and that's certainly the case, that seems to be the case for the insurance sector, nominal rates may have an importance in its own right for the equilibrium yield in the bond markets. And the second question, and this I think does bring us closer to the big questions, is we know from textbook comparative statics analysis that if you have an upward-sloping demand curve and an upward-sloping supply curve, that even a very small shift in the supply curve could have a big impact in the equilibrium outcome. And so as we look ahead to monetary policy normalization and we eventually think about reducing the balance sheet, that is going to put more supply into the market and to the extent that there are some players in the market who are behaving according to some myopic rule, for good reason, but nevertheless myopic rules, which are different from the far-sighted behavior that we normally attribute to them, we may see some sensitivity there that we may find exposed to have been surprising. And it also raises some methodological questions about how we can think about long-dating deals because we normally take the asset pricing models as giving us some decomposition between expected long rates and the risk premium residual. And that we find very illuminating as a first pass, but in some ways it is really postponing the big questions because you're not really addressing the question of why the risk premiums have been shifting around. And I think this is a way to perhaps recast some of the questions. If it is not a single rational individual that is the marginal investor here, but rather the interaction of many different individuals, perhaps we should look at some of the micro evidence a bit more. And I think there is also some interesting set of questions which have to do with how this type of amplification mechanism relate to the other ones that we're much more familiar with. For example, with the mortgage-backed security hedging demands that have been known to cause fluctuations in the bond markets in the US. So let me just conclude there. Thank you very much, Heun. What are you? I'm going to go over that. What are we? Do you want to? Okay, we can do that.  So that's what I'm going to do. So I thought I would focus on the issue which is what is the, what should be the ultimate size of central bank balance sheets. And then only speak briefly about speed as opposed to final resting point. So again, most of my time is going to be spent on the first and then I'll say a few things about the speed.  And I could see the point which is if you don't exactly know how things are going to work out, you don't want to commit in advance. But, I mean, at some general level that seems a bit strange to basically start going in some direction without knowing where you're going to end. I mean, let me take two examples. The first one is if we concluded that actually the size of a central bank balance sheet should be larger rather than smaller because we've learned things which make us think that it would be good. Then reducing it doesn't seem like a very good starting point. More realistically, if we think the size is kind of okay or maybe a bit large, but we think of the composition on the asset side is not what we want because it's a result of accidents along the way. Then again, you probably would not want to start just reducing the balance sheet. You'd want to know how you're going to go from the existing balance sheet site size to the kind of assets that you want to hold. It seems to me it's important, even if it has to be contingent in the end, it is important to think about where the process will end. And then on the speed of adjustment, I'll make some remarks. I will start with something which I think is obvious to us, but is really important, which is then we need to think when we're talking about QE and balance sheets, we have to think about the consolidated balance sheet of a central bank and the treasury because fundamentally balance sheet operations of the central banks are government debt management. And in the end what the public holds is determined by the decisions of the central bank and the treasury together. So I think we have to think when we ask the question is do we want a different composition of government debt in the hands of the public, including the possibility that actually the government holds some private assets which is what some central banks do. But we really have to start from the other reasons to think that the yield curve or the term structure or whatever you want about the various premium is what we want or do we want changes either permanently or temporarily. Then the issue, if you think this way, then the natural issue is, well, who should do what? And I think then it has to do with the division of labor, what the central bank knows better than treasury where the central bank can act faster than treasury. So for anything which is relatively permanent then it seems to me that in the end treasury is where the action should be. So I had not thought about the issues before I was asked to, not seriously thought about the issues before I was asked to come here. And I went through the literature of I must admit that I missed the Jackson Hall Conference of last year which apparently was on that topic but I was in France on the beach and I missed it. But so some of what I'm going to say was that if I had known of the existence, this is a paper by John which is if I had known of the existence of the topic of the Jackson Hall Conference I would have looked but not knowing, I had no way of knowing that it was there. Anyway, I concluded that there were five arguments of which I think one is a good one and four are not good ones. Not surprisingly if a good one is by Jeremy. No, I think Jeremy has made a paper that underlies his presentation is an outstanding paper. I think it's extremely convincing on the case for providing very short maturity liabilities. I'm not going to repeat the argument because Jeremy has done it very well which seems to be a demand at the very short end of the year curve and we're really talking short end. I mean basically a few months at most which is not satisfied in the sense that we see a fairly large discount at that end and the private sector sometimes tries to fill it and we're not sure that that's a good idea. So I think the argument is there and I think it's fairly convincing. Now the point again is that there is no particular reason why this should not be done by Treasury. I think that Jeremy is offering as a solution could be equally well offered as the issuance of infinite maturity floating right that. I think it would have the same characteristics. It would avoid having auctions all the time which is one of the issues when you have short maturity and the Treasury could buy and sell it at par, buy back if it needs to decrease it. I have a sense that maybe the central bank is more aware of needs of the financial sector and can adjust faster but the example you gave about seasonality of tax receipts I think it's something that Treasury knows very well. I mean central bank can know it just as well but no better. So the question is should we basically keep a large balance sheet? Not so much because of the asset side but the liability side which is what Jeremy has argued. Am I convinced? Again Jeremy talked about it. I mean how much do we care about the 40 to 60 basis points at the very short end of the yield curve and he has given some arguments for why we might care a bit but in the great scheme of things this is for an ignorant macroeconomist like me. It looks like small potatoes. It might be important. The other thing to say is that Jeremy gave this figure about the average wedge at the short end of 40 to 60 basis points until the crisis. If you look today the wedge is not gone it's about half of what it was and this is after the balance sheet of the central bank namely the Fed in this case increasing from 1 to 4.5 trillion. So if I extrapolate which might not be the thing to do then that suggests that the balance sheet of about 8 trillion would be about the right number. Jeremy is fairly careful actually about not giving a number for what size balance sheet there should be. Clearly the larger it is even if you reduce the duration of the assets you're still going to get increasing maturity mismatch which risk but in his paper he says that 4.5 trillion could be fine and suppose it were then what are we doing trying to reduce it? I mean we have to think about it. So I think on this distribution is extremely large. We really don't know but that's clearly something that we think about. Now this was again, I think that's a liability side argument I find very convincing. Let me go through the others. So the second which I heard at the Fed is and suppose that there is a sudden stop something that we've learned can happen to some financial institutions even in advanced economies and so the central bank wants to provide large amounts of liquidity to a financial institution or to a set of financial institutions and we've learned that that can happen. If it doesn't want to increase the money supply too much it basically has to sell an equivalent amount of assets. Now the conclusion of this is that you should always have on your balance sheet enough assets to sell in case you actually need to provide liquidity and this gives you a lower bound on how much what the size of your balance sheet should be. It's the same thing with effects intervention. Now is this convincing? I find it not convincing. I see that Benoit noted the same way which is that that's not an issue. I mean basically if you want to pump back liquidity you can just issue central bank bonds when you need it and maybe you want to have them all the time so that the market exists but it doesn't seem to be a big deal. You inject liquidity at one end you take liquidity at the other end it goes where it should go and you'll find so you don't need in steady state to have a large balance sheet of I think it's useful to have the ability to issue the central bank bonds but that's a very separate issue. It's more an option that you have rather than something on your balance sheet. The third argument I've heard and this is in the context of Japan which is suppose we conclude that it's only the long rate which affects aggregate demand and I think we've become wiser on this. I mean all kinds of rates affect all kinds of things on the aggregate demand side but we have a sense that many decisions should depend on something like the long rate and then you know so the central bank decides to basically set the policy rate at the short end and the relevant spread on the say 10 year of a three month maturity bonds and intervenes in the market to do so. It needs to do both I mean it cannot if you just go for fixing the 10 year rate without the policy rate being determined you get an indeterminacy which get you into trouble but you can do both you just set the policy rate of the corridor on the policy rate and then you intervene to create the relevant spread and you achieve exactly the 10 year rate that you want. Now if you're going to do this this is a market with extremely elastic demand and supply so you'll have to have a large amount to sell and a large amount to buy depending on which way of in which way you are you want to move the spread and that requires potentially a very large balance sheet if you want to sell it. Now do I find this convincing? No I don't find this convincing again and again there's a whole question that Ricardo Reis is working on in the case of Japan which is I mean can you actually do it? And I think the answer is no you know there's a limit to how much you can affect the spread. I mean the analogy is you know can you both fix the policy rate and the exchange rate? And I think most of us would say no you can't you probably can affect both but the notion that you can fix both we think in the kind of markets that you know these assets are traded it's just not possible. You can probably decrease the spread to zero or maybe even a bit negative as you suggested but surely not much more than that and it would need an enormous amount of intervention and it seems to me that although we have learned that the yield curve doesn't quite move like we thought in the old days the expectations hypothesis is clearly not completely right but I think that by moving the policy rate you can more or less achieve a 10 year rate that you want I mean you'll have to take into account what the spread is but I think you can do a very decent job. So again I just don't find this convincing in any way. This is argument four and five actually because I've put them on the same slide in order to save space and time. So the more general issue is are we happy with the term premium structure of the spreads on different assets whether it's by term or by type of assets. And I've heard two arguments. The first one is the term premium actually is too high and if you think about how you can justify it based on risk you conclude that it's basically higher than your measures of risk that you have would justify and indeed we see many financial players playing carry trade which suggests that there's money to be made that way they don't do enough to reduce the spread but so I think if you can convince yourself that there are distortions somewhere in the term structure then you probably want to use debt management or QE and I'll come back to which one in order to change that. The other argument which strikes me as very relevant is again the probability of hitting the zero lower bound again which is that if you lower the spread on long bounds then other things equal this means you can achieve a higher policy rate than the same 10 year rate and therefore you have a bit more margin to play with a policy rate when times get bad so you can decrease the probability of a ZLB. Now we've talked about many ways of decreasing the probability of hitting the ZLB from a high inflation target to nominal income targeting to other schemes. This one strikes me as a rather costly way of doing it. If you think that there is no distortion to start then creating a permanent distortion by reducing the term premium in order to basically have a bit more flexibility when and if my 4% inflation target surely has distortions but I would trade those for the inflation target distortions. Anyway it seems to me that's an interesting debate to have but in the end I see absolutely no reason where central banks should be in this business because here you're talking about permanent changes in the term premium and it seems to me that that's exactly what that management is about. So evasive case for issuing more long bounds or taken collectively in the hands of the public I see absolutely no reason why this should not be done by treasury and I see no reason why the central bank should basically be taking on some of the operations and some of the risk. So I think arguments four and five where the two lines that there's a distortion you want to correct or that you want to decrease the probability of a ZLB. So conclusions on scope and again that's not because Jeremy's on my left. It seems to me that's the only one which in the end convinces me that I might be. But again, I mean the size of what you need to do in order to reduce something from 60 basis points to five at the very short end of a curve. Not sure that the central bank should really be in that business. Why, you know the political economy of mismatched central bank balance sheets is a big issue. On the other arguments, it's a footnote but I mean I've talked about coordination of that management between the central bank and the treasury. Now it's clearly easier to argue on the other side of the Atlantic. I mean here, you know there's a whole set of issues which is well you have 19 treasuries which are not coordinating between themselves anyway and the whole but they coordinate between themselves and with the central bank on the debt in the hands of the public I think is a hope. And so I think that's the issue then of division of labor and whether the ECB should do more. So I think that's specific to the ECB. On speed, you know I think you on has said extremely interesting things. Clearly the markets don't seem terribly worried. Let me kind of again go back to the basic economics which is you can think of increasing the policy rate or increasing the spread. These are basically the two margins and the question is which one should you use in which order. My sense is that we have a much better understanding of the effects of moving the policy rate because we've done this for decades than of increasing the spread. We have some understanding of what happens in the second case but it's not as good. So it seems to me that given that we're still on this side of the ocean anyway, not on the other side, in a situation in which there's a large output gap, there's very high unemployment and some of it is clearly higher than the natural rate, then I would use the instrument which I know works. I think that just follows from first principles. Would I do some of the reduction in some parts of the balance sheet? Yeah, I would basically look at where QE has created dangers. I mean, do other insurance companies in trouble and so on and so on. And then if there was a need to do something to help them or by going the opposite direction of what you were talking about, then I think there would be an argument for doing this. But for the moment, I would undo the order in which we went in. I would basically first use the policy rate and then they slowly increase the spread by decreasing the balance sheet if the balance sheet has to be decreased which I tend to believe. Let me stop here. Thank you very much. Before I open the floor to the audience, maybe I could give a chance to each of you to comment on what the others have said. I have a question, so I will abuse of my position but just once, so that's a commitment. Just to kickstart the discussion because Olivier discussed at length the division of labor between the central bank and the treasury. But as we learned from Jeremy, initially that's a three-player game. You have the central bank, you have the treasury and you have the private sector, commercial banking, deposit-taking banks, let's call them that way. And Olivier has given very good arguments why the treasury maybe has a comparative advantage in providing the kind of short-term assets that Jeremy asked for. But we haven't discussed that much as a choice between having it done by the public sector or by the private sector. And I feel, after your presentation, Jeremy, I feel a little bit uncomfortable. Also in the light and with the looking back at all the regulatory work that has been done since, say, the Pittsburgh Summit, so that was eight years ago. And I have a little bit of the impression that we're going in circles. So the starting point was, oh, well, these banks haven't managed our risk properly, they've been too big to fail, they've been bailed out, that's bad, which I agree. And so let's have them stand on their own feet and build the right buffers, the capital buffers, liquidity buffers. And that's most of what we've been doing for the last eight years. That's why we have 1,000 colleagues in another tower in Frankfurt and these kind of things. But now, eight years later, you're telling us, well, finally maybe the central bank would be more better equipped to do it. So does it imply that everything we've done has failed in terms of the regulatory efforts on the liquidity side? Or what's the key argument to have this being provided by this public good being provided by the public sector rather than the private sector? So that's a great question. So actually, we have a proposition somewhere in a paper which is exactly your proposition, which your intuition is exactly right, which is if you had a regulatory instrument that was a frictionless regulatory, so imagine you could impose a Pagoovian tax on banks for doing excessive liquidity transformation. If you could impose that tax without slippage, that's the first best instrument and you're done and you wouldn't do any of this sort of distortionary stuff with central bank balance sheets or treasure, okay. In a second best world, where the regulation is imperfect, so it creates some deadweight cost or there's some arbitrage, that gives you some reason to not wanna not do the regulation, you still do the regulation, but basically you can't get all the way there and so then you would do a little bit of both. So the premise implicitly, it's not that the regulation was a bad idea or was a mistake, but it's a recognition that it's gonna be not perfectly effective, especially if the economic incentive is too strong. I think that's the story. No, I withdraw from that discussion and if you want to add anything. I thought it may be just worthwhile bringing the international dimension into this discussion. I think Jeremy mentioned the role of foreign banks in the US money market. I think that's been an interesting observation. I think it also speaks to the question that Benwas said to me, which is why our long rates are so resilient because I think there may be some international dimension to that question as well, in that the long dated bonds across jurisdictions may be substitutes provided that you can hedge the currency risk and so the link between the FX market and the fixed income market really looms into view. And if there are major central banks and other jurisdictions which are engaging as a purchase programs, it may be that even if you try to tiptoe towards the exit, it may only have a limited impact if the demand from other investors who can hold your bonds on a hedge basis still keep rates quite low. It's worth thinking back to what happened in 1994 because 1994 is a good case study in that it sets a very good benchmark. If you remember, 1994 was the year of the bond market crash in the US when long rates really rose very, very sharply. And that coincided with an upturn in the US economy. In Europe, it was in a slightly different cyclical position. It was actually, if anything, needing further stimulus rather than raining in. But what we saw was that long rates really moved in unison and long rates actually rose in Europe as they did in the US. And what we may be witnessing is that the spillover now is much more evenly balanced and that the fixed income investors from the Euro area and from Japan are having quite a big impact in the dollar-denominated market. And if we combine this with the evidence of what's happening with the cross-currency basis, the deviation from covered interest parity, I think the story seems to fit in that we do see the dollar borrowing costs implicit in the FX swap to be much larger than the equivalent market dollar interest rate. And that's reflecting, I think, this underlying demand for dollars that come from the long-term investors who want to invest in dollar-denominated fixed income assets on a hedge basis. So I think if we open it up and take a global picture, I think many of the points that we've been debating, I think, perhaps make, we can shed more light on those questions. Okay, let's open the floor. Four questions or comments?


The Cube building is the newest building on our green campus. This building owns the biggest lecture room of Tilburg University. During your student career, you will definitely have a lecture or event in this lecture room. Within Cube 01, there is room for over 600 students, but the lecture room can be divided by a big screen in the middle. Furthermore, this building has two big lecture rooms where you can work on an exam. There is room for over 150 students who can make an exam all together on the computer. The most relevant thing to know about the Cube building for you as a student are the study places. You can work on an assignment with your group in the middle or you can reserve one of the rooms that are located at both sides of the building. Here you can work alone or together with co-students. There is always enough space in this building and are you in for a break? Then the restaurant is just around the corner.


Renkinographic examination in this 38-year-old white man discloses moderately advanced periodontosis, involving three of his five remaining teeth. The molars appear to be only slightly involved by the pathology. Removal of all remaining maxillary teeth is the treatment indicated. Difficulty may be expected in removing the molar teeth if attempted by other than surgical methods. Upon clinical examination, broad prominent ridges supporting the teeth are suggestive of exostosis. The edentulous portions of the ridge have resorbed normally. Forceps extraction of any of these teeth may cause fracture of adjacent bony structures. To prevent excessive loss of bone by forceful forceps extraction, surgical removal is indicated. The gingival margins and interdental papilla are incised with a scalpel to facilitate reflection of the soft tissue. The small end of a periosteal elevator is introduced into the incision and the buckle soft tissue is raised using care to include the periosteum. The sharper, broad-bladed molt number 4 curette assures elevation of the periosteum without tearing. A broad blunt elevator, molt number 9, is used to retract the flap, exposing the assi structure. A long beveled number 52 bone chisel is used to remove the buckle cortical plate from over the teeth. Sharp, long beveled chisels are necessary to ensure that the bone is cut as intended. Dull or short beveled chisels tend to fracture portions of bone rather than cut them. Bone is thus removed to the desired degree and the danger of uncontrolled fracture is practically eliminated. Bone chips are removed from the operative field. Sufficient bone has been removed from the buckle aspect of the teeth to permit their removal to the buckle without fear of tooth or bone fracturing. Mobility of this bicuspid is due to pathologic loss of supporting alveolus, which makes it easy to remove. Excision of bone over the labial aspect of the canine and lateral incisor teeth permits their easy removal without fracture of the teeth or adjacent assi structure. Soft tissue trimming enables the operator to produce more perfect contouring of the ridge. Inflammatory elements of the soft tissue are also eliminated. Parts to be removed are incised with a scalpel. Removal of the soft tissue can be completed with any suitable instrument. A rangure is useful for this purpose. Final contouring of the bony ridge is completed with a rangure. Care is exercised to avoid excessive bone excision since it is usually desirable to preserve as much of the process as possible. The sharp long bevel chisel is useful in removing undercuts which are not easily accessible with the rangure. Occasionally the palatal tissue must be reflected to gain access to sharp projections of alveolar process and other irregularities, which if left would interfere with comfortable fitting of dentures. These irregularities are shaved away with the chisel and a gently rounding ridge is produced. Here of the wound is with interrupted sutures, a very important feature of the procedure. Tissues heal in almost the exact position in which they are placed. The needle is passed through only one flap at a time. Note particularly that as the needle is inserted into the palatal tissue, it is just under the mucosa. It is then passed horizontally into the deeper stroma and makes exit about one-quarter inch from the wound margin. This careful placement of the suture rolls the palatal tissue over and produces a broader rounded ridge, more suitable for prosthesis than a peaked ridge of flabby tissue. Note again this method of suture placement to roll the palatal tissue over the ridge. A well-contoured ridge can thus be produced in cases where isolated teeth must be removed. A single tooth in an otherwise edentulous ridge is removed in the same manner. Incisions are made to permit reflection of the mucoperiosteal flap. The flap is reflected carefully. This tissue is frequently thin and friable, easily torn. Small, delicate, sharp elevators placed in close contact with the bone assure elevation of the periosteum without tearing. The broad blunt retractor, molt number nine, protects the delicate flap as the buckle process is excised with the chisel. The tooth is removed to the buckle without fracture following partial excision of the buckle process. Contouring of the ridge follows the extraction. A small wedge of tissue is excised from the buckle incision so that slack in the buckle flap will be taken up with suturing. The wound is closed in the same manner as previously shown. Again there is no attempt made at complete closure of the sockets by close approximation of the soft tissue margins. It's post-operative. The extraction wounds have healed and the patient is ready for prosthetic appointments. Although somewhat irregular, the ridge is well rounded free of undercuts of an average height and will provide a firm denture foundation.


Well, let's get started. It's 320 Welcome to the Afternoon session. Why is my volume in air state? The goal here is a basic introduction to troubleshooting your sender configuration I'm Kurt Martin. I've been a cloud Engineer at HP for some time. I started working on open stack back in the grizzly time frame along with my co-worker Walt We we basically introduced fiber channel support to open stack and we've been involved with sender ever since the grizzly time frame Jay Bryant another Sender subject matter expert. He It works for IBM and we put this session together as both both of us and Everybody else in the sender community. We get asked a lot of these same questions As you notice both Jay and Walt are core members, so their IRC names are hidden So it's harder to find them when we need plus twos on code at the end of the release cycles I'm easily recognized with my IRC name a K. Martin plus United lost my luggage. So I'll look like this all week So for the agenda, I'd like to go over the sender volume states that we have there's quite a few of them You'll usually see these in the in the log file, but most noticeably in horizon if you're in what volume you're You're what state your volumes in I'll briefly go over the architecture and volume flow As well as the log file locations for the sender volume sender scheduler API and during attachments where those errors may occur over in the Nova log So I'll go over that then I'll invite Jay up on stage and he'll go through the some of the common errors with sender Walt will come up and do some of the attaching issues Both over with ice scuzzy and fiber channel, and then we have some useful links and we'll take Q&A So as you see there's quite a list of volume states that a volume can move through during its life cycle a Lot of times you'll just see air in Horizon and what did that exactly mean? We'll give you some hints on where to where to resolve those issues some other things notice Notable here is Sometimes the volumes this gets stuck in a particular state. It'll this spin at creating, you know Where do you look for those types of errors or? Something will when you go to attach a volume it will this sit in attaching state for a while and then go flip back to available So that wasn't successful. Where do you go look for those errors? So I'm sure not everybody's a sender expert in the room. So I just kind of wanted to explain the process What I've done here is Nova has just as many components if not more inside But I kind of wanted to blow up sender show the main pieces. We have inside sender We have the sender API the scheduler the volume manager and sender backup Calls come in through the client And they actually Will go all I'll describe like how volume is created. So basically on volume creation the sender Clients sends an API request over rest of the sender API Sender API takes that request turns it over to the scheduler where it uses and figures out its Capabilities that are reported up from the back end or the storage controller and it will decide based on Configural filters and wares Where that volume would best be suited and then it goes to sender volume and will actually create that volume on the back end storage array Or LVM or a distributed file system on Nova attach So this is when you want to actually attach a Volume to a VM it goes through a slightly different flow Nova will actually call sender through its API and it will pass a message to sender volume through the through the message Q and we'll return some information to Nova like IQ in worldwide names host names of where It wants that storage to be connected to Once it gets makes senders does some information to make sure and Nova to make sure that that it that those Connections can be made permissions chap support, etc. It will actually go and come back to the The connection information is returned that was passed to Nova and Nova will actually do the connection with the with the either for fiber channel through the HBAs or with ice scuzzy through the Ice scuzzy initiator and target and make those connections So that's the flow there in the purple of once the volume is attached and once it gets attached to the sender Or to the compute host that hypervisor takes over from that point and actually does the device mapping to the VM instances So here are some of the log log files that are pretty important and we're will you where you will find some of the Heirs so on the controller node that's sender Most cases, but it's not always required that all three of these are running on the same system the scheduler the Volume manager and the API, but it's not required but in this demonstration will consider that it is The sender API log is very useful Determining if you have any connectivity issues that are in point issues The scheduler does all the scheduling of volume creation often times your volume won't be created And you won't see anything in the sender log You should look in the scheduler log The sender volume log is a main sender log for all driver related Configurations and any type of errors there The compute node is on the Nova side. So this happens during the attaching phase If the attach fails like I said earlier will actually go through it like it's attaching and instead of it being in use So we'll go back to available what happened chances are you'll have a an error in the Nova log To enable debugging on the log in the log files Simple debug equals true. That will turn on debugging. You can also turn verbose to true for info messages and Those options are in in the sender.com file Anytime you change that you will need to restart the sender services. Unfortunately, we're working on making some of that dynamic in You know a future release of OpenStack some vendors for example the three par Driver has some client information and that's another option to turn on to have some additional debugging there All this most all the drivers do a real good job in the configuration reference guide documenting any additional features like this I now like to invite Jay up to go through some of the common errors with sender. Okay Thank You curtain So what I want to do here is hopefully now you've got a good decent idea You know if you weren't familiar with sender already How the pieces fit together and where to look in the log files now? I'm going to start walking through some specific log files and errors to give you an idea of What can go wrong just when you're first trying to bring sender up? When you get beyond that driver initialization problems. Okay, so my my vendor driver hasn't started the way expected Get you beyond that into well Why is my volume not being created properly and then finally it'll pass it off to Walt and he'll talk about you know some issues with attachment So hopefully give you at least a flavor as we go through all the different places where you may need to do some Debugging to help you get started and and give you pointers for the future So and these examples are all very basic Some of them are using LVM Which is what you get with dev stack by default is your setup and then also some examples using Sands with store-wise in the in the three-par so covering You know basically all the different varieties that you may see here as much as we can in 40 minutes so first I've got my my sender setup and I'm trying to connect to it with my client now And I apologize all these examples are from the command line But kind of the assumption is in your early setup you're going to be primarily using the command line You'll see some of these errors through horizon as well But at this point the best place to do debug is from the command line using you know going into the log files on the control node so Anyway, so at first if you don't have sender you may have installed it, but the services may not have come up When you go and try to use your sender client to talk to the system You'll get an error like you you see here on the first Picture where it says that's not able to establish a connection to sender That means okay I can't talk to the API and you want to go and check you know using etsy and net.d in that directory check our open stack sender API and open stack sender scheduler and scheduler volume are those are those all running That's the first place to check I can see in the example here I started the API service and then I was able to do a basic list command and see that I hadn't created any volumes yet So now I'm able to interact with Sender What happens if I I do that list command and I get an error back that says that it gets an HTTP 500 I can't talk to the server Unfortunately, this is kind of a you know a generic error that you will see in a number of situations I've gotten the question many times before why why do we report this generic HTTP 500 error? Well, some of it is for security reasons some of the error messages that come back to a client if if we didn't keep them You know if we printed them all out you could get user names or information we want the back ends to be very very much Hidden from the user they shouldn't have to worry about that That's why that information goes into the log file and we spit out this kind of generic error that says okay Something's gone wrong go take a look or talk to your your cloud administrator to find out why why you can't interact with the server in this case I went to the Varlog Sender volume file and you can see that there's an error there that that says it was unable to talk to the database So that's another thing that often goes wrong when you're first setting up the system as you forget You know you've rebooted or something you don't have all your startup services set properly and oops the database isn't there And sender is not going to be able to go very far without with that access to the database. So in this case You want to make sure that your database is is initialized If you're using my SQL you've got the open stack DB command if you're using DB to use open stack DB to And then use a net and the the sender service and it'll make sure that you have a database It's created and that it's populated with everything you need to run sender. So That's the the very first couple of things you need to get through Also again kind of going down the theme of do I have all the right services started here If you don't have a MQP running if you don't have either Cupid or rabbit depending on on which distro you're using That's also going to cause errors and you'll see in your your volume service that it's not able to connect to the to the messaging service and it will time out and it'll keep Trying ever every so many seconds to communicate with that. So in that case You know, you want to make sure that your your messaging service is running And also then check your sender comp file to make sure that you've got the Cupid hostname set correctly For your controller node and that you have the right The right port set that's another place that can be a little tricky depending on whether using SSL or not The port that you're using may be different from the default. So that can cause a timeout talking to the messaging service as well Another another tricky piece here is when you if you're using a sand like a store-wise or a three-par Behind it and if you don't have your IP addresses correctly set There's often for people are new to the storage area. There's confusion about the management port For talking to the sand versus the ice because the IP addresses And so they'll use the management port setting for for everything in the Configuration file or they'll get those those flipped and that will cause issues with your driver initializing It won't be able to actually talk back To the the sand because you know, it's using the wrong. It's using the wrong the wrong port and either SSH isn't listening or something like that So so you want to make sure that when you're setting up your configuration that you get the right IP addresses in the ice Because the IP address as well as the sand IP address fields and And this is one of those errors that the fact that that has failed may not be initially Immediately obvious because when you go and look at the volume file the volume Log file you'll see that the driver is not initialized and it's trying to do a Status update and it's not working and you have to actually scroll back up to when it was first starting to see that The first SSH command to try to talk to the back end has failed So that that's a general tip for those of you who may be new to debugging open stack You often have to go back further up the log file because there are probably breadcrumbs That you need to find before to find out the final reason why you've encountered a problem So okay now we've gotten beyond you know, we've got our services up and running we're thinking thinking cinder should be be running and We still can't create a volume. Okay, so in the the example of an LVM a Setup the common error is I don't actually have the volume group that we're trying to use either because the creation of it by by you know the by the stack setup has failed or You know, you've got don't have enough storage something failed along there in your in your test setup You want to go and make sure that that you actually have a volume group So in the case that you don't you'll see a nice error like this. That says the cinder volumes volume group is not available And so you want to go and make sure okay? In the case of LVM, you know use some use the VGS command Do I have any volumes volume groups defined in this case? You can see no I don't so that's that's a source of problem You may also have a volume group with a different name Created and you need to go back and change, you know the volume group line in Etsy cinder com to to use the right volume group name for your configuration So that's that's a common problem that you'll run into with LVM startup next So if you're using a sand instead of LVM, it's often SSH that bites you during the driver initialization process so why do we need SSH well because At least in the case of store-wise and three par a little bit They're they're more based on rest but in the case of the store-wise drivers it needs to SSH to the the driver back end to Do the actual SVC commands to set up your volumes create them export them to hosts And so if it can't do that it's not going to go very far So that's that's one of the things that it does here early in the the process is checks Can I talk to my back end? If it if I can't then you're going to get a failure There are a number of things that that can cause that failure The first one that bit me most often is if you're using SSH keys You have to make sure that those keys are accessible by the cinder user not the root user You know that so they need to be in a path That's that's accessible by the the cinder username other words You're going to get an error trying to read the key or people forget to put the keys in the location They've specified in the config file. That's another way that that things can potentially go wrong So check your sand private key path and make sure it's pointed to the right location and it's a file that's accessible If in the case that you're you're using username and password the authentication failure is nice and easy to spot It spits out an error right there that says hey authentication failed so that one's not so bad And again, this is a case in both cases where when you get further down the file You're going to see this this warning that it can't update the volume stats and the heart of the failure is really further back up The file where it where it had initially tried to talk to the back end and that failed Okay, so hopefully that's gotten you beyond you know common pitfalls in getting your driver initialized And now we can can try to create a volume Which is where we all eventually want to get to so here is the The kind of things that can go wrong when you're trying to create your volume and you end up with that that irritating error message That it's like okay. What's that mean most of the time you're going to go and it's this case You're going to see the failure in the scheduler file It's going to give you this generic. I can't find a host. What does that mean? Well, it means you've made a request and it's gone out and it searched the availing available volume services and said hey Where can I make a volume like this and they've all said nope? I don't I don't have enough space for that or you know in the case of my example here My sender volume services and started so I go and I use the sender manage command Along with service list and it shows okay. My scheduler is running again nice happy face But I've got that irritating triple X next to my my sender volume service So when the scheduler went and said who can make me a volume nobody responded and you end up with your volume in error state So that's that's the first thing to check and if you know if you don't have a volume service running You know go back and make sure do you have this is the is cupid running properly is the volume service able to talk Back to the control node is it started, you know do the debug there again? Is there enough free space? Obviously, okay your volume service may be running, but you may not have enough space And so in the case I show here. I've got a 10 gig Volume group and I've already used up eight of it and I tried to make another eight and that fails In the case that that you do that. I think I show here. Yeah If there's not enough free space at least there we give you a little bit better hint that says, you know There's two gig available and you wanted eight. I can't help you now One trick here is when you're using LVM remember that it needs some additional space for like metadata So if I tried to create a two gig volume that wouldn't work as well You know I could create a one gig one, but but at two gig, you know, you need some that extra buffer space in there so You'll see the error message sometimes and think well, that's not right But have to remember that you're dealing with with some space for metadata in there, too Oh, this was another good one that that Walt ran into while creating the presentation You have to make sure that if you're using volume types, which I'm sure a lot of you will be or have started using volume types You have to make sure that your back end name that you're using if you've got multiple back ends Specified you have to use the same name in your volume type as is put in the cinder Configuration file. In other words, it will also throw the no-host found error So that's another good one is you know make sure that you've got your volume type set up properly in and are in Lined up with what you have set in the cinder comp file. Oh a couple of ones I ran into actually while creating the presentation are you know, you can end up in with cases where The the volume is stuck in the creating state and it's like well, what's going on here? Well, I'd forgotten I'd stopped my scheduler service to do something else and Went and tried to do this and it's like It's LVM. This shouldn't be to know it's on the on the control node This should not be taking two minutes and I go and I use Cinder managed service list and I saw oh my scheduler is not running So that's one case where you may may see that you know It looks like things are stuck and so I went and I started it and it was okay, and then Magically my my volume was available So that's where you can see it stuck in creating Also, it's interesting if your volume service if somebody pulls it out from under you While you're creating your your volume you can get in a state where it sticks in the creating state as well So those are a couple little gotchas you want to watch out for In the case that that you think your volume should have created nice and quickly and it's not showing up So go check those two things so Hopefully that's gotten you through being able to to you know get get your volumes created And once you've created them you want to actually use them Which means you have to attach them and I'll turn it over to Walt to talk about that. All right. Thanks, Jay Let you get by here. Thank you, sir All right, so Kurt and Jay actually talked a little bit about Problems that you have when you're standing sender up and or when you're configuring sender and making sure that those actually That sender comes up properly and that you have sender configured So there are different problems that can arise when you try and do a volume attachment And those actually could be related to sender comm changes that need to be made and or problems that actually happen on the Nova side because there's tools in the tool chain that May not exist that that are required So one of the things here the first thing I'll talk a little bit about is when you try and attach An ice-cazi volume using the LVM driver One of the common things is to ensure that the ice-cazi IP address is actually set correctly Because if you don't you won't know when you start sender up Right because what actually happens when you try and attach that volume happens The code path that actually gets called only happens at attached time which is What it tries to do is it creates an ice-cazi session between the compute host and whoever serving up the volume, right? So in this case sender won't complain because the IP address is actually not accessed until you actually attach the volume so The error that we show here is there's no session found And that's basically what happens when you have an incorrect ice-cazi IP address So it's an easy one to resolve you basically confirm that that's the actual IP address Make sure it's pingable make sure it's on the network And then go back and read at cinder dart comf and and bounce cinder All right, so Now we're talking a little bit about fiber channel and fiber channel is a little bit more tricky because Of some of the things that are required on the Nova compute site So there's a couple things that you need obviously is that The coordination between Nova and cinder is what drives this whole process, right? So Some errors might happen That might originate in the Nova log some errors might happen and originate in the cinder side of things so One of the things that is required actually to have to to exist on the Nova side is is a couple Utilities that need to be there because Nova has no concept of of what fiber channel is right for the most part It just uses we just use utilities that exist within the system to actually discover Such things as the fiber channel HBAs and the HBAs need to be there to actually make the connection to the other side Where the array actually exists so? Nova actually uses the sysfs utilities utility called sys tool to actually discover what the HBAs are to find out what the worldwide names are worldwide port names Okay, I guess we're okay and so the problem with this is that When you stand Nova up you won't see any of these errors, right? You only see this when you try and do the volume attachment and that's why this one's a little bit tricky is because the air happens Deep embedded with all the noise that happens within the Nova log within the cinder log And if this one is really kind of hard to track down and And you can see Where is it here on this line here right here is that sys tool is not installed Right, so that one is kind of hard to track down But you have to install that as well as another package which I'll talk about here a little bit later But for redhead systems use it's called sysfs utilities installed with yum Ubuntu you install it with apgett So that's that's one of them the other is obviously you need to have a fiber-channel Kernel driver there on the Nova compute side in order to actually even see the HBAs at all and For the Amy looks cards, which is what we use in in our lab The actual kernel module is called LPFC and and that is included in the Ubuntu Linux image extra and then the version of the kernel that using generic package that doesn't come installed with the stock Kernel so this one bites us occasionally with some other out of our engineers that we actually are Training they'll go well. I don't have any HBA cards. Where's the driver? Where do I get it? So that's that's one of the package One of the kernel modules that you have to have And so the one of the other support packages that need to be there in order for fiber-channel to work as the sd3 utils We actually use SG scan utility to find out some information about the block device Which is used at detach time? And that error will also show up only in the nova log when you're trying to do a volume attach it You won't see that in cinder except the fact that the attachment actually failed Okay, so this shows a little bit of an example output of what you should see if you manually run the sys tool We basically show a little bit of an output here to show that the not only is sys to installed but also that the One of the HBA ports is actually online the last line here is what's the important piece here Because if it's not online, then you're not going to be able to do any attachments to it, right? So this usually means that it's not wired up. It's not enabled different hardware different manufacturers have different Requirements for making that online, but for ours it needs to actually be plugged in and wired up for it to actually be online And so if it's offline it doesn't really matter if you have sys to installed it you you can't do attachments to it, so All right, so things get even more complicated if if you're familiar with fiber channel Most people don't have them directly connected up. You need to have a zone manager or switch in between there right to create your fabric and so Cinder as of ice house has a new capability called the fiber channel zone manager Which actually creates zones for you on the fly So you don't as an administrator have to pre-zone everything up because that's completely impractical, right? So we we try and do our best to automate the zoning at attachment time So some of this is we show the first block up here. We show the part of the fiber channel zone manager config which is in cinder.com And you can see for the switch that we have We're talking to here's a brocade switch and you can see the address and the user credentials so Obviously you need to make sure that the credentials are correct. You can actually talk to the switch and The problem that we have today is that when you stand cinder up it starts up We don't actually go and verify that this which is actually there yet So you only see this error at volume attachment time one of the things that I'm looking at fixing for For the next release is actually doing some validation and cinder that when it starts up You can actually make sure that it can talk to the switch So trying to debug those problems won't happen only at attach time Which is kind of a pain? So we show an error here At the bottom of a configuration problem that we have when they misconfigured the zone manager with incorrect credentials And that's in the cinder log Okay, so that doesn't cover all of the problems that you might see with Attachments, but it covers a good majority of some of the simpler cases that are actually pretty hard to debug and Track down but it gives you kind of an overview of some of the issues that you might see So here we provide a couple useful links here the cloud administrator guide and the Configuration reference guide gives you good information on how to actually Configure cinder and some of the drivers that are involved here as well as the zone manager Configuration because they're they're non-trivial things and each driver has is their own specific configuration items that you need to get right and if All else fails and you're still having problems We provide a launchpad link here for so you can file bugs so All of the cinder team here usually sits in the open stack cinder channel on free node So if you have any questions Now's a good time to ask them or you can get a whole of us on IRC I Guess that covers it for us. Yeah, go ahead come up to the mic, please Hello, I have question. We are using IBM store-wise as cinder plug-in and Have you tested life migration at fiber channel when you boot from volume? Instance because we tested and we have several issues, but not only with store-wise, but We've signed there in general When you want to do life migration on volume back instance So I personally haven't tested live migration with store-wise because I work at HP and we work on three par But we have done a lot of testing and a lot of work on live migration recently, especially in the Juno and And the trailing in the vice house time frame and there were a lot of problems with it a lot of drivers And cinder were under the assumption that you can only have a volume attached to one host in any one given time And that's actually kind of sort of not true But most driver riders actually write their code under that assumption and we actually had a bug that we had to fix At the end of ice house so and and one of the problems with live migration and has been for a while Is that it's actually not in the gate yet? I don't know if they're actually fixing that for kilo or for it ended up in Juno I'm really not sure but if we get it in the gate, then we'll see a lot of these problems actually exposed early on in the system That way we can actually as Vendors, we're riding drivers. We can actually Attack some of those problems and fix them up front in terms of store-wise specifically a hand that over to yeah So that's a good question. And if you don't yet have bugs out there Please get them out there and get a hold of me and I can can work with you to get those We've got a team and CSTL that's dedicated to store-wise And I know they'd be be happy to work with you to get those resolved So, okay. Thank you. Thank you sure Anymore? All right, great Also question if it's possible to map one volume to multiple instances That's a great question. I actually started working on multi-attach, which is basically what this is a Couple releases ago and it got stuck in in reviews and Nova and It's still on my plate to do and it's something that we're actually looking at trying to implement But I don't have it done just yet There are a couple patches up there for work in progress that's it requires some pretty extensive changes Actually within Cinder the API the Cinder client as well as Nova To support this so it's it's a non-trivial thing to do and as anyone knows getting reviews through Nova's is difficult even as a core member so We want to do the job right and make sure we're not going to break Nova especially With doing volume attaches because that's pretty much a lot of that's pretty much everything, you know With respect to Cinder, so we need to be careful there But it's it's a work in progress right now. We're we're gonna try and get it done as soon as we can So so thinking it will be in and kill her release. I don't know. Maybe I hope so Don't quote me on it though. Yeah, go ahead. I'm just curious. So you said like I'm working on a way to you know figure out the Problems well before in time. I'm the Nova attach problems, but the Nova attach would happen from the Nova side, right? So, yeah, are you gonna like what's the plan again? I'm gonna check with Nova during Cinder Is that like are we able to see this IP or like how is it? So so there's a lot of problems there right is is that? Right now there's no there's no concept between the coordination in Nova and Cinder to actually do some checks up front Right, so one of the things that I'm looking at doing for Cinder Is because we can control that code is that in terms of fiber channel is trying to do some basic validation at stand-up time for Cinder because tracking down problems at attached time is Kind of a painful thing to do, right? So but one of the things that I would like to see happen on the Nova side is when Nova stands up to actually get a count of Some of the requirements that are actually needed to be there for in terms of fiber channel, right? But in terms of actually ensuring that ice because the IP addresses are there and pingable You know that that's kind of up to the administrator at that point Nova can't know all of the the endpoints of all of the back ends that Cinder has and ensure that it can ping all Of them, you know that that's a little bit more difficult of a problem and kind of out of the scope of Nova I Would say that if you have Cinder configured correctly and as a driver writer I do actually do a lot of Validations in our driver code a stand-up time when it starts up to ensure that hey Yeah, I can talk to our ray in the back end and it's actually there and and do some validations there And if we fail then then then that driver instance gets put into an uninitialized state and you can't provision against it And you should see that immediately on the logs and be able to to try and track those downs with Looking at some of the air messages that you get in the Cinder log. So I don't know if that really covers everything that you ask but No one the Cinder, you know the Nova compute and Cinder could be on a different system, right? So it's possible that because of some networking issues on the Nova side, you know Could not work, you know, so right so I mean ideally I mean this is as you know I'm just thinking a lot ideally they should be a way for Cinder to tell Nova that this is what I'm using and you know Can you just verify whether you are able to ping it or something like that? Why can't we do like that? Yeah, that that API doesn't exist today and Right now the only way to track that down is that attached time, right? Is that Nova needs to be able to see that array in the back end and if it's not there it's not there right now There's there's no way to Nova to to pre call and say hey, you know what Cinder? What are all your back ends so I can ping them to find them that doesn't exist? I mean before doing a Cinder attach, you know, maybe split that touch into two operations first is a pre-check operation and then a real attach kind of thing I'm not sure what the value to that is because it's gonna fail either way, right? Yeah, but then it will not take the whole, you know, as you said the lip word and the QMU log will not come It's a pre-check itself will fail send this IP, you know center wants you to connect to this IP But then the pre-check itself is filling in the pre-check it will just do a ping or something from the Nova So, you know, we'll not have that baggage of the error, right? I'm just assuming You still need to get notified out of the way, right? So Right Okay. Yeah for doing I scuzzy. Yeah I guess we could add that to to the liver volume driver for ice because he attaches to ping it prior to actually doing an ice because he admin log in But at the end of the day, it's gonna be an error that happens at the same point in time in terms of the log So the value of that I'm not sure Okay, thank you. Sorry Hi, sure. Go ahead. We have one more question short if we can Admitting you are in an error state for whatever reason at creating time of the volume You find the root cause and you fix it but you have to fix the database the SQL database to Because the quota has been affected Reservation has been affected. Is there any way in a flow or current work to avoid this reservation? To avoid the reservation itself. I'm not sure about that Revolting them. Well, what you what you can do today is if if you get a volume stuck in the creating state Due to whatever reason you can actually reset the state of the volume Yeah, if you want using the the command line utils. So that that is one mechanism Yeah, now in terms of fixing the the quotas and the reservation. I'm not as familiar with that That piece of sender Personally, but J. Do you have any I'm not sure that it actually does get reserved if it's stuck in creating It's still it is reserved, but if it gets put in an error state. I don't know if that actually gets left in or not Yet. Yeah, it should be reverted When if it if it's not when you're deleting the volume in error state it in an error state Yeah, yeah, quote in the reservation. Yeah, that should should be reverted and if not something we should look at Okay, that's all the time we have for today if you want to get in touch with us We're always on the IRC channel open stack sender. Thanks guys. Yep. Thank you for coming