Thanks, everyone. I wanted to take a moment to introduce Huiget Lebel, someone who is well known to many, if not all of you, I'm sure. I'm happy to say we welcome Huiget to be a board member of GFI last year, and right from the outset she was – it was a very comfortable fit. Not only is she an expert in international development due to her time as President of the Canadian International Development Agency, but like Thomas Pogge, who spoke a bit ago. She is a PhD in philosophy, and much like Raymond Baker, she probably flies about a million miles a year, so she fits in quite well. During her very distinguished career, she's worn many hats. She's been Chancellor of the University of Ottawa, head of the Canadian Red Cross and as well as the Chair of the Board of Directors of Transparency International, to name just a few. In 2011, she was awarded Canada's highest honor, the Order of Ottawa, which recognizes the highest level of individual excellence and achievement in any field, and I would say from her bio it could say in excellence in many fields. Please welcome Huiget Lebel. Thank you, Tom, and good afternoon, everyone. You've got a lot of staying power, obviously, but I'm glad you got around to go and get a coffee. Otherwise, you know, people start getting a little bit low after a while. Well, I think we need to start with the dedication of the book that is being launched today, and that dedication, Raymond, is to those who suffer the indignity of poverty, and I would add to that the danger of living in conflict areas and with insecurity. So we're really talking here of something very, very important, and I want to congratulate GFI, which of course I'm part of, but for the work that's being done, and for the very long-standing collaboration between GFI and Transparency International, and would like also to come in the fact that we have some colleagues from TI in the room as well. I don't know how many of you were in Monterey. That's where I started, which then of course moved on to Doha and to Addis. Although Monterey did say a lot of the right things, it was a very different conference. There had not been all the discussions, the hearings that we have seen en route to Addis. But Addis, like Monterey, was not necessarily easy to achieve. Countries were debating, both behind the scenes and in front of the scene, because many had different views about what should be in the declaration in Addis. And the responsibilities, you know, as well as the collaboration and the advocacy of many organizations was very important, I think, in pushing forward some of those discussions, including civil society organization, the academic group, and some leaders of industry in terms of doing this, not only at the global level, but within countries. So this year we have heard is of course a very important one. And when you look at the goals that are part of the sustainable development goals as well as Addis, we've got a lot of the right stuff there, a lot of the things we would like to see. And of course the problem will be what do we do after September, and I'll come back to that in a minute. But before that, when you, I don't know how many of you have taken the time of reading the Addis declaration, I'm sure you do that every night before going to bed. I have read it a number of times, but, you know, what is rather interesting there is that, of course, the main goal is to end poverty and hunger and then achieve sustainable development with three dimensions, promoting inclusive economic growth, protecting the environment, and promoting social inclusion. I will leave aside the commitments that have been made broadly, the cross-cutting areas relating to the SDGs and that were also raised in the Addis declaration, and come to the action areas immediately to deal with the topic at hand today, which is illicit financial flows. What is interesting is that the first item of the action on Addis started with domestic public resources, and right immediately, item 23 to 29, dealt with some of the work that we are all concerned about and that we have heard since noon, whether it is redouble efforts to substantially reduce illicit financial flows by 2030, with the view to eventually eliminating them, reduce opportunity for tax avoidance, ensure transparency in all financial transactions, and ensure that all companies, including multinational pay taxes to the government of countries where economic activity occurs and value is created, encourage information sharing among financial institutions, encourage the international community to develop good practices on asset recovery, strive to eliminate safe havens, strive, I underlined the word strive here. It's a weak action, strive as well to strengthen regulatory frameworks at all level to further increase transparency and accountability of financial institutions and the corporate sector and governments, strengthen international cooperation, scaling up tax cooperation, transparency, natural resources, access to beneficial ownership information. It's not quite public register of beneficial owners, but it's at least getting there. Take into account the work of the OECD and G20 on base erosion and profit shifting, strengthen national audit institution, establish transparent public procurement frameworks, and so on. So I'm sorry, if I'm listing those, it's to make the point that a lot of the commitments that have been made in Addis are very much in line with what we are talking today. Why have we got these kinds of commitments? I would like to believe that a lot of it is because of what has happened since Doha. What has happened in the last, let's say since about 2007, 2008, and Raymond I would probably say with your first book on capitalism's Achilles heel that you published earlier, I guess in 2005, where illicit financial flows began to be put on the world agenda. But what has happened since Monterey and Doha? First of all, Norway played a very interesting role, the government of Norway, in holding a number of conferences about these issues. The establishment of what is now called the Financial Transparency Coalition, which the GFI was very instrumental and which brings together a number of organizations, including Transparency International from different parts. And this is, I think, playing as played a very important role in keeping the issue on the world agenda and continues to do so. And of course, this was supported also by the government of Norway along the way. A third point, I think of what was happening during this, was that the OECD started to have a number of conferences on between tax authorities from around the world. And this and their early work, I would like to believe, was important in getting the G20 to take note. There were more things that got the G20 to take note, but that I think was one of them on the whole issue of beneficial ownership and tax exchange and tax information. The financial crisis, I think, got western countries to realize how much money they were losing on tax evasion and also on tax avoidance. And that became, I think, a very important issue around that time. As well as some whistleblowers started to put some information on the world discussion, as well as some legislation, and I'll come back to that, but the discovery of the complicity of a number of banks, large banks, in participating, either by not doing due diligence by looking the other way or by being complicit in illicit money from persons from PEPs and others. Now, something else that happened during that period was the U.S. Foreign Account Tax Compliance Act, and this was referred to earlier. And I won't go into that because a lot of people in this room are probably very familiar with it, but it has meant that it is sort of a sense forcing other countries to cooperate on other financial institutions in terms of searching the records for reporting of persons who have assets outside of these countries. And I think this is launching a global discussion about reciprocity in this regard. I think another point was the G20 and some of what they have been committing to, including the asking the OECD to launch its base erosion of profit shifting initiative to develop multilateral instruments to facilitate the automatic exchange of information on tax, aiming at curtailing, of course, seepage of revenues from developing countries, but also from Western countries. Something else that happened was what did happen in the EU Parliament, where in 2013, 2014 legislation was passed requiring countries of the EU to report taxes paid on a country-by-country basis where their multinationals were operating, as well as approving legislation requiring public registers of beneficial ownership of companies in member countries. The UK started early on that as well. January 19th of this year, the high level panel on asset financial flows from Africa, chaired by Tebow Mbiki and Raymond, you were part of that panel. I think the publication of that panel again put the whole issue of illicit financial flows on the agenda and reinforced the work that was taking place. Now, what is happening as well and what you will see in the Addis Declaration is what I'm calling at least to some degree a new interest in what's happening at the subnational level. We've concentrated a lot on national level public sector and otherwise. A lot of the activity is happening when we're talking of illicit financial flows, not just at the national level but locally or provincially or statewide. I think that when you're looking at the greater Tokyo or Shanghai with close to soon will be 50 million people within the next few years, not in a decade from now. These are bigger than many countries put together and therefore their activity, the activity at the city level, I think is something that has been taking greater traction in the past while. Also, I would say that because of a number of scandals, because of publications from the World Bank, IMF, regional banks, OECD, UN, the whole area of infrastructure is also there. It's been there, but it's certainly not fading away as where so much money is lost, which then becomes part of the illicit flows running around the world. That, I think, remains a very important aspect. And I think the role of, in these last while, the role of funding organizations, and I'm referring to Ford here, Norway before, in supporting the work leading not only to this publication but to a number of other efforts remains very important, and I think other organizations need to be encouraged to follow through. Now, my points here have been, a number of things have been happening while leading to ADIS, which hopefully have had some influence in getting the commitments that were made. But we have seen commitments made before with very little action after. So going forward, I think, remains tremendously important. And some of what I've just said, I think, refers to that. But to me, going forward is really a new beginning to ensure that commitments made are respected, commitments that actions happen following that. And I will identify a few critical areas where work will be important, and I would like to commend for you to read the book. And Tom, the, your last chapter, I think, is very good in disregard and very helpful. And I will have some of the points that you have made here, but a few more. I think what we all know is that if you don't have political will at the top and a readiness for action, you won't get very far. In order to get that, you also need the people to begin to understand that they don't have to live with the loss of the resources that belongs to them, and that they are ready to unite, come together, and put that pressure upward, but also be able to provide constructive support and recommendations. And in between, we have all the institutions that we heard about, whether it's customs, whether it's the judiciary and many others, that if they don't have the strength, the capacity to deal with enforcement and prevention and detection, not a lot will happen. So, I think going forward, we're talking of financial transparency and all revenues, all disbursements, following the money, as was said before, remains vital. Otherwise, we won't have the information. Neither will the people in the countries to hold their governments accountable. The issue of public register of beneficial owners of companies, trusts, and other financial institutions in all countries. I mean, the G20 has now taken this on, but if it doesn't become a global issue where all countries participate in this, people will move their illicit money somewhere else, where you don't have the requirement for public registers. My third point is to really advance the work on misinvoicing and to really continue to engage, I think the IMF, the World Bank, the OECD are three very important international institutions in this regard. Complete the unfinished work on transparency and cooperation in tax information. And again, bring in those from outside of the circle that have been working on this, because again, you need a global solution here, and you need that cooperation, and not just bilateral, but you need the multilateral work, which has started to happen. I will also add one thing that we haven't talked about today yet. And to me, this is the whole extractive sector. And I'll start by an example. I was meeting with President Kagame a few years ago, and he said, these three minds are closing in the next two to three years. Rwanda has never received one penny in royalties, in tax, or in concession fees. I said, well, why? Because the fine print in the agreement said that any net revenues here could be spent where we're exploring somewhere else to the extent that we're exploring, and of course, you're always exploring. And that is certainly one aspect. But then the whole movement of some of the high-end goods from the mining sector, again, is feeding what we're talking about today. So I would say that the whole area of extractive industries need to be not only supported, but in countries to some extent. I was asking President Kagame, why did this happen? Was it corruption? Somebody paid money under the table? Was it the lack of capacity of those who negotiated compared to who was on the other side of the table? All of the above or more?  But what we know is that this is not unique to Rwanda. And a number of countries have started to renegotiate Chile as an example, their agreements, because of the money leaving the country. And some of it also from extractive industry turning into illicit financial flows. So we can come back to that. I think my other point goes to that middle group that I was mentioning before, the institutions, whether it be the judiciary, the police, the customs, the tax, those who are the custodians in a way of what happens when money moves in and out. And in many countries, and I would venture to say in some of our western countries, the sophistication of the facilitators is much better and higher than those who are responsible to make this happen. And here I'm talking of prosecutors of judges or of police. So much is happening. It is better in a number of countries than it was. But we're very far from being where we need to be in terms of having the capacity to prevent, to detect, and to sanction illicit flows and the related areas. I'm talking of audit. When Ellen Sirleaf Johnson arrived in Liberia, she just arrived immediately after the whole debacle and the war. And one of the first things that she decided she would do would be to recruit a forensic auditor who is Liberian but had been living in the United States as a professional for a long time and said, come back home for five years and I want you to establish a strong audit responsibility in this country. And I saw with the result of what they were able to find, sanction, and then give themselves the means to prevent in the future. And that I'm using this as a small example. We've talked a lot about data today. It remains so important in countries around the world, in particular in developing countries. I was meeting once with the Minister of Finance of a developing country and I was asking him about data, about information that he had. The conversation was short. He said, I've got three economists and I'm responsible for national, provincial, and local financing. So he rested his case. But I think that that remains the data and the support to ensuring that information is available. We can talk about transparency, but if the information is not available, we're not getting anywhere. I would like just to make a few more comments. I think that the whole area of due diligence for the banks on knowing your clients, on ensuring that they have the preventive measures to stop illicit flows, to report it as they see it coming and make sure that it becomes part of the justice system to me remains a very big part of an unfinished business here. And we haven't talked much about the recovery of stolen assets again. This is very much an unfinished business and I have been in countries in Africa where the negotiations had been going on for decades on trying to recover assets at a great cost to that country, but with very little in result, if any. And we just have to think about Abasha. We can think of Marcos. We can think about a number of other world leaders whose money for their family or themselves is still out there. And I would also add that we need to spend quite a bit more time on trying to see how we deal with intermediaries. And our last panel spoke about some of that. And it's not just the lawyers and the accountants who are the facilitators. It's the real estate agents and it's many more. So that to me is and finally, I think that the work of civil society, of leaders of industry being brought on board of the academic sector of the international institutions will remain more and more important. If we want to make sure that Addis and the SDGs are not just dreams, are not just words, commitments that don't turn into action and to make sure because you need the global cooperation for this to happen. No one country alone, of course, we know can do it. And I'd like to go back to the fact that we need to engage the people of the countries as well. Find better ways of doing that. Provide constructive mechanisms and tools, not just identify the problems because then we're not doing better than the leaders who approved in Addis or who put forward their major commitments. So we need to see these as a reality. If we want to go back to the first point of the dedication of this book of dealing with the fact that poverty is such an injustice, but so and the lack of peace and the lack of security also is. Thank you very much. Thank you very much, Hugh Get. We hadn't mentioned political will prior to Hugh Get's comments, but I think it's always important to bring that up and also sort of puts things into light as much of a challenge as it has been to get illicit flows into the Addis agenda and the SDG agenda. We know what's ahead of us, which is trying to build that political will around the globe to really address this problem on the ground. Hugh Get also mentioned the issue of data, and that will be the topic of our next panel. Very good segue. Thank you for that. We talk about data all the time, not only within GFI, but within the broader NGO community, within the FTC and elsewhere. We always try to strive to get the best most comprehensive data possible to try to put out as accurate estimates as we possibly can. We're constantly asking how do we improve it? How do we broaden it? And that's what our next panelists will discuss. I'd like to ask Dr. Devkar, Anders Eggerskoff, Tamara Rosen and Porter McConnell up front, please. Thanks very much.


You know I've talked over and over again about the risk of eating too much red meat But if you still want your steak, so be it But not all steak is created equal so when you're at the store and you're buying your steak to go home and cook or Even if you're at a restaurant you have to know what to look for and what to avoid and that's why I'm here and I'm from Omaha, Nebraska the beef state and Let me tell you I know beef. In fact, believe it or not our high school prom We're always held at the Omaha Stockyards. No kidding.  Why? because that corn fed beef is Marbled with that succulent delicious fat that gives us the mouth feel that we're we associate with a great tasting steak And it makes the steak incredibly tender, but here's the problem corn is just pure Omega-6 inflammatory fats now traditionally cows that are fed on grass Have in their fat an omega six to omega three ratio of Anywhere from three to five Omega-6 parts to one part omega-3 fat and that's a natural ratio When we add corn to the mix that ratio can go up to 25 to 50 to 1 Instead of the three to five that we're looking for so that juicy tender mouth watering steak becomes an inflammatory Fat nightmare those fats actually Increasingly are becoming realizing as one of the biggest drivers of inflammation in our country particularly They turn into compounds called aldehydes One aldehyde that you're probably aware of is form aldehyde the things that we Preserve bodies with I spent a lot of time with around form aldehyde in medical school And I assure you you don't want to eat a steak that turns to form aldehyde in your body And yet that's exactly what it does now the problem with marbling and actually the problem with Grass-fed is there's actually no law no federal law that tells you what grass-fed means technically all cows eat grass for some part of their life if You then feed them grass for even one day as an example and then take them to the feed lot and Give them corn and soybeans for the rest of their lives Legally there's nothing to stop the company from telling this is grass-fed beef what good producers will do is They'll feed them for grass most of their lives But then to give you the mouthfeel you're looking for They'll take them to the feed lot and feed them corn and soybeans or corn only for about six weeks To fatten them up for slaughter and to give you that mouthfeel The way around that is to look for grass-fed and grass-finished beef now you see it in Some of the high-end markets you can certainly find it on the internet But if you're going to have your beef, that's what you're looking for because it's going to have much better proper ratios of Omega-6 to omega-3 So labeling is really deceptive and if you have any doubt Contact the company and ask what they do Now I'm old enough to remember that back in the old days, but even in Nebraska Meat's beef was very expensive and they certainly weren't an everyday thing and Cheaper cuts were what the average person could afford like a chuck steak or a round steak They were cheap because they had a lot of gristle and That gristle is actually where Gelatin and college and come from and it turns out our microbiome actually loves to ferment the Gristle and does not ferment muscle as well so inadvertently poor people people who couldn't afford a lot of meat and Long-lived people in general were eating these old tough cuts of meat and They weren't eating very much of it and they were doing one other thing I think that's particularly illustrative of how we ought to approach beef Lamb and pork in general So they cooked them all day long. My mother would cook a pot roast all day long for us and we'd have it actually multiple days and That gristle would break down and unbeknownst us. We were actually having gelatin and Collagen on almost a daily basis in these meats We also actually benefited from gelatin almost every day in the form of Jello Now Jello is not a health food folks But believe it or not almost every day of our lives We were having gelatin in the form of Jello. Don't rush out and buy Jello There's a lot of easier ways to get gelatin into your life but the point is we were also having a soup in those days called consomme and in fancy restaurants you even had chilled consomme as an appetizer and consomme was Simply high-end bone broth now don't rush out and buy bone broth because bone broth The arsenic and lead in bones is actually remarkably high so be careful when you're eating steaks and for goodness sakes please please please Avoid the steaks that have all the marbling you're much safer to have a filet That's grass-fed grass finish Then a ribeye even if it's grass-fed grass finish because it's going to have much less of the harmful fats Now something that I've been researching For my upcoming book and something I research in going to countries like Portugal like Spain like France like Italy is that these people particularly the poor parts of these countries use a lot of sausages fermented meats in their diet in fact the country with the longest lifespan is actually a tiny country between France and Spain called and Dora and Dorian's have a lifespan of 87.7 years pretty spectacular and Andorians do two things they eat sausages every day now. These are fermented sausages people are poor And they can't waste animals So they go they eat everything the saying is from snout to tail But to preserve these foods They actually grind them up and make sausages and the sausages are preserved by fermentation The fermentation does a lot of really cool things that you'll hear about in the upcoming book But among other things just to tease you fermentation makes these incredible mitochondrial Uncouplers polyamines and polyamines promote longevity in multiple ways the other thing that does is Is it breaks down some of the really health-destroying compounds that I write about in beef lamb and pork called new 5 GC So these folks Because they were poor Actually got tremendous health benefits by fermenting their beef products so sausages traditionally prepared From Italy from Spain from France from Portugal are actually pretty doggone good health foods But that doesn't mean you can go to the grocery store and say hi I want a pound of sausage meat or Fresh sausage and think you're going to get the same benefit All contraire These are not properly prepared and they will do you more harm than good All right, so if you want to have your meat Have it but in moderation of course Fill your plate with vegetables and make meat the garnish You should also try to decrease your consumption of meat for many reasons that you'll learn about in the new book But if you want to treat yourself to a steak every once in a while, I get it When my wife and I have a steak We buy a filet and we split it and we get the flavor we want But we're not killing ourselves in the process. I Hope you enjoyed this episode of the dr. Gundry podcast Make sure you check this one out too.


First of all, thank you everybody for sticking around for the last session of the day and you're missing out on free beer from HP, I guess. So what I wanna do real quickly was just talk about cloud and our focus is on delivering an open stack distribution to enterprise customers. And so we're gonna go real quickly through some of the highlights of why we think enterprises care about this. And then Rick and Cameron are gonna walk through a demo of some things that we've added into the open stack distribution that we think it makes it easier to be consumed in the enterprise. So I was asked by my CEO at one point, why cloud computing is important? Why do enterprises care? I mean, okay, we understand why Amazon does it, we understand why Rackspace does it, but why would an enterprise care about it? So we came over these slides as kind of just a little dramatization of a conversation between a business executive, line of business, and the IT. Business executive says, we need faster rollout of services. I'm getting killed by competitions doing things, I'm having to respond to the market. That's gonna cost you a lot of money, right? I mean, this is kind of how the conversation goes. How much? I don't really know. I don't have the visibility of my infrastructure that I might need. Well, come on, there's gotta be unused capacity. I hear about this virtualization stuff, and it's freeing up capacity, so you must be able to take advantage of that. Yeah, but I don't really know where it is. I've got this pool of servers out there, I gotta go in and look at each one of them in inventory. Well, you know what? I'd even rather pay by usage. And the IT manager says, no, that's not gonna happen. I just can't run the business that way. So the whole idea is once you go and implement cloud computing, then the conversation changes a little bit. We need faster raw out of services. It's later today, okay? Not weeks, days, hours, you know, minutes. How much? $1.50 or $2.50 cents an hour. You know, I can tell you exactly how much it's gonna cost. Same as Amazon. What if I have to expand or shrink the user base? It's fine, you're now in control. I'm giving you more control over the infrastructure that you can take advantage of. And I love you. Yeah, it's still not gonna happen, right? So what's really happened is the contrast here is that the IT manager in the enterprise has become a cloud provider. And that's really the core focus that they're looking at. But there's a follow on problem. If you do that, how do you give control, but some control, but maintain some control, right? I mean, you still wanna make sure that your line of business is deploying compliant images, security patches are applied in an appropriate fashion. And just in general, you wanna make sure that the line of business has got, make it easy as possible for them to consume things, but at the same time, keep some control. We've talked about, obviously we're here about OpenStack. We've been participating in OpenStack for a couple of years now. We actually are a Platinum member of the foundation. Alan Clark, who's a SUSE employee, is the chairman of the OpenStack board. He's also a longtime experience in the Linux foundation, the OpenSusa foundation. So he's a great experience in helping drive community, community development, community projects. We focused a lot on technical contributions around hardening, securing, improving Zen support. We've promoted OpenStack in the OpenSusa community. So all of our development now is done in what we call the OBS OpenBuild service so that our community members can more easily consume OpenStack as they test it out. But we're also delivering an OpenStack distribution. So this is kind of our 2.0 product that we're gonna start working on and Rick is gonna walk through some more discussions around that. All right, looks like we're scrunched up a little bit right there. So, well, SUSE Cloud 2.0, so we released our first version of SUSE Cloud back in August of last year around LinuxCon. And we are looking at doing our SUSE Cloud 2.0 in roughly the Septemberish timeframe. And there's some interesting things that we're looking to be able to include in that 2.0. And you can see we've got the color separation, the separating things out right here. And the colors actually indicate a few things. First off, the orange right here indicates the pieces that are part of the general OpenStack distribution. That's the part that's coming from the community that everybody's been doing a great job and we've been learning a whole lot about it. But there's more to it typically than just the OpenStack community code. There's a lot of moving parts that come into any kind of OpenStack implementation. We believe that we have a few enhancements and decisions that we have made to try to reduce the number of choices that you have to make as a customer to make things a little bit simpler. As we like pointing out, there's almost 800 different configuration options that you can choose in OpenStack. There's a lot of complexity. You can get paralysis by analysis. Some of the decisions that we've made, for example, is that we've partnered, we've taken advantage of the Crowbar open source project which was originally sponsored by Dell. And we are looking at establishing partnerships actually with, we announced yesterday, partnership with Dell to be able to offer a go-to-market solution of Dell hardware with Suiza Cloud to provide an enterprise-class cloud environment. Along with the Crowbar server and the services that come with it, we've made a choice to go with the RabbitMQ, messaging queue. We made the decision to go with Postgreffs as our back-end database to be able to handle all of the transactions back there. Then we have a couple of Suiza products that we feel complement these particular technologies and help you to get the most use out of them. The underlying one, of course, is our Suiza Linux Enterprise server which is the base product that we offer. The idea being that you can have an enterprise-class operating system that has all the testing, has all the security penetration testing, has all the auditing, has all the hardware certification, software certifications, all the things that keep you from getting fired for making a bad decision on your platform. Suiza Linux Enterprise server serves as the basis right there. We ship within Suiza Linux Enterprise server the Xen and KVM hypervisors, and as such, we support both of those as part of the open-stack offering. We are looking to expand those options in our Cloud 2.0. We are optimistic that we'll be able to get hyper-V support in there. We would like to get VMware support in there if the code is up to snuff and it looks like it's a supportable configuration. We'll have to see how things progress on that particular front. Some other products we're actually gonna talk about a little bit today. Up in the very top here, we have a Suiza Manager. Suiza Manager is a tool that you can use to be able to actually manage all of your RPM-based Linux distributions from a single pane of glass, whether they are your private cloud guests, any public cloud guests you have, your cloud infrastructure, your local infrastructure that's not part of a cloud, it gives you that single window into all of it and lets you have a hybrid environment that you can manage in one place. The other piece up here that we're gonna also show you today is one called Suiza Studio. A lot of talk has been made about going... It's your laptop. It's not my laptop. I don't know the password. Oh, okay. All right. A lot of talk has been made about getting OpenStack up and running and all the bits and pieces you need to get that done. But there's also another key point is that you don't just turn on the cloud just for the sake of the cloud, you want to actually run something on top of that. And typically the process of creating these images that you're going to run is a fairly cumbersome process and Suiza Studio helps you to really streamline that process and make it a lot easier for you. And then there are partner products. So we maintain all the OpenStack APIs. We don't change any of them, anything like that. So any kind of partner that... Any kind of third-party product that integrates with OpenStack will be able to integrate with Suiza Cloud as well. And we have a couple of them that we've worked with around billing and portals and governance and these kind of things. So we've got a number of partners that we're talking about. We already talked, obviously, about Dell. We have ones like Cloud Cruisers and Stratus and a couple other ones that are out there. There are several of them out there. All right, let's go ahead and move on. Okay, is there a question? So the question is, do we use MySQL at all for the database? The answer is no. We use Postgres. The engineering team is right there. They can answer that question for you if you want. Who wants to be with Oracle? That's the big one that I always tell people is nobody wants to be beholden to Oracle any more than they absolutely have to. Postgres has some really nice features in terms of HA that make it a really good scalable database. There are some other features that you can make the argument that Postgres is a little bit more enterprise-ready than MySQL is. Not to say that MySQL is bad, it's a great database. Okay, yep, good question. Thank you for bringing that up. Okay, so I'm gonna have to move a lot faster right here, but this is just kind of a quick overflow of how we see the overall lifecycle of a cloud deployment going. So basically you can have developers who are creating code applications. They're getting to check their code in. Pete already mentioned we have the OpenBuild service, which is a tool that we sponsor where you can compile code for any major Linux distribution and all the most popular hardware platforms all from a central place simultaneously. And from those custom packages that you can be creating, you can run those out into SUSE Studio or into SUSE Manager and then be able to use those to create the workloads that you need to be able to use on your cloud. All right, so we've mentioned SUSE Studio a little bit. I'm not gonna go into too much more of it simply because we're going to actually see the demo and you're gonna actually see us do some demo without a net. We did not learn from HubSpot. We're gonna try and do live demo here today. So let's go ahead and do some showtime. And I will increase the font size, I promise. Okay, where's your studio instance cam? Oh, that visible in the back there? I'll take that as a yes. All right, so SUSE Studio is a web-based tool. We do have a version of this that's up online at susastudio.com. That is kind of our community version where we sandbox things, we try out new ideas. It's not necessarily something I would want to base any kind of production environment on, but it does give you a great opportunity to be able to kick the tires and see if it's something that will work for you before you invest in SUSE Studio on site for your data center. All right, so let's go ahead and let's go ahead and create ourselves a new custom appliance to be able to do something. We're gonna start off, we have a number of different templates you can start from. This one is called Juice, just enough operating system. It's literally just barely enough OS to be able to boot your system, and that's it. So in and of itself, it's not terribly useful, but what it does is it gives you the ability to then take that bare bones and layer on only the pieces that you need to be able to accomplish your task. This makes your file size much smaller when you're dealing with your image. It also means that from a security and auto perspective, you have less exposure and risk because you don't have to worry about a specific service being turned off on the system if it's never on the system to begin with. All right, so let's do the 64 bit. All right, you sir out here in the front that was asking the question, what's your name? Prakasha? Oh, let's see if I can do this like that. We're not gonna go there. What does it say, Prakashas? No, it's a man instead of an H. I gotcha. Wonderful server. All right, so Prakashas wants to make a wonderful server. In this case, we're gonna do something simple. We're just gonna make a simple web server that we can come out here and just so you can get exposure to the ideas. Let's go ahead and let's see if we can find a way to. All right, we can see most of what we need to see right here. Okay, all right, so we can come in here. We can see the name. Let's go ahead and come into our software. We have a number of different ways you can add in software. You can add in if you have your own custom repositories, whether that's with the open build service or some other method, you can add them in right here. You can upload single RPMs. You can simply do a plain text search for whatever it is you want. We also have what are called patterns. A pattern is a pre-built configuration, a compilation of one or more packages that fulfills a specific role. So in this case, we're gonna do ourselves a lamp stack. Actually, we'll search in here. A lamp, we're gonna build ourselves a lamp server, just a simple Linux Apache MySQL PHP. We can look in here and we can see a plain text description. We can also see that by default, it's just the Apache web server, but we have these other additional recommended software we can add as well. Right now, we're just gonna do the basic one and add it in and we're gonna have ourselves in the Apache web server. We're super complex here. All right, so let's go ahead and configure it up. So we can go ahead and set things. We can set our time zone. We can enable our firewall if we have it, do the network if you're gonna use local authentication. You can set up your users and passwords right here. This one is surprisingly popular. Lots of people like to be able to customize. So you can select a logo. So if you have your company logo, you can upload it and put it in there.  Make sure it's a good hair day before you do that. I'll leave it up to you as to what your definition of a good hair day is. On the server side right here, if we had set this up to say a Postgres or a MySQL database, you can actually upload the MySQL dump file and set up your users so your database can be pre-populated and ready to use as soon as you launch your workload. I don't want to show this off the screen. Let's see if we can get to it. No, can't get to it. Well, okay. There's another button over here where you can actually click on the scripting and be able to see where you can do scripts that can be run at the end of the build or whenever the appliance boots, whether that's the first boot or every boot. So you have a lot of flexibility in terms of how you can configure up your system. You also have the ability to upload overlay files. So if you have a tarball-based application or if you have specific config files for say, authenticating to your active directory infrastructure or any of these kind of things, you can upload these configuration files and make those be part of your appliance. And then when we get to the build section, this is actually the kind of fun part, where you can see, you can build this for pretty much every platform under the sun. In this case, we're going to be doing cloud stuff. So we're going to select the SUSE of cloud slash open stack slash KPM and go ahead and kick off our build. And this is going to build us a QCOW2 image. You can see that we have a lot of other different options. Now in the interest of time, we're not going to wait the five minutes for that to build. We're actually going to go over and look at another one that we've already built while we're waiting for that one to finish off. So let's come over here and show you the next neat little thing right here. So we've got our appliance we've already built over here. You can see we've got a feature over here called test drive. What test drive does is it'll actually launch up a copy of this image on your system so that you can then log into it. You can activate networking and SSHM and copy in large quantities of files. And it'll keep track of all the file system changes that are made during your session. And then you can go in and grab those changes and pull them back and make them be part of your overall appliance.  You get a lot of control. We're booting up over there. Okay, so while that is booting, let's come back over here and see if we're how we're doing on our build. In just a moment here, I'm gonna turn it over to Cameron who's gonna walk us through the actual cloud portion and seeing how this image that we have is going to be consumed on the cloud. Got a spinner. Everybody cross your fingers. Demo without a net. This is a lot of stuff to be running on one laptop, so. It's taxing that CPU on there. Yeah, we're picking it up. All right, I'm gonna go ahead and turn it over to Cameron while this is building. And he's gonna walk you through the actual cloud portion. Live demo. Gotta love it. So, we're gonna start out with a couple of things here. You're all familiar with OpenStack, of course. And so let's go ahead and open up the SUSE Cloud dashboard here and see how we SUSE-fied this little dashboard. Pretty common dashboard that you've all seen. In here, we have some images that have been loaded in there. As you've seen from Studio, this is actually one of the images that was dropped in there in a little bit. As this is building out. Actually, I'm gonna bring this down and see if I can't get this thing to fit on the screen just a little bit better. SUSE Cloud is getting hammered right now. Her studio is getting hammered. As you can see, that CPU is just chugging. This will probably help if we shut down Test Drive. So, you can imagine this whole, as we're talking about this SUSE Studio stuff and we're creating images and building your image so that it can be plugged into a cloud environment, namely OpenStack in this case. Think about this whole life cycle of these tools working together with the cloud. Building your image, having that image with everything that you want on it, your application and whatnot, and having that image automatically deployed right into your cloud environment. And that's exactly what we're doing here. Eventually, it will actually show up right under the images within this dashboard here. I've created a user in here. His name is www.joe. Now, there's a couple of things I've done with Joe, web user, and if we go ahead and log out of admin and log in as web Joe here. Can you increase the font size again? Sure, yeah. We're gonna log in as admin and reset as password. Okay, we're logged in as Joe. He's a tenant on the web team and he's going to start up an instance that we've actually created in our studio environment. And so we're gonna take a look at the images that are available. We have one that's Slesi 11 SP2. Soon we'll have one that is an entire LAMP stack that might be better for Joe web user. And we'll go ahead and launch this particular instance in the cloud. And we're gonna name this summit 2013. We're gonna launch that in the cloud. So it's starting up, spawning the KVM in the background, finding a compute node that's suitable for the size. It's, of course, a tiny one. And he's gonna start booting up. And during this boot up process, there's a couple of things we dropped into this image. This particular image contains information such as the necessary functionalities to plug right into SUSE manager. So when it boots up for the first time, he's gonna automatically register himself to the SUSE manager's management framework. So now we've created an image using SUSE Studio. And now we're plugging it in to our management framework as well after we start that instance in the cloud. So now we can do several things with that. This takes just a minute to actually boot up. Let's see where he's at. He hasn't quite got any networking yet. Any questions at this point? So he's asking what kind of control do we have over the images that we're putting up into this cloud environment? Is that right? So the control is with Mr. Joe user here. Whoever you're designating on your team to take care of those instances in your cloud environment. That's who's got that control. And you can go in there, you can stop those instances, terminate them at any given time. I think his question was on the images. So if you're building multiple images, how do you clean those up and how do you control where those go? Yeah, so that's a really good question. So on the images, when SUSE Studio builds an image, we have what's called a webhook on one of the control node that basically plugs in with Glance. And it's continuously listening for information from the Studio server. And when it gets information that it's got a KVM build that's ready to go, it will go out and grab it. If it's any other kind of build, it won't touch it, it'll just leave it alone. So it goes out and grabs that and then uses the Glance commands to import that into your cloud environment. So you can actually go in to the OpenStack web UI and go in and remove those out if you need to at any time. So you might see multiple versions in there. If you're building a new version of it, you'll see a new version in there if you've created an extra version. So you can also modify the listening scripts. For example, with the versioning scheme, you can set it so the only point releases, don't get automatically pulled in. You can have a logic such that only a .o release gets pulled into your cloud. It's very flexible in terms of what logic makes sense for you and your particular organization. Say that again? Upload, yes, absolutely, absolutely. If you ground them that within Keystone. Right, that's a good point that you bring up because SUSE Studio isn't just for our SUSE framework. If you have a framework that's already KVM-based, OpenStack-based, you could plug SUSE Studio into that. There's a question in the back? Is that Doug? Oh, two minutes. So Summit 13, the instance is already booted up here. And so he should have registered himself with SUSE Manager. Let's go over here and see what's going on with that. Go ahead and log in to SUSE Manager. Oh, that's the wrong link, sorry. And let's see what systems we've got available in here. We've got some cloud VMs and we've got Summit 13. He's got himself registered to SUSE Manager. And so this whole life cycle comes into play now. We have built an image. We have configured that image to be plugged into SUSE Manager. And now we've put that image into the cloud and he's booted up. And now you can see him being managed in SUSE Manager. So we have these life cycle of tools working together with OpenStack to be able to give you that life cycle management. And let's see if our image is built yet. Over here in SUSE Studio. Studio. Studio is doing so well right now. Anyways, he's probably stuck building and building at this point. Yeah, he's pegged on the CPU. Once he's done building, that will eventually slide right into the OpenStack framework there and you'll see it as an image. So if you wanna stick around after we're done here, then you feel free to come up and we can show you and you can see it and finish off and show up. Okay, that's what we have for you today. Kent, do you wanna go ahead and do the drawing? Okay, we got a little card for everybody. Everybody got your... Everybody else? Who's gonna be our impartial observer to draw this? Who do you mean? Who do you mean? Who do you mean? Pretty much for coming.


Hello everyone, I'm Leslie McVane, the Development Director at Portland Media Center. These are really hard times for all of us. We're all affected. We're all trying to do the best we can to keep going and we at Portland Media Center are trying hard to keep you informed and keep on the air so that you can have some programming to watch while you're at home, perhaps, hopefully, you know, distancing yourself from others at this really crucial time.  That's one of our missions here is to make this media available to all in our community and this is one way we can do it. So if you have something you'd like to share with us, please send it, you know, through email I guess at this point if I find out a better way for you to do it, I will let you know, but send it either to Josh, J-O-S-H at Portlandmedia.org or to Tom H at Portlandmedia.org or to me, Leslie Mack, L-E-S-L-E-Y-M-A-C at Portlandmedia.org. We will do our best to share some of your ideas, some of your frustrations, some of your concerns and some of your joys during this time. We hope. So please, please keep in touch. This is a way to do it and this is a way we can share on our social media page and we can also share it through our television station 1303 and 1301 and also online at Portlandmedia.org. Thank you. Stay well. Stay at least six feet from people. Pay attention to the warnings about going out and possibly infecting other people. We will see you in better days in person, I hope, but for now let's see how this works. Thank you and stay well. Bye-bye.


Okay, so hello, my name is Alan Jude. I'm a free BSD and open ZFS developer and Today I'm going to explain to you like you were five how the ZFS arc the adaptive replacement cache works So like I said, I'm a free BSD core team member and a developer as well as an open ZFS developer And for my day job I work at Clara, which is a free BSD professional services and support company So today we're going to talk a bit about how The caching system in ZFS works So we'll start with a very basic overview of what ZFS is just in case you haven't encountered it before and explain Why you should start using it and then talk a bit about How caching works in general and then why the way ZFS does it is a bit different and then we'll get into more advanced topics like compressed cache So starting from the beginning What is ZFS? The biggest difference between ZFS and most file systems is that it combines The concept of the volume manager something like the Solaris volume manager or LLV or LVM or MDADM into the file system So now you Take a number of disks and create a pool of free space And then you build file systems on top of there and each file system only takes the free space It needs basically each file system is thin provisioned So all of your free space is available to any one of the file systems you create on top And so you can create hundreds of file systems and they each only need to take the space They need from the pool and then you can expand the pool later And it can also create what are called ZVOLs which are block devices Which you can then put other file systems on top of or use with something like NDB or iSCSI Another big difference in ZFS over most file systems is that every block of data that's written to the disk is also checksum And so when you read the data back We verify the checksum and make sure that the disk hasn't Accidentally scrambled the data and ensure that when we give data to an application It's always in exactly the same form. It was when it was written to the disk and Another feature that will come in to play during the talk is the transparent compression So optionally you can enable compression in ZFS and then all the data you write to disk will be compressed first In the past this wasn't that useful because of the performance impact With things like gzip but with newer algorithms like LZ4 and Z standard the compression speed is so good that It's still faster than your your disk even if it's an SSD and so there's no performance penalty Comparatively to having the compression The ZFS is also a copy-on-write file system So it never overwrites data in place It always writes the updated data to a different location on disk and then updates the pointers This way if the system crashes or the power goes out or something happens to the file system When it boots back up it finds the newest version of the file system where all the checksums match and mounts that So you never have to run fsck or anything to where the file system could be damaged One of the reasons it can do this is because it's copy-on-write If you're in the middle of updating a file when the system crashes or the power goes out You've not overwritten half the file and not yet overwritten the second half because you're always writing data to the new Place so the original version of the file still exists And then the last part is to make administration easier Each file system has a set of properties that allow you to control Various things about it like if it uses compression or not and these are inherited as you create child file systems So it allows you to very easily control all the parameters of the file system But the main advantage to being copy-on-write is that you can have instantaneous snapshots Unlike snapshots in LVM. There's no performance penalty to having the snapshots Because all it's doing is not freeing the data when it writes the new version You get very it takes no time to take a snapshot and having a snapshot doesn't impact the performance of the file system So this allows you to freeze this a file system at a point in time and be able to go back and reference it later or to actually serialize that and replicate it to another machine and This doesn't take any additional space until you actually overwrite a block So if you take a snapshot and then don't modify the file system It won't actually consume any space and when you do overwrite files only the blocks you actually overwrote We'll start consuming space in the snapshot But snapshots are read-only But there's another concept called a clone of a file system Where you can make the a writable version that still shares space like a snapshot does which effectively allows you to fork a file system so you can Take an image of a database or something before you upgrade it But fork it and then have the other version Very useful for doing development on databases and so on where you could snapshot a version of the database Test it in development and then throw it away. Well the whole time not overwriting The production copy of the database, but while sharing blocks, so you don't need twice as much space so caching Computers have multiple different tiers of storage right you have very high-speed storage like the L1 cache that's built into the CPU and then you have Slightly slower storage like memory and RAM and then now we have things like non-volatile memory Which isn't as fast as your volatile memory, but is still faster than your fastest SSD And then you have disks that can go all the way down to being really slow. You think like SD cards So back in 1946 when describing computers Von Neumann said We are therefore forced to recognize the possibility of constructing a hierarchy of memories each Having greater capacity than the preeding one, but being less quickly accessible So the point of a cache is to use that smaller but faster memory to keep the most commonly accessed data So that we spend less time going to slower storage But the cache is just a copy so we can discard that data at any time unlike on the regular hard drive where we don't want to Just be throwing away data just because you're not using it But as you go into these faster and faster tiers the amount of storage you have gets smaller and smaller And so you have to decide which data you want to keep in the cache Because that faster storage is closer to the CPU is precious So you need some kind of algorithm to decide which data you want to keep and which will be the most effective for you and so we use The main memory of the system usually as the biggest cache and this is why most operating systems have a buffer cache So almost all caches that are in use today are based on the LRU algorithm, which it means least recently used So the cache is basically a list of the commonly accessed things every time you access something It goes to the top of the list and once the list is full you delete items from the bottom to make room to put a new item at the top But that algorithms from 1965 and has stood up very well, but isn't necessarily the only way to do it So in this diagram you can see we have in the first step We have data block a and then But the rest of the cash was empty and we've used the first block So we've filled up one quarter of the cash with that and then in the second step There's a data block B and then see and then we just keep adding to the cash until it's full So once we've added D here in step three The cash is now full so when we want to add a fourth item to the cash We have to find a victim to evict from the cash So we want to find the item that was used the longest to go Because it means it's the one we're least likely to use again So in this case we overwrite a with data block e And then in the next step what happens is we actually access the data block D a second time and So we refresh it and give it a newer generation number So that when it comes time to add data block F We decide to overwrite Again the oldest item in the cash And we just keep doing this and always keeping the most recently used data in the cash Because it has the highest chance of being reused again So there's some pros and cons to this simplistic algorithm It's usually implemented as a doubly linked list which means it's very low overhead and it's not complicated to implement You know some LRU caches are implemented purely in hardware, so it had to be very simple And the advantage is the locality principle says that if a process is visiting a location in memory It's likely to revisit that same location or somewhere very near it It's more likely that it will come back to the same place than to go some any other random place The downside to something like LRU is it Somewhat ignores frequency So just because I access this data once every five minutes It doesn't mean it's more precious than this other data that I access many times a second Both those will stay at the top of the list and not be purged But it doesn't have any consideration for how frequently you access it just how often or how How recently you've accessed it? The other problem is it doesn't adapt over time So it doesn't look for patterns or anything like that And the biggest downside to something like LRU is it can be disrupted by a large scan So if you take a backup of your whole hard drive or something then your small LRU cache You're just constantly going to be filling up with files you're accessing one time only And then the cache basically becomes useless because during a backup You're going to read every file once and never read the same block twice And so it doesn't consider Recent history to try to improve the algorithm So there's a second algorithm that came around in 1971 called LFU Which is based on the frequency that you use the data rather than the recency So it's the same idea except for instead of keeping The time stamp of the last time we access the data. We just keep a counter of how many times we've accessed data So each time we access a page or a file of memory then we increase its counter and the cache is basically a sorted list and When it's time to make room in the cache we delete the item that has the lowest hit count So when the cache is full we evict the least frequently used file So unlike an LRU cache this cache is immune to the scanning problem, right? If you take a backup or read a whole database Because you've visited each of those blocks once they're not going to move blocks that have been accessed many times out of your cache But so that gives it the advantage of being immune to large scans and providing Very good performance if you're very frequently accessing the same data Because the advanced locality principle says the probability of visiting a location Increases the more times we visit it So if we visited this location a thousand times, it's more likely we'll visit the thousand one time then visit some other location But because you now have to keep the list sorted it's higher complexity to implement And again has the problem of it doesn't consider recency. So just because I accesses this file a million times Last week and I haven't touched it in a week It still has more more hits in its counter than any new data And so the cache doesn't keep up with what you're actually doing on your system very well So it can accumulate data that you used to use a lot, but aren't using anymore And then your cache hit ratio is less because it's full of data. You're not even using anymore so in 2003 Two researchers gave a paper at the use makes fast conference Called the adaptive replacement cache and The concept here is to combine the best of the LRU and LFU caches together Along with some novel tweaks of their own to make the most efficient cache possible So you take your cache of whatever size the default is the variable in the algorithm is called C And then you partition this by default you start with partitioning it in half So you're going to take however much memory you have to cache with half of it will keep track of the most recently used files because The LRU algorithm has proven to be pretty good in every OS buffer caches used in But the other half will be the LFU so that frequently access files can stay in the cache and Then in addition to those two we also keep ghost lists which I'll explain more in a second and Then Each time so we have these two caches and they're full up And then when we have to remove an item from the cache to make room for something fresher What we do is we throw away the data, but we keep the the hash key and put it on this ghost list So we keep track of items that were recently in the cache But we've thrown away the data because the cache was full So we have the four lists the LRU and LFU and each of those also has the ghost list Which doesn't contain any data, but just has a list of the hashes of things that we've Recently evicted from the cache So if we request a page and it happens to be in the LRU or LFU cache Then that's a hit we can pull the data out of RAM instead of having to go to the slow disk And the cache has done exactly what his job is to provide faster access to that data but net Once we have a hit That means that that files was already in the cache because we used it once and we've now used it a second time so it gets moved from the the LRU half of the cache To the LFU side because it's now been accessed more than once so we put it on the frequently used list instead of the Recently used list, but if a page was not On either of those two lists then we look at the ghost list Which means If it's on the ghost list it means that we used to be in cache, but our cache wasn't big enough So we had to throw it away and So that means if the cache had been just a little bit bigger The thing on the ghost list would have been in the cache and we would have had a hit and So we use that information to actually modify how we split up the cache Right by default. We're starting with half recently used and half frequently used But if we find out the block we just tried to read from the cache Used to be on the recently used list, but we had to delete it to make room for something else We decide that what we'll do is we'll make the MRU the most recently used cache slightly bigger Stealing some of the space from the frequently used cache And we keep doing this every time we read a block that would have been in the cache It's a cache had just been a little bit bigger So every time We get a miss that could have been a hit if the recently used list was bigger We steal more from the frequently used list and make the recently used list larger Just sliding the slider from 50 percent towards a hundred percent But if the hit was on the frequently used ghost list then we slide that slider back the other way And this is what makes the cache adaptive as you your usage changes on your system. It's Decides which cache is more effective You know if you have a database where you're using the same data frequently the frequently used cache will get bigger and Give you a better hit rate But if you very don't very often use the same data over again But do use data that you've recently accessed then the recently used list will grow and And this allows The arc to give you a better hit rate than either An LRU or an LFU cache would have given you by giving you the hybrid of both that best suits your current needs So each time We find a block that could have been in the cache But isn't we change the size of the cache until we find the perfect mix of the two types of cash That'll give you the best hit But ZFS doesn't stop there you know you only have so much memory in your system and sometimes is Physically not possible to install more. There's only so many slots on the motherboard for more RAM So ZFS has a level to adaptive replacement cache So this allows you to use a fast storage device like an NVMe or SSD to cache data Even when you're running out of memory So rather than waiting until it's about to kick an item out of the cache to make room for a new one It watches the bottom third of that the two lists and as an item gets towards Eventually it's going to get kicked out of the list. We copy it and write it to that SSD and then We can remove the data But keep again the hash in the Cache and just a pointer saying this data is now on this SSD at this address and It allows us to go from only having you know hundreds of gigabytes of RAM to having terabytes of level 2 arc So it still takes a small amount of RAM, but it doesn't take as much as buffering the actual data in memory To avoid the problem with large scans again where if you're reading the entire data set You don't want to write copy all the data you've only used once out to the SSD There's a rate limit on how fast you write to the SSD So it won't take everything that's about to fall out of the cache and put it on the SSD It'll only take a fraction of it to try to give us the best chance of Keeping the data that you're actually going to use again on that SSD Because you know your SSD has limited write cycles It will only be able to store so much data before it wears out and So we wanted to put a limit on that and so with this we've now made a very large cache That optimizes itself based on our workload even as that changes over time to get the best possible cache at ratio But it turned out even that wasn't always good enough So one of the companies that works on ZFS delfix had a customer who had a database that had grown to about a terabyte But they found that as soon as the database exceeded the amount of cache that they had to keep that data in RAM The performance got terrible But it wasn't physically possible to put more than 768 gigs of RAM in the server. It only had so many slots And so what they came up with was the compressed adaptive replacement cache Since ZFS is already doing transparent compression as we write data out to disk We first compress it and then store it on the disk in the compressed form ZFS has an optimization where if after compressing it we find that there's not a gain of at least 12% We actually throw the compression away and store the data on disk Uncompressed so that we won't be spending all of our time trying to uncompress data that barely compressed But this means that the data on disk is Almost always compressed especially when it's a database because you know if your Database is full of just text and numbers and so on it can compress four or eight times to one And so the way the art worked before this new invention We would read the data off the disk Decompress it and then catch that decompressed data the original form of the data And then when the application needed it we would return it With the compressed art optimization. We basically delay the decompression step So after we read the data from the disk restore the compressed version in the cache And then when an application wants the data, we take the copy from the cache and memory Decompress it and feed it to the application So this means when you access a file many times even though it's coming from the cache every time You're still decompressing it each of those times But because the LZ for compression that's used by default for this is so fast and can decompress at Two to ten gigabytes per second per core depending on the type and speed of your CPU That's going to be faster than you probably could have read off the disk anyway and so isn't actually going to introduce additional latency and Storing the data compressed can actually give you lower latency because you're reading fewer bytes off the disk So with this step we now are storing the data in the cache in the compressed form where it's somewhere between 50% and hundreds of percent smaller So now the cache can hold that much more data and we can get that much higher a cache at ratio So with this change that one terabyte database now Could fully fit in memory in only 460 gigs of of bram leaving them a couple hundred gigs to grow in the future and still having a hundred percent cache at ratio Every time they wanted to read any block in their database. It would always come from the the cache in memory There is a slight optimization to this as well There's a second level cache called the debuff cache That will keep commonly decompress blocks Basically when you decompress a block would keep the uncompressed version in a very small buffer in memory So that we don't end up decompressing the same block multiple times per second So while we can end up decompressing it many times we won't do it many times per second because that would be wasteful But then even then the algorithm wasn't perfect The other thing that the arc now considers is metadata Especially in ZFS where you need to have things like the hash of every one of the blocks and the tree of indirect blocks that make ZFS able to do copy-on-write means there's a lot of metadata about each block on disk And you need that metadata to be able to read files very quickly But at the same time you want your cache to also actually contain some data and so by default ZFS limits the size of The amount of metadata it will keep in the cache to one quarter of the cache Although you can tune that depending on your workload if you have a very large number of files and you scan them frequently You might actually want to keep more metadata and less actual data But in most cases you want to limit the amount of metadata you keep in the cache to avoid it starving Taking all of the cache space and leaving none for actual data Compared to the original arc algorithm described in the academic paper ZFS's implementation had to be a bit different For a number of reasons the first one is the size of the cache is not fixed ZFS's arc Can be shrunk if the system needs memory by default ZFS will take a large fraction of all the memory on the system up to 95 percent But depending on the machine if it's your laptop or something that your browser is probably taking a good chunk of the available memory on your system And so ZFS has the ability to shrink its cache and give memory back to the operating system for user applications So unlike the original arc algorithm described in the paper This cache is dynamically sized as well. So in addition to changing the partition value the p-value of how much is for MRU and how much is for MFU The actual size of the cache can grow and shrink over time And unlike in the original arc paper You can't evict any data that's in the arc Sometimes the data that's in the arc is currently being used right if you're actually if someone's reading the data out of the arc right now Or is locked the buffer then we can't go free that one just because it's the oldest on the list So the algorithm in ZFS had to be a bit smarter to be able to deal with the fact that some blocks will be in use And we will not be able to Evict those ones and we'll have to evict, you know the third least frequently used file because the other two are still in use And Lastly compared to the original out algorithm Where all pages in the cache were four kilobytes? The arc in ZFS is built out of disk sectors Which can be anywhere from a 512 byte like a regular hard drive or the record size in ZFS goes up to 16 megabytes So you can have very different sized objects in the cache And so the arc algorithm had to be a bit more complicated than the original one described in the paper And so when I talk about the the compressed cache in ZFS, I often asked it asked questions about comparisons to things like Compressed memory and swap cache. And so I thought I'd talked a little bit about the differences there So most of the memory compression schemes like swap cache that are out there Are generally taking the idea of okay when the system is running low on memory We're going to find some bit of data that's in memory that we're not using very much and compress it to free up space To keep other stuff in in memory the problem with that is When you're in the situation where you're running low on memory the last thing you want to do is need more memory To run the compression algorithm and to be able to copy the larger data into a smaller buffer to compress it So it's really the worst possible time to try to compress memory is when you're running out of memory And so it's a problem a lot of the memory compression schemes run into Specifically reacting when the system is already under stress is only going to make that stress worse not better Whereas what the compressed arc is doing is taking the data that was already compressed when you wrote it to disk previously and just deferring the decompression step into later and so The compressed arc is only doing compression as you're writing data to disk Rather than every time you're modifying memory so compared to swap cache compressed arc is just taking advantage of the compression you've already done and Using decompression which is much faster and cheaper than compression To optimize the amount of memory that you have available for caching because again Free memory is wasted memory if you have free memory You might as well use it for cash because if you end up needing that memory you can always kick the things You know shrink the cash to free up that memory again and use it for the user application But in the meantime The larger the caches the higher the chances that the data you're trying to be read will be in memory instead of on the slow disk and so the last thing I want to talk about a little bit is Tuning in ZFS. There's a lot of different knobs you can tweak But depending what you're using ZFS for there's a couple of highlights I'd like to point out so if you're building something that's specifically dedicated to being a file server Which is basically the defaults in ZFS Then you want to use as large an arc as possible if it's a file server It's quite unlikely to be running a bunch of other application But you're not going to have web browsers and databases running on this machine So you'll be able to use almost all of the memory for the cash But if you have very very large numbers of files Then increasing the size of the metadata cache from 25% to say 50% Will mean that you can cache The metadata of those files more and it means doing things like directory searches or trying to run our sync On large directories will be much faster because the metadata will all be resident in memory and with the compression the metadata compresses extremely well and You'll see compression ratios of three or four to one and so you can fit a lot of metadata and not that much memory and Optionally if you have a working set that's larger than the amount of memory you have For the cache then you can consider the L2R using an SSD or NVMe or NVDim device To keep commonly used files somewhere a little faster than the spinning hard drive you might be using but That's maybe not as cheap as buying terabytes of RAM But if you're doing something like block storage if you're going to do the NDB or iSCSI like the previous talk was talking about Again, you probably want to use most of the memory on that for caching Especially if you're backing VMware or something like that or Zen But one of the biggest considerations is the block size Especially when you're using the block device You have to match it up with the layout of your disks to make sure you don't use a lot of padding or have Read modify write cycles if using a very large virtual sector size on the block storage But the guest OS inside the VM expects to be using you know 512 or 4k sectors Then you end up reading a 32k block Modifying the middle of it and then having to write that whole block out again For databases, there's actually two different approaches depending on your workload The Previously the recommended way to do it was to keep the arc cache quite small or even By using the per file system parameters caching only metadata and not actually caching the data and Then using the databases buffer cache like MarioDB or Postgres would use and have the database take care of the caching Because it has more knowledge about the rows or whatever in your database and what's in use and what's not then the file system is going to Because it will Understand your usage better However, once the compressed arc was invented Now you have the advantage of if the compression On the cache is going to be great enough that Having that much larger of a cache is actually going to be a win You might instead decide to have a medium or large arc and keep the databases buffer cache quite small and rely on ZFS to do the caching because with compression they could cache two or four times as much data then the Databases buffer cache because of its higher cache at ratio and Lastly if you're going to run VMs on the machine co-resident with ZFS for your hyperconverged Type setup then you probably want a relatively small arc because it's likely you're giving most of the Ram on the hypervisor to the individual VMs And if you have ZFS trying to use as much RAM as it can and then having to give some back For the guest and then fighting back and forth. I can create more contention and it's That's good that way. So in that case it probably makes sense to have a specific upper limit set on the size of the ZFS cache and Dealing with that The other problem you can run into is double caching You know you're caching the blocks in the arc on the host for the hypervisor But then the OS inside the guest is maybe running ZFS or maybe not But using the OS buffer cache and ends up caching that same data and now you're keeping that data in RAM in two places Which obviously isn't as efficient as keeping it in only one place So now I'm going to take some questions for a bit and then we'll see what else we need to talk about Anybody have questions? How much would you recommend the ZFS file system to desktop users? Sorry, is it ZFS applicable at all to desktop users? Yes I'm giving a talk that will make a lot more sense for that at five o'clock in the BSD dev room But the reason I run ZFS on my laptop Is for a feature we call boot environments being able to snapshot the The file system before I upgrade my packages or upgrade to a newer version of the OS Means that if it goes wrong, I can just roll it back So for example, I updated the software on my laptop and then I came to go do my presentation and then it didn't work I could just reboot select an older version of the OS from the free BSD boot menu based on keeping those older versions as different ZFS file systems and just instantly revert To an older version of the OS But my home directory is a separate file system and so it didn't go back in time So my slides were still on the laptop even when I reverted the OS So, yes, I find it extremely useful on desktop student and laptops I perfectly understand how the compressed cache works when an application uses reach to read from a file, but how does The compressed cache deal with applications that try to memory map use huge ranges of files to access it directly So when you end map the file, it will deal with the page faults individually. So when you go to read the block, it will Copy it decompress it and and write that version into a different region of memory where the end map will be backed by So just as individual pages fault when you're trying to use an end map the first time you try to read them It will get the data from the art decompress it and put it there so that you can retrieve it. Yes Yes, and map is a case where you can end up with double caching in ZFS even without the compression actually because of the way The compression or the way and map interoperates with ZFS Because ZFS the data you have in the arc needs to be read only If you write new data, it goes to anonymous buffers that then get compressed and then sorted in the arc separately and so M-map and ZFS sort of not They don't use the same buffer cache and so you do get some double caching there. Yes Okay, so I think you refer to SSDs for a slot devices so for L2 arc Yes, we're out of and you said that they are varying out Which is obviously true but is there also some optimization for other memory technologies like 3dx point for example Which does not bear out that fast and you can actually write a lot of petabytes So there are two tunables for that the first one sets How many megabytes per second you might want to write to the device and then there's a second one called the boost which is Since starting up the device hasn't ever been full yet Then the boost amount of megabytes per second is added to the limit So by default if it's only gonna write say 20 megabytes per second out to the SSD But we're just freshly booted in the SSD has never been full yet Then we might add a second 20 megabytes a second to that so that we fill it up at 40 megabytes a second until it's full and Then switch into a lower speed and so For if you're using a 3d crosspoint or octane or something you might just decide to add a zero to each of those numbers or something Because at the same time you don't want to use up all the bandwidth just writing Data that maybe isn't going to be used. So having some rate limit is still useful Just as you want the device available for reading more than for writing Hi here So you talk please are you is that in a case with bake ups?  Too much allocation for the LRU in a specific case right the the each time we get the Ghost hit The p-value only moves by one byte so it takes a lot to actually move that counter many megabytes or gigabytes But in the case of a backup because you access each file only once You're not going to hit the ghost list very much So the the p-value won't actually change in the case of a full disk scan or a database scan And so it's not actually going to cause your MRU to get really big it will still cycle the MRU and make it less useful but at least some fraction of your caches MFU and Frequently used files are not going to suddenly drop in performance because of the scan Hi So how do you evict stuff from the ghost lists? So the ghost list is because it's just the keys. They don't take very much space And eventually there's just a size limit and they fall off just like a regular. It's basically a secondary LRU cash and so When it gets full you just delete the oldest entries. Hi, what's up with the set standard? Pardon what's up with your set standard work? What's that standard? Yes, so I there was a little note on one of the slides there about adding the new Compression type Z standard that's originally from the author that did LZ4, but it's much more modern and designed to Kind of like how GZip has levels one through nine The standard has like negative something all the way up to 29 Gives you a lot more control over the level of compression So I've been working on implementing that in ZFS and it all works The problem is if you turn the compressed arc feature off and you have an L2 arc It means that the copy of the data in memory is not compressed So when you go to write it to the L2 arc you first have to recompress it So that the checksum will match the copy that's on disk which was compressed And that works because we store the compression level and can do that the problem comes when you go to read it back if We upgrade the version of Z standard in the future for example over the course of the development Yon released a newer version of Z standard that all compression levels 10 and above are now like 8% better compression So now when you take data it was written before and then you recompress it with a newer version now When it ends up on the L2 arc it has a different checksum because it's actually 8% smaller And so when we read it back We see that the checksum is wrong and report that your your L2 arc SSD is failing when it's not So we're trying to decide what the best way to avoid that problem is whether it is to avoid the compression or Check for the checksum mismatch doing the writing phase and skip it in that case Or what or just pick some version of Z standard and always use that version never change But giving up the performance gains that are coming out on a regular basis doesn't seem like the best option either So it's kind of hung up on solving that problem That's why there hasn't been much progress in the last six months but Hoping to figure that one out and get that feature upstream It's so I originally developed that on previous D But someone has ported it to the as a pull request on ZFS on Linux And so if you don't have an L2 arc or at least if you leave Compressed arc on it works and you can you could play with it today Although it's until it's committed. I probably wouldn't want to use it in production right So recently I had a Windows image and I had this running on a ZFS volume and I had a L2 arc that was Somewhat larger than the vid windows petition, but every time I spun up the windows petition Image and span it back down. I kept getting hard disk hits. Would it be correct that I actually need to have an L2 arc That's at least twice the size of the image to Not see any hits anymore. I think likely what the problem you're running into there was the fill rate limit because you were only Allowing by default I think 10 or 20 megabytes a second to go out to the L2 arc But the data in your arc was falling out at a faster rate than that So if you just increase the fill rate Then you should be able to fill up the the L2 arc you have and get closer to the hundred percent cash Interesting. Thank you very much. Yeah, it's basically taking a fraction of the data That's falling out of the cash and writing it to the L2 arc And if that fraction is only 10% or something then you're it would take a lot of cycles before you had all that data on the L2 arc separately there's another feature coming in ZFS on Linux and eventually to free BSD called Allocation classes that allow you to have an SSD that you specifically define as being for metadata or for Blocks that are like 4k or something so that you could force specific data to always be on the SSD And then the rest of your data be on the spinning disc Oh, hello So I read that PostgreSQL wanted to implement RSE Yeah, 15 years ago, and then they decided against it because of the patents situation Can you say a bit about the patents situation? I think IBM has a patent on this IC algorithm. I Cannot comment on issues of patents Not qualified and really really don't want to There's a question up there Yeah Over on your over here Yeah, I was just wondering about the Compressed arc and Obviously that pulls the data compressed off of this or if you've got a ZFS volume that you've enabled compression on and a load of existing data on there that's uncompressed at the moment Does that then get loaded even though you've got compression on your volume compressed arc? That still gets loaded into the arc uncompressed because it's uncompressed on this right with compressed arc the copy in the Arc is always exactly the same as the copy on disc so that the checksum can be verified each time without having to store a Second checksum, right? So if I wanted to take advantage of the compressed arc I'd have to send you basically write the data to make sure it gets written compressed Yes, or you could use the ZFS replication to send to a seconds evolve Yeah, and have a written compressed all right. Okay. Thank you So pro tip turn compression on before you write any data Hey, three more minutes Question Suppose and NVMe is your main story. So I don't have any hard drive How would I configure ZFS in this case and I will start like NVMe has quite different performance Characteristic to hard disk so you would rather store things in a different format than in hard disk Does any of that the biggest difference with NVMe is that you can actually be Executing multiple reads at once or multiple writes So most NVMe's can run somewhere from 16 to 64 commands in parallel Whereas the SATA interface only does one at a time you can queue a bunch of them up But it's only gonna actually run one at a time So you don't need to do anything special in ZFS But there can be a big performance gain from increasing the queue sizes. So instead of by default for spinning discs you don't want to send more than a single digit number of work items to the disk because They form a queue and if you have a really important thing that comes up it goes at the end of the queue So you keep the queue short so that when you have an important Job come up it gets to the front of you quickly But if the thing can be executing 64 different commands at once having a queue of four doesn't Take advantage of that So you just need to up the queue max to allow it to queue say 128 commands to the device knowing that because it has 64 parallel execution, you're never gonna have a queue that's longer than two So just some tuning for NVMe, but you won't you don't need a fundamentally different file system or anything Thank you for an excellent talk and if you want to know more you can check out the books and I do a weekly podcast as well