And we will start our workshop with just a few people in the audience, but it doesn't matter. And it will be about fixing issues in your packages that fail to build on alternative architectures. So, let's start. The agenda is quite short. We will talk about the common issues that we saw in the past in the packages that we were building. We will let you know about the resources that are available for developers. Some general directions for architecture support because with the development it can change. Or with the projects. We did some work on CI for upstream projects. And, yeah, we have some recommendations. We can talk about improving our documentation and seeing some rather complex examples of how to work with, let's say, complex issues where something doesn't behave as expected. So for the common issues, the most common one is about the data in Dionys, which is the order of bytes or written memory. It depends on the architecture. Sometimes one of them, the most important byte is at the lowest address, and the other one is at the highest address. So this is the most common issue where projects written originally for Intel CPUs, they assume the little Indian format and without any change they can easily fail when running on big Indian platforms like big Indian power or its mainframe. The reasons are usually wrong typecasts when trying to get the smaller amount or the smaller value from a bigger one like typecasting from 4 bytes value to 1 bytes or from 64 bytes to 32 bytes. In the little Indian architecture it just works because it takes the lower bytes but it rates wrong for unexpected data when running on bigger big Indian architectures. And generally data formats for files or for storing information in general either contain the definition of the Indian is directly in the format or they expect one and they need some byte swapping or value swapping functionality in reading, writing these files. Another issue that's common is that the chart type in C language is by definition undecided and the implementation of the language can decide which will be the default one. So for Intel's default is signed and for all other architectures it's unsigned can lead to strange issues when functional returns minus 1 for Intel but it's all ones in the byte and when the chart value is unsigned it's 225 and instead of for example failing or returning from a function it does 25 cycles of some work. Another thing is about the size t type which fortunately went away when we dropped the 32-bit mainframes from building because it was defined there differently than all other architectures I know about. It was a problem with C++ code where some templates couldn't deal with the size t being unsigned long while some usually unsigned end so troubles there need some if-deaths or similar tricks in the code. Another common issue is using the atomic operations or working with atomic operations. There are various ways how to do it properly I think the best one is using the atomics from the CU and C++ languages. It's implemented in the like this or current versions of GCC so no problems no need to define any functions for that usually an assembler and usually just for U architecture so use the standards. Similar ease is always the clock counting or tick counting where some again per CPU architecture or architecture specific instructions are used to get these numbers and again it requires some code in assembler again often is missing for other architectures and if this assembler code is mandatory there is no work around like using the get time of day functions or side stuff and it just fails the mills. Other things are architecture definitions or checking what architecture is the code is being built. Are you using this in the source code? They not always list all the architectures we care about but yeah it's quite simple to add the missing ones but this should be meant as a warning whether there is some better way how to decide on which architecture it is without requiring really explicitly list of the architectures. In the source code one warning or additional warning there is no standard definition for the little indian power in the preprocessor defines it needs to check first for being power PC architecture and then decide whether the byte order is little indian or indian. This is a proper check for designing or defining the code fast for the ppc64 begin architecture and the ppc64 le which is the little indian version. Again general warning about any pieces written in assembler it goes against the portability of the code so be careful and do it just when it's really necessary. Hardware support I think is no problem overall along ARM, natural intel, power but there can be some differences on the mainframe because it's due to its heritage. So it's now over I think 50 years it's missing some more desktop related or PC related pieces or classes of hardware like USB, there is no Wi-Fi, some other pieces that are missing. And the question here is what to do about it whether to build let's say the unnecessary pieces, unnecessary libraries and make the packaging cleaner so there are less if arch in the spec files. I have less exceptions or whether you want to be more precise so you will even still building these packages. My opinion is rather to be cleaner with the packaging, reduce the number of exceptions and rather build a few more libraries. So this is still quite cheap even in the mainframe case so we can accommodate with a few more RPMs than if you deal with some BIOS if arch and potentially broken dependencies across the architectures. Yeah so I think that most about the common issues, the parallel debugging, it's a technique how to debug issues somewhat specific to an architecture, it means that you are doing the debugging process in parallel in a working situation and in a broken situation. And yeah when the results will differ then we have the wrong piece of code. Should I talk to the mic or? Before recording the first part. It will be really short and it's meant more as a work on session really doing some work so the talking part should be short and all the information is listed here. Yeah the next part is the resources. There is a general wiki page dealing with some architectures in Fedora which is pointers to all the architecture specific wiki pages that tell us some additional information. Every package in Fedora can access the machines provided by the Fedora infrastructure team.  You just try it so it works. So that's the easiest option to get access. And when dealing with the endianist problems, when something is failing for beginning architectures, usually in most cases fails for both power and security. So the easiest way is to debug on the power machine that's really easily accessible here. There is a need for a strength machine specifically just let me know and we have one guest on public internet where any maintainer also upstreams can walk in and can do their work. Yeah, they are ARC channels where to find us, you can ask questions there. If there is a need for longer term for some projects or you would like to integrate power machines into some CI or such stuff on the upstream level, there is also an option to get virtual machines from the open powerhouse that are on various places across the world. And I think either you can find them on the internet with the contacts or we are cooperating. We are the alternative architecture team located in Bernal, Czech Republic. We have cooperation with the University of Technology in Bernal that is hosting one of the powerhouse. So we can also offer this service. There are other ones. There is Unicamp and then there is OSU. Yeah, Unicamp, OSU, Oregon State University and maybe there is something else as well. Yeah, and for people wanting a really workstation based on alternative architecture, there is a good chance to order now the Talos workstation which will contain the Power 9 CPUs that should be generally available later this year. So it's not only for working on the alternative architectures but also it's focused on the privacy and security minded people with some additional features. Yeah, but it's probably the only alternative with reasonable performance to make it your primary workstation should be available later this year. Do we have any of those? Sorry? Does our Intra have any of those? The Talos 2. The Talos 2 don't think we will have any. We don't have any? I will have one. Oh, not you. It's still not released. Yeah. I thought they released it very recently. It's announced it's open for pre-orders so it's not physically available, you would have to pre-order it. Okay, I thought it's already released. Not yet and for the federal purposes we have good cooperation with IBM so they are providing us power machines directly from IBM for the various projects we have. I think we as federal don't need it. We don't need the Talos because it should be just a regular power-based machine following all the standards so it should not make a difference. So it's really warm. For the individuals who will be interested in having it. Yeah, so that's for the resources. Yeah, and now a slightly more interesting piece which is the CI. We are working now internally because it's an exercise now. Definitely there's a plan to make it public but what we saw is during the F-26 release cycle there were very big issues getting Firefox built with Firefox. The F-23 version, they made lots of changes there. The F-22 version was fine. There were some fixes from the federal Firefox maintainers so it was fine but the F-23 version really was broken for a very long time for most of the architectures including 32-bit arm, 64-bit arm, power, S-20, everything. So we started to think what we can do about improving the situation and our idea was to start doing some kind of CI or testing on the upstream level. And the problem or the issue here is that I don't know if most but definitely some of the upstream projects are not interested in supporting the alternative architectures we care about in Fedora or Reddit in general. We want to make the other thing but since they have these architectures as the lowest level or lowest tier in their supported architecture plan so we probably need to find or have our own infrastructure for checking these upstreams. So yeah, we started working on that and I think you can show some of the results here. It's now tracking a couple of projects where we know that we had some issues in the past. We started with DTC which is the device compiler just to learn about the Jenkins because we picked Jenkins as the central piece. You're just as a test project but the first real one was the 5-4-1. So the difficult part in 5-4-4 is that the Fedora packages still carry some patches that are not upstreamed. So you cannot take the upstream code from their markable repository and just try to build it. It will fail immediately or almost immediately so it needs some additional patches. That's why we split that into two steps that depend on each other. First is to prepare the Fedora sources that are patched with Fedora-specific patches. As you can see with the blue ball it's succeeding so we're able to apply the particular Fedora patches. If it would fail, it would mean that the Fedora patches would be databases or maybe dropped. So when this task would fail, it will mean that we can do something or we need to do something to get Fedora sources first. Once we have the source code for Firefox, we then do the builds. One build is with some default configuration which is failing in general but succeeding at least for Intel which is a good sign. So at least one architecture is succeeding and there are some common issues. Some of them are already reported being worked on by the Firefox upstream. So sometimes it should change by using directly on the Firefox code. So Firefox-Fedora task is a much more closer to the Firefox configuration in Fedora. It's using some libraries from the system so it's really more following the style of how Firefox is built in Fedora and I guess it's failing for all architectures. It's really taking the latest upstream code, I think even newer than the next Firefox version which will be 56. I see there is even a branch for 57 so it's really the master branch, the latest stuff that's in Firefox upstream. So yeah, it's failing. The Jenkins thing is able to put some nodes to the failed build so for it to put some reasons why the builds are failing. So it's easier to track these things when you can just check the differences between the known failure and the last failure. So if it is pulling the master branch, what spec file are you going to use? It's not using spec files, it's using the upstream instructions like configure, make, install, this kind of work, it's not working with spec files. So how about the dependencies then? It needs all dependencies installed locally on the builder or also on the slave. So it's quite a manual action to get it ready. So it's not that smart enough to figure out everything and do the build. Yeah, that really needs a prepared machine with the dependencies pre-installed and just doing the build. So any plans of automating it so that it can probably use the... In Fedora side probably can use the raw header spec file. Yes, it could be the next step and I think we would have to talk about the tools that are available to do that. Because yeah, this is really doing the builds natively without any mock or any kind of choice, virtual machines and things like that. It's a simple setup so definitely there are some possibilities to make it more smart and maybe use the spec files or really the stuff directly from Fedora. I think it would be a good example to use the rebasal tool that's available for the Fedora packages where it really automatically creates patches for the current state of this git or the packages in this git to do the latest release version. Here can be also a problem that this CI works with the master branch of upstream. That's not released yet so it's really directly talking to the git repositories or to the circular repository. So yeah, let's say it's open question how to improve that and how to make it closer to the Fedora packages but the question also is whether we want to make it closer to Fedora or closer to the upstream. So it's also a question whether to really to follow the upstream development or Fedora packaging because Fedora packages are based on. Definitely when you pull it in like not Fedora or any other pistol, definitely they will add some patches, right? Yeah, that ideally all the stuff is in upstreams in Fedora, the patches are quite rare and as we can see for example on the Open Blast mass scientific library that's being built in Fedora and this is built directly from their git on Github. So all the stuff is already in upstream so if the upstreams are really cooperative and accepting patches from downstreams or from the alternative architecture, people across the distributions it can really work or these checks can really work on the highest upstream level where the upstream is really up to date with regard to the alternative architectures. Yeah, so Open Blast is one library on the statement directly from upstreams, a live DAP4 is another one and I think with the DAP library using DCI it found one issue that was not visible in the Fedora packages because Fedora builds cannot use. Network connection so there was a small number of tests or in upstream for this library there are tests that need network connection and this test was failing. It failed for little and didn't power because there was a wrong check for block size of some things, some caches and it used the block size from the XFS fan system which is 64 kilobytes on power machines for whatever reason. And not expected for kilobytes so I spoke with the upstream guy and we fixed that and now it's all fine. Any question Mike? Well no, just to comment, the reason they made it that size has been a batch of page size. Sure, yeah. But yeah, we have seen that be a problem. Yeah, so that definitely was a reason to make it this way but the upstream code was never run in this configuration. So it failed, we fixed that with upstream and now it's real as well. Image magic is one of the recent things that got broken. So we are trying to improve that here or at least monitor to stop here. Okay. So yeah, as I mentioned the plan is to move all these things to public so now the discussions will be how to share resources with some general CI federal infrastructure, how to share that with upstream, share with other issues with anyone but definitely when it will be public it will be much easier for anyone to check for the results. The good thing about integration in Jenkins with the source code management tools is that it allows you to see even the direct changes in the upstream repositories so it's then easy to see what changed. Yeah, well that's the reason why the Jenkins tool exists. Yeah, so that's it about the CI and about the plans and yeah. So other stuff. Yeah, here. Protections should be taking all the packages that failed to build alternative architectures. Our opinion is that excluding the packages is really the last resort solution. We would prefer when we would be conducted first for our opinion about this stuff unless it's clear or clear that the package doesn't make sense to be built on the particular architecture. But sometimes we can do just exclude the failing test like we did for the image magic issues we have. So yeah, really, we would prefer not to exclude the packages if possible. Yeah, but it depends on the maintainer. It's their decision. Yeah, if you are looking for help where you are looking at, shall we improve our documentation? So that's our question to us. And yeah, the last one here is about the stuff that's changing during the time. And it reflects the state of the upstreams or can it reflect the state of the upstreams, the various projects, like the current development on the goal language compiler, where the PPC system or big NDN part is not on par with the little NDN one and because of the missing functionality, the goal language package maintainers decided to drop the big NDN power from the supported architecture. I don't know if they make it for F-26, but definitely in F-27 there are no packages built on the PPC-664 or if they carry some ongoing piece or if they follow the goal language guidelines and use the goal arches background. So that's changed recently. It can happen in the future as well. It really depends on the state of the upstreams. How they support the other of the architectures and that's all. What else is here? Sydney has a nice example of a complex issue that she is debugging right now. So she can give you an overview of what she is doing. And I think that's all from me. After that we can do some open discussions. And yeah, we can discuss also some arrange related pieces that one might be interested in. So yeah, I'll put it off here. This is basically an example showing how deep a problem can be. The answer for the fix could be a single line, but while debugging it may go deep down to multiple projects for debugging. So recently I was trying to build a federal cloud atomic image on ARM V7, which is ARM itself. And while doing that, there is a cross step where anaconda installation happened to create the image. And there was an issue which was in the storage configuration. The error you can see it says that volume group given physical extent size 4 megabyte. But it must be 1 of 1024. By looking into error message, it was not quite obvious. Can you just raise the context please? Sure. So by looking into error it's not much obvious what is the problem. And it works fine on other arches. So I thought probably it might be at a specific issue or maybe something else you don't know in the beginning. So I will tell you what approach I took. Since it was a problem in anaconda installation, so I opened the anaconda source code and I looked for the error message which could be the text stream match, which could be volume group given physical extent. Something like that. And I found that in anaconda file called kickstart.py. So this is the first step. So you can see that there is a line called volume group device that gets supported PLE sizes. So it basically calls and gets the physical supported sizes here. And it's trying to basically 1, 0, 2, 4 bits. And the anaconda needs the physical extent size of 4, expecting the default which is used as 4 megabytes. So what I did, I opened the Python interpreter and tried to reproduce it on the machine and tried to reproduce on ARM and both XLSC64 to see what basically the results is here. So what I saw that the result obtained is on ARM HFP was just 1, 0, 2, 4 bits. It basically returns a list of sizes starting from 1, 0, 2, 4 bits to 16 gigabits on other ARCHES, XLSC64 or ARCH64 or Power ARCHES. So it seems to me like it's not getting expected results because it's only 1, 0, 2, 4 bits returning on ARM. Or it might be that that's the only supported physical extent size on ARM. So I wasn't sure at that time. So what I did was later I went, I have to go detour that from where exactly this information is coming. So basically, the second step, which is it basically calls from, it's basically called from Blewitt package. So I went there, so there is also just there making call to blockdev.lvm.getSupportedP sizes. So this is the function which they call. So obviously here is not the definition of what exactly it's getting returned. So next step was it's basically coming from blockdev module. And so I need to look there. It might be the definition is there. So I took a guess that where might be this module because it's not a Python module. So there is a live blockdev package. And that package is suppressing here in C. So obviously it's not now in Python. So now you can see that the code flow is going from Python to C. So it was basically a live blockdev provides us some binding, some functions which can be used in other languages. So in Python they were using those functions. The way of using those functions basically in live blockdev is a page. There is a file which gets generated called blockdev.2.0.typelib. So typelib is a kind of another form file format which basically can be read by gobject introspection which is another project. So it's a binary format which is read by them. So you can basically get back what exactly is defined in this typelib file. And there is a binary given by gobject introspection which is called this gr generate which will generate an XML file from the typelib. So when I open the XML file you will see that this is the function name. So lvm getSupportedPais sizes which you can see there in the Python, the blevid package it's calling the lvm.getSupportedPais sizes. So that's the function which Python should be calling. And it's basically mapped in the scene function as bt.lvm.getSupportedPais sizes. And it says array zero terminated equal to one. So I will tell you later. So these are the things which I understood while debugging the code. That's how they are sending back the data from Python to C to Python. This was not just standard C Python used. It was like it uses the gi and it gets understanding from how it reads from the typelib, etc. And yeah, the return type of the function is u in 64 which is like 8 by 2 sizes. So later what I did was I invented this function in C which is bt.lvm.getSupportedPais sizes. I did the debug and print statement there to see what basically it's returning the data on ARM and on other Arches as well. So I took x8664 as the example. So in both Arches this function was basically returning the right data. And this returns exactly the same list which is 1024 bit to 16 gigabit. So basically it's the multiple of two from 1024 bit to next will be 2048 bit, etc. So to 16 gigabit. So now I know that it's not a problem. It's not something that ARM v7 doesn't support other physical extent sizes other than 1024 bit. It basically supports but there is some problem happening while returning the data from C to Python back. Which is basically happening only on ARM v7 or maybe some other 32 bit Arches as well. Because I'm only debugging on ARM v7 and there are i686 so I'm not using that for now. So I can just say that it might be 32 bit issue. Okay, so now I know that we blocked the written right data. Now I have to know how exactly it communicates. The Python and C both communicates to each other. So there is another project which is called PyG Object 3. Which basically has a function called, if I call and this function interestingly at the runtime, it gets the function name and calls the function with the matching function name. It will probably look into the tightly information and it will call the function if it matches. And it makes call to the label of the defined function. And so yeah, so most of the places it's fine. And now still I haven't been, went to where exactly it's returning data. So there was another, there was another PyG Object 3 which has, which basically returns data to, data which can be returned to Python. Which in the function looks like this. This is the function which is going to start PyObject star and just call this truepy array. So there are various other functions as well, but this is what it gets to turn in our case. So this is the standard way of returning the data to Python. It's like PyObject star and it will map back to the Python, to the Python data types. Which whatever data types and gets out of there, understandings are there. So yeah, I reached at this point. So you can see that I have to go into various projects to understand where the problem could be. I'm still struggling. I know that, I know the flow and where kind of the problem. But still what exactly is the problem is still in progress. I hope I will get answers soon. So far from my experience it seems like the problem is the type being used. Like also they have various data types used which is like depending upon arches. Like unsigned long, when I do the size of on arc64 or x86 or any other 64-bit arches, it gives me 8-byte of size. But on ARM v7 it gives me 4-byte of sizes. So there are a lot of data types being used which is basically actually you defined in Glib to library which has the definitions which uses other primitive standard data types and doing the type divs. So there are basically a lot of type divs and some are architecture specific which might change. So I am looking into various possible options. The fix could be a single line but debugging may take you from one project to a lot of other projects which it depends. This was my current experience with debugging this issue. I just wanted to share the information how you can debug various issues which might be an arc specific. There is an Indian issue and there might be the 32 or 64-bit arc specific issues too. There might be an issue which is only happening on just one architecture. So essentially we had like PPC64 available due to some changes added in the PPC64 in G0C. So I think it is already for... Is this bug fixed already? No, I said I am working on it. The recent problem I encountered and it is still taking me time because I have to understand how exactly these all flows are happening. Never debug this kind of issue before. I think that we already fixed all the simple ones. Another good and interesting thing is that you cannot use GDB or something which can give you a trace from Python to C function call. So you just only can debug Python and then if you are going to C you need to use... You don't know basically how exactly the flow works so GDB doesn't give me the results. So it is really like going and doing print statement or something like that. Whatever understanding is there to use that. So thanks a lot really. I think it is a really nice example of a complex issue that needs to be developed. It might be worth to try it on a second with Intel because it is more likely to... Yeah. A long type should behave the same so there might be some wrong definition in between... Or the interface between the C library and the Python library where they would understand correctly all the types and would really return the one value for the second with Intel. So I think it really showed the flow of all the things of projects and all the stuff that needs to be debugged. So I think most of the workshop... We were expecting to get some issues if people have... Really the idea was that some people are having some issues so do you have something to show or do you want to really help with? Unfortunately not many people came here so we cannot help them. There is one thing that I think is pretty good because I want to publicly say to Jerry James who is a federal maintainer for a long time who did some fixes for picking general issues in various strategies even when our initial discussion was okay upstream doesn't take care about, for example, mainframe. They just ignore that. He was leaning to putting an extra charge and I don't know what week later I saw that from in federal leakage. It's been that he goes to fix so he is a really good guy in fixing, say, obscure issues related to the beginning architecture. We will find some way to thank him more publicly. It really shows a nice part of doing collaboratively on the distribution. It's a really hard task to fix it, maybe he likes challenges. Or he doesn't like when packages are failing. But yeah, I think our goal in general is really to make all packages available on all architectures if it's technically possible. Anyone can pick his favorite architecture. There are reasons why people are still using mainframe. We have some time. I have to ask, is Fedora actually running on any big Indian architecture? Yeah, it's running on mainframes. It's running on the architecture. It's still big in the end and I don't think it will change. Any specific examples? Which architecture? Is it mainframe? Which mainframe? There is only one, it's IBM mainframe. Oh, okay. The system Z or Z-Systems? It's the C&DX, the shortcut and UPC-64, the beginning power, the original power. It's still still big in the end, Fedora. I think it's going well now with the C&DX, okay? We are working on this a little bit, going forward. It depends what you define by support. The little I did will be the priority. And we will try to keep the beginning running if it's possible. Oh, okay. I thought that we are totally trying to remove it. No, we are not actively trying to remove it, but there will be time where we will be.


Suzanne Wolf. It's a pleasure to have you here. We are attempting to gather the history of ICANN. I'm going to ask you two questions and then we will be up and running. The first question is the easy one. Okay. Talk about how you got involved in ICANN, what the time frames were, what your roles were, and that will give us a baseline that we can work from. From then to now? That's going to be kind of, it's going to take a few minutes. Do it. So my involvement with ICANN predates ICANN. And that always provides a unique moment when I have to introduce myself and talk about ICANN. I became involved with ICANN because I was working at the time for John Postel, University of Southern California, ISI, wonderful place. Still miss the environment there. Still have friends I made there. But when ICANN had to come about, new co, a new home for the IANA functions, there was a great deal of political and social and you know, people were talking about all kinds of things. I'm a techie. I was a system administrator and network operator in the research group, the Postel around. I worked with the networking guys out of all the groups at ISI because they were doing the cool stuff. But I was technical and I looked after name servers and did a lot of the day-to-day work that supported IANA. And at a certain point as the drama unfolded around us, I was in a conversation with him and some of the other folks that worked on IANA. And he turned to me and said, you realize that when all the yelling is over, somebody is going to have to be in the middle of things making sure that the systems actually run and people can do their work. And I thought that that sounded like an interesting task of making the operational and technical side of ICANN actually happen. So I agreed to do that. I said, okay, you know, as we're planning the offices, we're doing the mundane things that go into making sure the operation works. Time me up for that. So when ICANN started as a separate organization, they had office space that was sublet to them by USC and the building that USC is still in, you know, ISI is still in. And I was the one that did everything to make the bits flow, everything from getting the phone system installed to procuring and setting up the servers that the IANA staff needed to work on. And it was a couple of the IANA folks and me and Mike Roberts who is the CEO and in all seriousness, a couple of lawyers. And a very small number of people, very small, very shoestring resources. The board was much larger than the staff. And there was a lot, well, Mike and the board were doing all of the big picture stuff to make ICANN viable and get through those initial difficult days. There were a couple of us back at the office making sure that IANA, which is what it was all supposed to be about, was actually able to function. As you've implied, you were at ISI and as ICANN was formed and that function was moved from ISI to ICANN, you moved with it. I did, for only a couple of years it turned out, and ICANN didn't have apparel capacity at that point. In fact, I was loaned by USA. I've never been an ICANN employee. So that's an interesting detail, but let me ask, were there others who were in the same position who either moved or were loaned by ISI? Yeah, there were, wow, I would have to go look, I would have to make sure that I knew the names, but there were a couple of the IANA staff. Josh Elliott, I don't know, most clearly, I believe there was another staffer, and I am embarrassed, I've forgotten the names, but there was really a very short, it was a very small number of people, and in fact, we used to try to imagine what would happen when things were stable and we really knew what ICANN was going to be like, and, oh, not that much bigger than it was then. It was hard to see. It was hard to see it growing to be a tenth of the size it is now. Yeah, and it would be interesting to know what happened to those people just as a curiosity. And how long were you in that role and then what happened because you went off to do some other things? Well, the whole thing fell apart without me. I was there until early 2001 because I went through the Y2K, which of course was filled with great drama partly because ICANN was ICANN and partly because, you know, with all of the IANA functions and so on, but also partly because there were a great many of fears that turned out to be somewhat overblown but obviously needed attention attached to the root name servers. So there was a large ongoing project that had to do with making sure that the worldwide DNS, the root name servers, the functionality they depended on was reliable and stable through the Y2K drama. Early 2001 though, I figured I'd been doing this for a while and I went to Bay Area Networking Company and after that to ISC, which is well known as the open source company that provides a lot of network infrastructure software. So I was working on DNS again. Bind in particular. Bind many generations ago then, but I was actually at ISC for quite some years through many versions of bind, through many revolutions in how the network operates, what people expected of their software, what people expected of the structures around the software and the corporate, you know, how the industry grew and changed and so on. And I haven't been with ISC for the last couple of years, but one of the things that happened while I was at ISC was ICANN reorganized. ISC is also a root name server operator and was part of a set of events. ICANN reorganized and I believe it was 2003. You would remember better than I because I was somewhat disconnected at that point. But there was a reorganization around ICANN of how the board was structured and how governance of the corporation was structured and a great many changes around how the community was integrated with the organization. And one of the things that happened was the root server system advisory committee that I participated in was asked to send a liaison to the ICANN board of directors. And I was asked to take that on. Partly because when I was at ICANN, one of the things I did was get RSAC started. I ran the first meeting of the root server operators that became RSAC because it was considered important for ICANN to make relationships with the root server operators. It wasn't clear to anybody exactly how that should work, but it mattered to ICANN, it mattered to the community, it mattered to the US government that there be something and that was what RSAC was. Interesting. So you got RSAC started while you were at ICANN. And you went to ISC, which was a root operator, and became re-involved in some sense with ICANN. Got dragged back in. Now, if memory serves, I may be wrong, but weren't you also one of the charter members of RSAC? I was not a charter member of RSAC. I was drawn in after a particular incident involving the varicine.com wildcard. And I had been involved in ISC's response to analyzing what happened there. And quite many people were extremely angry about that incident. And one of the things that came out of it was some response by ISC as a software vendor and an operator of DNS infrastructure. And I believe that turned into some conversations because it was something RSAC was looking closely at. That turned into an invitation to participate in those conversations. I see. And from there I was invited as a member. I see. So just very quickly, of course, was a chair of RSAC from its beginning in early 2002. The event that you're referring to is extremely famous and is a pivotal point in our lives and everybody else's. September 15, 2003, just to be specific. I wouldn't even have been able to tell you in 2003. A moment that will live in my memory for quite some time. We had a pretty interesting day at ISC that day, too. It was an interesting. It was a Sunday afternoon and it was... We don't have time to even begin to... We'll go to some other day. But as you mentioned, there was a reorganization. There's another term of art that was being used. But anyway, the bylaws were changed. The advisory committees, RSAC for one, RSAC for another and so forth, positions were created on the board for liaisons from those organizations. You became the liaison from RSAC. Right. There was a lack of interest within RSAC, which I decided I should go fill for, not so much for ambition, but because it would help position RSAC more visibly inside the board. And so I double-hatted myself as both chair and liaison. So we joined the board in parallel positions at the same time. Somewhere along the way in 2003, summertime or fall or whatever that was. You were there ahead of me, but not by very long. Well, so we've had a long march together. It's been pretty interesting. So that's the sort of rough history. That's question one. It's the outline. Question two, and there's many pieces to this. Okay. What's most interesting is to get at the underlying story, the why and the how of things happen, not just the what, when, and where to pick out. And then if you have trouble picking out, I'll help you out. Particular sequences or stories that put the pieces together and relate events over a period of time that help explain and inform where we are today so that if you imagine somebody sitting in your seat today on the board or working for an ICANN, trying to figure out why is it that these things are the way they are, either relationships or structures or whatever, that there's a story to tell that says, well, it came about in the following way. I'll ask you for a prompt and then I'll decide if I'll use it or I'll use the first thing that came to mind. Well, you triggered one when you said something about the root servers and then needed to be organized, which reminded me that there was a singular event that took place a few years earlier, which you were probably part of, that led to the creation of ICANN, which was a small reorganization of the addresses for some of the root servers for which Ira Magaziner said he woke up the president. I've heard that side of the story. I can't vouch for it. No, Knork and I. There are great many people involved in these policy and political arguments over this, but what was really going on was there was an operational function of deciding updates to the root zone of the DNS, what names were valid for domain names in the DNS that had been treated up until then as a strictly operational matter, which Postel and consultation with other stakeholders, we didn't call them that then, but we had made these decisions in a very informal way because the other people involved in operational responsibility for those matters trusted IANA. He worked closely with the IETF, which is the standards body, was still his, worked closely with the root server operators and with other constituents, but it was all very informal. And as one aspect of how the political story was unfolding, people were saying, this doesn't work. This isn't going to scale. There was an occasion where, as an experiment, there was an occasion where some of the root server operators were asked to change the source they used for the root zone data. This was very alarming to the political people, the policy people because it uncovered a power structure they didn't know about and they didn't know how to deal with, which was personal relationships based on, gee, things will break if we don't play well together, so we'll play well together. But this wasn't computing. And in fairness, this had not exactly been explained to them very well. So it's no mystery to me that there was a great deal of alarm, but what came out of that was after the heart attacks were taken care of, the immediate reaction, what came out of it was an assertion by the United States government that really nobody should change the root zone of the DNS without their express permission. That was a big change. Even though at that time the root zone did not change very often. It was not dynamic. We had a few TLDs. It seemed to be working that way. The country codes were being delegated. There were a few, what's now called GTLDs. It wasn't a big operational drama to manage the root zone, but it was kind of important. And this hadn't been clearly housed in... She's a lot of visibility yet. Well, but it also hadn't been carefully housed in any kind of institutional arrangement. So there was this immediate felt pressure to create a structure which was don't change that unless the United States government says so. I'm going to say that's not going to work in the long term either. That's not going to scale to the enormous expansion of the Internet and the interest of people having the Internet. So we got to figure out something else. And that's largely how the policy side of ICANN came about. Now, something that is so obvious to you and me but needs to be said is that a large number of the parties involved were DCCTLD operators who then became very dependent upon the IANA function. But the other obvious comment is each one of them was started by Postel awarding them the letter, the code. So there was a direct personal relationship that was very positive in the sense that these were his progeny in a sense. They were. And those personal relationships both with the root server operators and with the TLD delegate operators. And it was actually extremely unfortunate. It was a formative event also when he passed because he didn't get the chance even though that was the plan, he didn't get the chance to do the personal work of turning his personal relationships into institutional ones. And there was a great deal of trouble and discomfort and distrust that followed from the fact that he wasn't able to be part of that process because there was a certain amount of ICANN, what's that? And you guys, who are you and why should I listen to you? Well, so let me ask a kind of provocative question here. So it's extremely unfortunate, no question that John passed away sort of on the eve of the formal start of ICANN. So that's an event heard around the world and everybody knows that that's not good. But I could imagine some alternate sequences. It could have happened that others could have stepped in and made the rounds to assure everybody that John-like people would be in charge and that the relationships would continue and attend to the relationships and the trust. That is a little bit provocative because from what I saw, there was a certain amount of effort to do that. Some of it was actually effective. We did end up, for example, with an agreement between ICANN and the IETF and a different set of agreements between ICANN and the regional internet registries that said, yeah, we'll continue to work with you. Which frankly, as far as I'm concerned, is a success of the work that was done of the kind you're describing because there was a while where it was not at all clear. Who were the prime movers on the ICANN side of things for that? I actually had limited visibility on that. There were a few of the senior engineering folks that had been involved with the Pustel principle through the IETF and IANA and RFC editor and so on. Fred Baker, John Clemson, I'm trying to remember who were the other contacts there. But there were several of those. I never focused on it and I wasn't around at that particular time. But the event was, I believe, instrumental in a great deal of this. But some of it was tried with some success, frankly. The whole thing didn't fly apart. And to me, that's a success. There was some of it that was tried, but the situation had arisen that there was a deliberate effort to select people to be closely involved with ICANN who did not come from the community, who were not familiar with John and his personal relationships, his institutional relationships, how the internet worked. So some people tried very earnestly to do what you described and it didn't always go very well. And there was also, for political and policy reasons, I'm not the person to speak to authoritatively, a great deal of discomfort and distrust had been generated by the process of trying to decide among all these constituents how ICANN would be set up and who would run it. The white paper process, the green paper process. And so there was, by the time we had that work to do, there were not a lot of resources available to help them. So I was carefully not participating and don't have all that, but that's certainly an area that has to be delved into very deeply as setting the context pro-WanCon for how ICANN got kicked off and what baggage came with it. Well, and for me, I look at this thing. I was distant from those processes. And my role then and frankly now has to do with saying, okay, it's important to support the policy processes and these institutional processes and so on, but what really matters is to keep things connected enough that the internet works. And it certainly would have been easy to become very confused about the relationship between some of the policy and institutional activities at that time and anything to do with keeping the internet functioning. And what about the CCTLD operators? To what extent? I mean, I'm flashing back to what little I know about the contentious process of the green paper and the white paper and all the different groups involved, but I don't have the impression that there were a lot of CCTLD operators directly involved in that process. I may be wrong. No, from what I saw of it, there weren't very many. That were involved on the policy side. For most of them, at that time, the CCTLD operators were the people that Postel had picked in many, many countries at a time when the internet was a new thing, was not to the attention of governments, certainly was not part of their sovereign heritage, that there was a TLD as part of it. This was not critical infrastructure yet. So what you had was many of the TLD operators were either technologists who happened to have access to network resources because that was what was important in Postel's criteria, or were just emerging past that. We're beginning to say this is critical infrastructure, this is commercially important, let's run it in a more institutional way. But it was very early in that process for most of the CCTLDs. I'll just note that we have two that I can think of quickly of the original CCTLD operators, closely associated with ICANN, Lito Ibera, El Salvador, and Adiel Acropogon, an African, I'm going to get it wrong, which country, Toko, I think, or... I think so. So there's deep history to still part of what we do today. It's just kind of amazing. Many of the people that were involved in building these instadies and these infrastructures then have gone on to other things. Many have become uncomfortable with how things have evolved, but many have stayed involved, and stayed committed to making the internet work better. Now that we're talking about the next billion, when then we were talking about the next million, or the next 10 million, I think we're actually very fortunate that there are so many people with the deep history. A lot of them are skeptical of ICANN, frankly. Not all of them are very comfortable with something like this meeting. But we actually have a lot of them still to draw on, and I think that's tremendous. Any more provocative questions? So that was a provocative question. I don't know whether we've plumbed the full depth of it, but we've got some interesting things out. What came to mind first, was it that, or was something else when you asked me? Well, that's an easy one from the beginning sort of days, but in many ways with the IANA transition process of the last couple of years, it's been feeling like we've come full circle. And there were a great many, I would say, community tensions around some of the history, some of the early history, frankly, that has resurfaced in the context of having to work out a shared proposal for the IANA stewardship transition. And for the most part, this is exactly how it was always supposed to be. Maybe it's taken longer than people initially thought it would. But the idea would be that the community would figure out its challenges and figure out a way to work together and would be able to move beyond the U.S. government contract and the U.S. government stewardship oversight. And in order to do that, I think people had to face some of the things that have been lingering, if you will, since that early history. And that's been fascinating to watch. So let's not leave that hanging. Say specifically, what are some of those things that were lingering? This is challenging to articulate because I've been living it so much for the last couple of years. I understand. And I know you have too. And once I get started, maybe you can help me articulate a little bit.  But regardless of where it came from, there's always been a certain tension about the idea that all of these constituencies work together and that the stakeholders, if you will, are part of an interdependent system and the notion that someone's in charge. And as one formulation of the principle, the Internet is founded technologically and to a great extent institutionally on the assumption that local decisions are made locally, that what the Internet is for is inter-operation and coordination. It's not an end in itself and it's not a structure for control by itself. And that creates some real challenges around who's in charge of what and who makes decisions and how do decisions get made, particularly in situations where decisions must be made. And I think at one level, the entire history of ICANN, particularly these last couple of years, have been about figuring out how much do we need somebody to make decisions and how much do we need to make sure we don't have anybody in charge. So there's been certainly with the community proposal a great deal of the discussion about it that led to the proposal was based on working out under what circumstances, how are decisions made? Not even who makes decisions so much as how are they made. Who are the stakeholders who have to be in the room? What are the structures whereby these contentions have to be worked out? And it's been really interesting to watch. But I have to say I'm happier than I expected to be with the outcome of that entire process, as painful as it was, because I think some of those lingering discomforts about really at one level what is ICANN in charge of and what are some of the other stakeholders actually in charge of. I think we actually worked out some of them in a way that I think will be good to go forward with. Today, in addition to all the things we've talked about, you're also heavily involved with the ITF and IAB, which means you're engaged in specific technical issues that relate directly back to issues that affect ICANN, but from a technical development point of view. Say a word about that and where that's all leading and what the unsolved problems are that have required the action that's now underway. Well, what I do, I'm currently a member of the Internet Architecture Board, where part of what we do is some administrative management of the ITF's relationships with other groups. But what we're really supposed to be doing is taking the big picture view of how the Internet architecture is evolving not just in the ITF, but in other standards bodies and other ways of doing technology in the industry and so on and so forth. I also co-chair the DNS Operations Working Group of the ITF, which is where people, where DNS operators and people involved with operating DNS infrastructure bring changes they think need to be made, and most of what DNS operations does is actually best practices and documenting some of the things people are doing at the technology. Some of them are not what was originally envisioned and how it would work, and that by itself is a challenge. Sure. Evolving how stuff works. What are the current documents or designs that are active in the DNS Operations Working Group? Oh, my. It sounds like you're looking for something in particular. Can I ask you for... Well, I'm just trying to drive down the path that I described before, which is based on whatever you say, to what extent, how did that relate to what came before and why are those still open problems and how does it relate to what's happening at ICANN, and I don't have the answers in my head, so I'm... There's probably a couple... Okay, so there's a couple of things that we can talk about. For one thing, the way DNS is designed, there's this dependency on root name servers.  It's an optimization. From a protocol perspective, it's not a very... With due respect to our good friend, Dr. Marco Petrus, that's not necessarily the best way to bootstrap any technical system, is to have preconfigured contact points with how you... It seemed like a good idea at the time. It did, and it was, but having the system work really, really well isn't necessarily a reason not to evolve past that. So we have some ongoing efforts to come up with best practices and ways of replicating the root data in a way that depends less on a specific set of servers. Interesting. And this is all purely technical practice, and it's not about the data in the root zone. It's about replicating, not changing, but on a wider basis than necessarily from the root name servers. There are many efforts to refine how DNSSEC works in practice, because the ability to authenticate DNS data because of all the things you can carry in DNS, the ability to authenticate a DNS answer is really important. It allows you to do all kinds of other things in a more trustworthy way, but it's also very difficult to do operationally. So we're still refining. Many years after the base specification was done and people have been trying to deploy. And as you know, I spent a good fraction of my life so I'm very empathetic. And part of what has to be done there is operators have to do... And this is a very internet thing, this is right, because there's all the interlocking pieces. ICANN has to help people come up with good ways of doing it, has to sort of nudge people sometimes in some of the relationships ICANN has, but also the standards and technology people have to do what they can for operators to do the right thing day to day. So it all interlocks. I believe... I never get to see... There's always a DNSAC day at ICANN meetings and I never get to see it.  So I'm very empathetic. I never get to do techie things at ICANN meetings. I'm too busy doing policy things. The sadness is your time on the board as the RCAC liaison, I gather, is coming to an end. Most likely, yes. It's been a role I've been in for a long time and I've enjoyed it. I feel like I've done some good work there, but also one of the things that's happened is that RCAC has matured. The root server operators are stepping into more of a role in the community and I'm actually pretty pleased with them. They've gotten to the point where they're willing to come up with a process and a plan for replacing them. And I hadn't quite realized your pivotal role or your seminal role in getting RCAC started, so I can well imagine that you are indeed very, very pleased to see the material, but you said most likely as opposed to certainly. So is there a glimmer of hope that the... I don't expect to be on the board past the HM. The most likely is because part of the work of RCAC at this meeting is to take the next steps towards finalizing the process and determining how to run it. The good news is that then you can go to technical meetings and do technical meetings. I look forward to that. But I wanted to swing back to the IETF working group and just flag the point that one thing you focused on was the priming process for... or the root scope process, as you said, for root servers. And I had not focused on that, but I understand that I know a lot about the details of how it currently works and what those issues are. And so there's a long, long history that's been very stable for a long time, and now you say it's being reexamined as to how to do that. What are the main drivers that are causing people to feel the need to push on that? There's a couple of pieces. There's always the drive to make the network work better. And the tension for DNS and for the root is that it's a global resource. It has to be globally available. And anything global makes network people want to optimize it down to something local. So there's the Anycast technique, the Roots of Operators, and many, many DNS operators and operators of other network services use to make sure that many, many servers can provide the same service under the same configuration. But there's also the notion that for any network operator, if you want to make a local copy of the root that is available to your users, there should be a wallet. And there are failure modes of that. It's challenging. So there was a need for best practices on how to do that effectively. And then the DNS sec was the other thing you mentioned. And there, I don't think we'd have any trouble documenting what the needs are because we've been tracking that for years. Well, it's certainly a significant challenge to get DNS sec into a signed root zone and to get all of the policy pieces and the operational pieces and the... You just said a key word. And I see we've got a couple of minutes, but not long. The root was signed in 2010. As you and I both know, it was not a trivial, easy decision or process or anything. Say something about the events leading up to the signing of the root zone from where you were sitting. There were two pieces. One was fast and one was slow. The slow part was an effort literally over several years to persuade the stakeholders that the technology was ready to deploy, that we weren't going to make DNS actually work less well for users, that there was benefit to it, that would justify the expense because there was significant expense involved in getting the security apparatus and processes in place. But the other thing that occurred and there were lengthy discussions that were leading in that direction, and it was frankly a long process of just making a significant change to how the root zone was generated and distributed. People were worried about that, frankly, justifiably. Now, where people like to sleep through the night and not be patient to in the morning because something broke. But also, there was a discovery as people were sort of taking pretty much unbounded time to decide whether this was a good idea. There was a specific issue that arose where there was basically a bug in the way DNS was deployed pretty much worldwide where it was possible, it became trivially possible, easily possible for a teenager in the basement that knew basic, a little bit of programming to create a situation where they could lie about DNS data, that they could redirect legitimate traffic by lying about DNS data, and this became the push that convinced people that we needed to use DNS data. And to fill in the blanks there, DNSSEC was invented to provide strong assurance that the data that you received is the data that was intended to be received using cryptographic signatures. From the time that that work started a decade earlier or more in the early 90s, other mitigating measures were put into place that reduced the chance and made it harder to spoof the data. And the event that you're talking about was Kaminsky discovering that those mitigating mitigations weren't actually effective and could be bypassed trivially. But it was much easier. The thing that Dan discovered was that it was much easier because he's a very smart mathematical guy that it was much easier to intercept the query and send a lie back. And what the NSSEC did was basically assured that you would at least be able to tell if you were being lied to. And all of a sudden that caused a big shift in the understanding of, yes, this is an important problem, we better get this fixed. What it changed was the perception of the cost of getting it wrong. And the cost of doing nothing. Which so many of these things come down to that though. What's really the cost of doing nothing? I see that we have come to a moment when we need to pause. But this is, I think we've got a lot of good stuff on the table so far. More to come. Thank you Suzanne. Thank you.


Good afternoon, ladies and gentlemen and welcome to this press conference from the 47th annual meeting of the World Economic Forum Here in snowy Davos welcome to everybody here in the room and also welcome to All of you on the live stream and of course a special welcome to everybody here on the panel For those of you here in the room if you made your way Through the snow It is probably hard to believe that water is a scarce resource. It is however and That's what we're Dedicating this press conference to today. The question is access to safe water And we have a wonderful a panel of experts and people who have dedicated their life to work on that issue of access to water Let me quickly introduce Our panelists to you to my immediate left Is Dominic Warry who's the head of public-private partnership at the World Economic Forum and it's also leading our work on climate change and water Further down the line. We are joined by by Gary White who's the co-founder and chief executive of Officer of water.org Right in the center of our panel. We're joined by Matt Damon who's also a co-founder of water.org On his immediate left. We're joined by Ricardo Tadeo who's the son president Africa of unhouser push in Beth And last but definitely not least We're joined by Usharao Monari who's the chief executive officer of the global water development partners and a long-time champion of water issues Also at the forum and on its platform. So thank you very much for being here today Dominic give us a sense of why water is an issue that the forum feels Deeply about and and why we're putting this at the center of attention here in Davos. Thank you York Welcome everyone We ran some analysis with our colleague Usharao Monari and some of you in the audience several years ago That when you look at an economy, it's a thirsty thing energy food Industrialization urbanization. These are thirsty businesses And in fact, it's so thirsty that by 2030 if we don't change the way that we use the water the fresh water That we have on this planet. There'll be a 40 percent gap for zero 40 percent gap between the water that is safely available to grow those economies and what we have So that's a choke point plain and simple We are profligret with water. We don't use it wisely Over 70% of the water we use is for agriculture. There are huge efficiency gains to be made Energy is a very thirsty business whether it's fossil fuel energy or others So if you add all this up together and we have a fast-growing global economy Particularly across emerging economies something needs to change That is why in our risk report for the past seven years and you'll see it in the global risk report for 2017 water Remains one of the top global risks at the World Economic Forum and its network of risk analysts identifies Now the trouble about water or the intriguing thing about water is it's all joined up So in fact that risk network is exacerbated by climate change Climate change might create more scarcity of water because it's warmer or it might change weather patterns It will dump water all at once So climate change and water become a double risk and Of course if you're a farmer or you're poor and you are susceptible to risk and there's no water And the climate is warming you might move Water climate and migration involuntary migration at the three top global risks of the World Economic Forum's risk report for 2017 All of that against a baseline of a world where there are over seven billion smartphones But 2.2 billion people don't have access to an adequate toilet Something is not right and something needs to change to improve the economic productivity And inclusive growth of our economy that is why the World Economic Forum is very interested in water as an issue Thank You Dominic Gary You've obviously realized the issue early on and founded water dot org What is at the heart of the foundation and and how do you see the risk?  Well, I am optimistic for a few different reasons. I think when we look at you know why water dot org It's it's a slice of this global water context and what we try to do is come at it from the perspective of people living in poverty So there are these macro issues and they need to be addressed from the top down And I think that there's things happening to make that make that work out But for us it's like what are those impediments to the poor getting access to to water and sanitation? And we want to have them have a level playing field so that when resources are developed when infrastructure is expanded from the top Down they have a level playing field and can come into Getting these water and sanitation solutions And I'm optimistic because if you some of the work that we've done on the global Agenda Council for water has focused on these coping costs The coping cost of people buying bottled water even very poor people Sometimes having to do that paying water vendors the health costs the direct health costs in terms of going to the doctor buying medicines But the lost time you add up all these coping costs, and it's it's it's about 670 billion dollars according to our preliminary results on this we need to do a little bit more refinement, but Those are all Funds that are in the system right now that could be redirected and what we've done is help people from the bottom up through Water credit get access to microfinance so that they can get away from some of these coping costs not pay loan sharks for funds And then use those microloans to get improved water and sanitation So I think that's really what we see we see that the problem Itself contains the problem contains its own solution in terms of the resources being there to make this happen And I think just in wrapping up one of the dominant themes. It's emerging here certainly is income inequality and I can't think of any greater manifestation of Income inequality than people not having the money to gain access to water and sanitation that juxtaposition of what we so too much take for granted at the turn of a tap that 663 million of us or two and a point two billion From the sanitation perspective don't have this basic commodity and that's how income inequality really hits home for the majority of the world's poor Thank You Gary Matt. Let's continue with you. You I imagine you have a pretty tight schedule normally So I hear from teaching pink this morning that you just came back from defending the Chinese wall. So So what's why did you throw your weight behind water? Was it beyond Gary's charm? So what was it that convinced you to to work in that water field and dedicate so much time to it? Well, I was just looking at issues of extreme poverty and I was just so impressed by how water and sanitation underpinned absolutely everything and And nobody was talking about it. It was something in the West that we have a really hard time relating to And so as I dug deeper and deeper I eventually came to the I'm trying to do this as fast as I can because I know we're on But I eventually came to the realization that I needed to partner with the the best expert I could find in the sector and and when they wouldn't take my calls. I ended up with Gary But but I should what I should do is just another minute on water credit Gary kind of breezed over it It's it's an innovation that that that came from Gary Through his time in these through his a whole adult life kind of interacting with these communities He basically realized that the extremely poor of the world were paying for water You know, they weren't they were oftentimes paying ten times more than the middle class of their country but the the concept of water credit was essentially applying the ideas of micro finance to the water sector and so the kind of thought leap that that required for these micro finance institutions was that Normally they give loans that are you know income-generating loans. You give me two hundred dollars You know and you loan me two hundred dollars I buy a sewing machine and start a sewing business and I pay back your two hundred dollars What Gary realized was that people were leaving jobs women primarily were leaving jobs to go stand in a line And collect water at a communal collection point and it was this terribly inefficient use of their time And so if you actually fronted them the money to connect to the existing infrastructure and say an urban slum in India Where where water is being piped right under their feet. They're just not connected to it What you could essentially do is buy their time back And so rather than wait at this communal collection point They now have water coming directly into their house and they can work more hours at their job and pay the Loan off with relative ease and so it was a really it seems like a simple thing But it was a pretty profound insight and so we we started piloting this and it's worked really better than we ever could have hoped I mean these loans pay back it over ninety nine percent ninety three percent of our borrowers are women and And it's something that that really can scale and is sustainable also because as these loans come back They go out again And so the philanthropic cost of capital per person just goes down every single time that loan gets recycled So right now we're down on our most mature loan programs to under five dollars per person To get them to get somebody clean water. So So it's it's been a really effective Tool of ours and we've been looking for these special Corporate partnerships to kind of turbocharge our work and that's one of the reasons we're here today to make an announcement Thank you. Thank you very much Matt. It's good to see you various Dominic pointed out that the risk are connected You're also saying there's also positive spillover effects into other areas. So that's that's great to hear Ricardo Matt mentioned the the corporate partnership And why don't you tell us a little bit about what what you're doing with water org and why you've chosen to do so It's very important to mention first that we believe that every brand needs a cause Right something really to stands for to stay behind it and For seller 12, which is one of the most iconic brands beer brands in the world where water is its most basic ingredient It was just natural that water became that cause and I think that our situation was more or less like Matt's situation I mean we start studied the problem we got into this situation when we saw that one out of ten people in the world don't have access to water and It actually was a surprise. I mean, I think awareness is something very important to drive awareness to that situation and then I think the idea was stellar to our codes Inspire people to help solve in that problem because the other important thing to mention is that this is a solvable problem It's not something that I mean will it's one of these world problems that will be there forever We can make a difference the companies can make a difference the consumers can make a difference and what stellar twice doing is trying to be a bridge for that happening and We are announcing here today the extension. We've been working together since 2015 So I mean we have helped stellar toa and its consumers has have helped More than 800,000 people have lifetime access to water and now we are expanding For more four years. We have this goal to get to 3.5 million people until 2020 so it's a very important day for us. We're here we thank a lot Mets for for the prep being being here today with us and We are very proud. I mean we are launching a new line of Chalices that can be purchased in the first year. We saw about 90,000 Chalices worldwide second year 35% this year. We're aiming for more than 300,000 Chalices every year a different set of Chalices We have Chalices that are made by artists from Brazil Cambodia Uganda very nice products So I mean we believe we're doing something big. We started In a relevant way, but still small. We are almost Three times now the size we were at the beginning and we can only think Where can we get if five years from now and we forget more companies more brands? Getting behind the same cause which is something that we are inviting everyone to take part of it Thank You Ricardo and Usha. I know you're modest, but Dominic He he snitched on you and he told me that you actually have been behind many partnerships And you have a lot of experience with these concepts of collaboration. Why don't you share with us? Kind of what's the impact and what makes that work? Thank you Absolutely, I think partnerships are what is needed in the water sector if you take a step back in water, right? And you look over the last century. It's a shocking statistic But water use increased three times faster almost than population growth That has resulted in the kind of statistic that Dominic quoted. We're looking forward by 2030 You're gonna have a 40% deficit between demand and supply That is a serious situation because water as Gary and Matt have have explained is important for life It's important for dignity but it's also important for economic growth and corporate operations and that's a fact that has escaped people that you know It water is necessary for the power sector. It's necessary for you Ricardo It's necessary for everybody, but somehow people have taken it for granted assume that it is infinite today We're unfortunately facing a situation where accessible water is not infinite And so we have to look at water as an economic good in addition to being a fundamental life-giving good I think partnerships are important in the water sector because they've never had them in the past the kind of stuff You guys are doing bringing the corporate sector in alongside possibly governments and Finance years like me is what is going to make that difference in the water sector and create the tipping point to avoid a 40% deficit future Thank You Usha We have some time for for questions So please we have a microphone on two sides of the room if I get a Show of hands to get a sense of who has questions We have a gentleman here in the front and the lady in the back So if you could state your name and organization for the sake of our online audience, please Yeah, hi, I'm Harry strail from the Swiss television You teach people. I'm sorry my questions to Matt. I'm surprised You teach people about your project also in a humor way and humoristic way Is it important for you not you know not to be showing the moral point in finger? Well, that's thank you for that question We've we've struggled with messaging in general. We've tried a lot of different things. We tried, you know Humor as you say, you know a few years ago. I went on a toilet strike for the entire year And you know, so we've really tried anything we can to get the message across one of the one of the kind of First hurdles we have to clear in the West is that it's it's just a very hard for people to relate to this If you grew up in Europe if you grew up in America, you you know You've never been thirsty in your entire life, and you've you've never been more than five meters away from a clean drink of water The water in our toilets is cleaner than the water that 663 million people have access to so it's really all it's a hard thing for us to Wrap our wrap our brains around whereas if you're fundraising for You know an AIDS awareness or cancer prevention or things like this, you know People can relate to that because they they have a family member or a friend who's been affected directly and Or they've been affected directly and so So we've really struggled I think with our messaging. It's one of the reasons we're really Grateful to have this partnership with Stella. They've they've they've done some really innovative things like for instance not beyond the chalice sales They've got this idea of in in the UK or in America if you walk into a bar and order a pint of Stella you are going to provide somebody in the developing world with clean water for a month Right now, so that's a very direct one-to-one correlation, and that's not something that we ever would have thought of so to be able to have their That apparatus and that relationship they have with their consumers and is really important to us And we have really high hopes for for where where that might go and then on the same side You know we've made this film that's gone up on the internet today and this digital Short Following this one woman going on a water collection and kind of just explaining her one specific story And how her life has changed by gaining access to clean water and it's they're just such good storytellers They're there and they've and they've done this Kind of film that's less than three minutes. That's a completely perfect kind of digestible story that really helps kind of Show the way this this issue directly affects affects the people that it affects so So our messaging I think is improving with the help of people and partnerships like this But we still have more to do and we're it's it's constantly a give-and-take as we come to forums like this and interact with the press and Talk to people You know, we're open to anybody's good ideas Thank you very much. Let's take the second question from the lady there in the back Thank you, Florencia Agiles for W radio Colombia Matt Damon. Thank you How can we explain all the interest in the in the topics like economy like sustainability like I don't know migration but the weather is not only in all this agenda of leadership How can you convince the leadership to is the to something to do something and also if you have a already The compromises of the application to your foundation. Thank you Thank you very much. Let's get the next question. Yes, please Well, I mean, I think I slightly disagree with your premise I mean, I think you know as as Dominic pointed out This has very much been on the agenda here for the last seven years And so I think there is a rising understanding of that and look the part partly that that's what we do We try to come to things like this and convene You know groups like this so that we can talk about it precisely so that we can raise awareness But I mean our work has been going better than we ever could have hoped in that in that Gary's hypothesis about these this microfinance play and how how How successful it could be was was absolutely correct And and and now we're just looking for ways to turbo charge that work and and and I think this partnership is a great Example of that. Let's hear from the other panelists. What what else is missing that we that we can add to do more to raise more awareness I Was just going to say that you know, I agree with Matt You know, there is a lot of awareness on the part of leaders I think Dominic mentioned the water resources group that the forum has backed And has sponsored over the last what? Four or five years that has really brought the issue up to leaders today We had a session earlier today where the prime ministers of various countries gathered to talk about how important water was as a Development agenda issue for them So I think it's getting up there alongside some of the other more popular issues. Yeah Excuse me. Could you use the microphone of your friends on the live stream? Well, otherwise, sorry When can we see the the results of the results when you see the results? Well, well the aim for this particular partnership is three and a half million people by 2020 But we've you know, hopefully you're gonna just see the results for in terms of water.org. It's it's ongoing I mean to date we've reached 5.3 million people with these solutions and so You know when we were in India a few years ago We asked our microfinance partners to identify the biggest bottlenecks from their perspective and what they all reported back to us Uniformly was affordable access to capital and so they felt like there was a massive demand And we could be doing much more. So for us it became waste. We started troubleshooting. How do we deploy more loans? How could how do we get? Get them the access that they need and we came up with this idea of water equity Which was to try to tap social impact investors and and and start to build a fund which we did We did the first fund which was 11 million dollars and we've deployed that and now we're doing a second fund at 50 And we're trying to bring in big banks We're trying to be going hide net worth individuals and and people and target a 3% return But but the but the real impact of the investment will be obviously the on the social side Can I add something I also think that leaders are attracted for things that work, right? People wants to see things that really work And I think that the fact that we started at 15 with a yearly contract reaching 800,000 people in two years and now we are moving for a long-term commitment. I think that shows that we are doing something that's Doing good for the brand It's doing good for lots of people and hopefully that example can be a model for many other brands many other leaders to follow And I think you know we share the vision with Stella that we can be the generation that solves this water crisis and We need to come up with with new ways of thinking about it This is really I think what the partnership has been built on is innovation of water org So being the organization that innovated water equity as a new as a new approach because we need to Bring water to people today and we're doing that through this Stella partnership But we also have to look over the horizon because we're we're you know Realistic enough to know that even water credit isn't the solution for you know the two and a half billion people Say without sanitation in the 663 million without water. We need new ways of thinking about this So we're always looking over the horizon with water org to figure out that next innovation And that is what's going to tie this back together with us being the generation that can end the water crisis Always looking at new partnerships new ways of doing this Social impact investing as Matt mentioned is coming on so strong now And we feel like we're positioned to drive a lot of investment capital to this because it's such a Basic solution to such a basic problem and it's bankable We're providing a financial return as well as a huge social return And then to your point also about how do we bring in more people into this? And how does we get more attention? We want to democratize that and create a platform this year That will allow anybody to come in and make a loan into water equity so that people can get Safe water in addition to the large-scale investors that have come in to this point for the first one You're you're talking just a second I think Dominic wants to add something just just to just to build on that I mean you are welcome in here at the World Economic Forum in in in Davos and you know that this is a platform for public Private cooperation, and it's not a better topic Water that illustrates the power of that collaboration as you just heard across the panel You know we have partnerships will get us to over 500 different partners across multiple countries You have government leaders who recognize the economic kind of weakness that poor water quality presents And you have wonderful innovative partnerships with water.org and companies and brands like stellar as well as many other Companies and brands who are working on those on those partnerships. So here at the forum This is exactly the kind of platform where you can bring together the public and the private and the civil society Organizations to drive that systemic change at scale. That's what we aim to do in these few days Thank you So the gentleman in the back and then the two ladies here in the front So my name is my name is Tobias Bossers from Swiss television You're talking about awareness which increases on the leader for all these problems, but now we have to change perhaps in in the States so Matt Damon what do you personally think about the presidency of Donald Trump and could it perhaps have some changes to environmental aspects and things like that because the awareness of that one is not so high with him I can imagine there's a big interest to hear that it's a bit of an off-topic question So Matt if I don't know no, it's fine We don't know really what his relationship to foreign aid is there was a slightly alarming article in the New York Times a couple days ago that Laid out kind of the the type of questions that came from his transition team to the State Department in terms of aid to Africa That was a little worrisome But I don't know. I mean, I don't know him. I've never met him personally. I don't I Don't know what his relationship is with the developing world. I don't know if he's been to the developing world No, I'm serious. I mean his interest is in high-end hotels and golf courses and that's his business interest takes him to First world places I think and so I don't know how much interest or Time he's really spent in the developing world engaging with these issues And so I think you have to give him the benefit of the doubt and and you know You know at a certain point our work isn't dependent on the political kind of you know wins and and and we just have to keep Doing the work that we're doing and you know president Clinton said to us a few years ago, you know as it was clear that our method was very successful He said just keep running your numbers up keep running your numbers up because he was just pointing out that we just gained Credibility with the more people that we're reaching and so that's what we're here today to talk about really with this this great partnership with Stella and in terms of what President-elect Trump's relationship is to foreign aid. I think we're gonna have to wait and see So a time for two more questions Gunilla, please Hi, my name is Gunilla phone. I am a Swedish journalist once got uploaded my question is to miss the dammel I'd like you to explain to me what access to things like clean water and toilets will make for the economic and social Development in poor countries and can you give me some concrete examples of? Changes concrete changes that your work has led to in these countries so the very first person that I ever went on a water collection with this is ten years ago, I was in rural Zambia and This girl was about 14 years old and we walked together I was waiting for her when she came home from school and We walked about a mile to this well that was about a mile from her home and we collected water together and I talked to her about Well everything we had quite a bit of time to talk and And when I got to the topic of what she was gonna do with her life You know and would she continue to live in her village She she smiled and said no no no. I'm not gonna live here. I'm gonna move to the big city I'm going to Lusaka and I'm gonna become a nurse And I just really connected with her because I remembered being 14 and saying to Ben Affleck We're gonna move to the big city of New York and become actors And I remember what that feeling was and and and what that age was like in that kind of world of possibility That's out laid out in front of you and and as I drove away I was struck by the fact that had someone not had the foresight to sink this borewell a mile from her home This girl would not be in school. She would be spending her entire day scavenging for water for her family and She wouldn't be dreaming about moving to the big city and someday contributing to the economic engine of Zambia In fact, she'd just be in this death spiral of poverty where she was scavenging for water and trying to survive from one day to the next And so leaving aside the really horrible You know, you know a horrifying Death, you know every 90 seconds of child dies because they lack access to clean water or sanitation leaving that giant issue aside There's this whole other issue of what kind of life somebody can expect if they don't have access to water versus what kind of life They can expect if they do have access to clean water and sanitation. So So I think when we think of you know 3.5 million people that we want to reach together Those are 3.5 million of those very stories because that's a story you see repeated over and over again I remember we were in Haiti together probably five or six years ago and a little girl I asked her, you know, how this water system that we've been put in her village was gonna change her life She had been spending four hours a day collecting water And I said is this gonna give you more time to study and she looked at me She was 13 years old and she looked at me and said, you know, I don't need more time to study I'm the smartest girl in my class and the way she said it I knew she was telling me the truth, you know And so I said well, well, then what are you gonna do hot shot with all this extra time? And she looked me right in the eyes and she said I'm gonna play And it just it didn't really Got to me because I had a 13 year old at the time and just this idea that she hadn't been afforded the The childhood right to play because she'd been so busy scavenging for water and now this was gonna give her that opportunity So there's you know, so hopefully, you know, we're by 20 20 will create Through Stella three and a half more million of those stories and that's that's a big deal for us Thank you. Okay, let's take the last please a short question because we're already running over time Sure, my name is Farah Bouchouac. I work for France 24 Just a really quick question and observation really when I saw Stella Ottawa with what is a dot org? I thought it's greenwashing. I was wondering why a company like yours who uses so much water There's so much waste to make beer. So Why did you get involved in how far actually are you planning to be involved? I'm sorry if you think it's please don't suggest that company should stop making beer. Just selfishly, please don't ever I think it's a great question and thanks for Actually water is a great part in terms of our our footprint. We take very good care of water We have some very important global commitments We've been decreasing the usage of water by 50% in the last five years And we are in this process of exactly defining the new goals for the following years Especially I mean I'm in Africa We used to 3.4 liters of water for each layer of beer We want to reduce by next year to 3.2 which is already a very good progress I think this part of Some other points are being discussed here about all the companies trying to reduce their water footprint And I would say that it's not only in our facilities, but also in our clients Educating the people who work with with beverages and beer to in restaurants and bars to make sure that they treat Water very well. I mean there's lots of learnings and and education panels and efforts that we do in that effort So this is something big for us. I mean as you said Every time we build a new brewery. We have to make sure that we have access to to excellent water So that's exactly why we're here and that's exactly why Stellar twice is here. I think that's We have the power to dream right so We are pretty confident that we are going to reach to this 3.5 million of these great stories that we've we've heard Here today, I think they're quite inspiring and I'm really very happy for being part of this for representing a company that is part of this and I mean We talked about 600 million we are aiming for 3.5 million in three to four years Perhaps we could double this if we could get more 50 Companies a hundred companies doing the same things for definitely we know that there are in many different categories Brands that could be part of that effort to join forces with us. We could solve this problem Right and and that's what you're here for. We're not here just to To be a drop in an ocean, but really a constant force and drive for for changing this for good Thank you Ricardo and thank you to all the panelists here today. Thank you for joining us Go and write stories about this because it is important and there is still awareness to be raised There's still not more to be done. Thank you for being here and thank you for watching


This is Brett with Summit Automotive in Fond du Lac, Wisconsin You're new and use Jeep headquarters and today I am on assignment at the 2020 Chicago Auto Show And I'm checking out all the awesome Wranglers and gladiators that they have here. We're right next to the test track But this one really caught my eye. It is a 2020 Jeep Wrangler unlimited Rubicon recon edition Really glad they brought the recon package back for the JL Really a good-looking package all the way around. It's very distinctive. I knew right away what it was when I saw it You get the paint in polished aluminum rims. This one's in bikini pearl coat one of the best-looking colors on the Wranglers You can see this was named the 2024 Wheeler SUV of the year and Here's a little bit about the recon package feel free to pause that I'm not going to stick on it too long But I'm just going to walk around this thing and show you what I see With the recon package including the American flag sticker here you get the recon badge The granite crystal Jeep lettering with the red outlines You get the Black hood decal black and gray 410 locking differentials with the 44 Dana's You get the sport hood The black Rubicon lettering on the hood the gloss black grill The red tow hooks the heavy-duty metal bumper This one does have the LED lighting group. I Love the accent of the bikini with the black grill. It's really a good look especially with those rims and Vulcan wild-peak tires on there These are mud terrains and these are LT 265 285 70 R 17s You get the heavy-duty rock rails on the side there This one has the color match fenders The backup parking sensors blind spot monitoring you have the receiver hitch the four pin and seven pin wiring the backup camera as With all the vehicles here at the auto show. They always have them all decked out. So this one's got the alpine sound on it, too You get storage back here with the door hinge and roof bolts and windshield bolts. You get the JL Badge on the back here. They made these rear gates a lot lighter The LED tail lamps right here as well. You get the metal fuel door on There inside the recon package is a little bit different than the normal Rubicon package You still get the black leather seats with the red stitching and you still get the Rubicon stitched backrest You also get these stitched Steering wheel with these seven inch LCD display This one has the heated seats the heated leather steering wheel adaptive cruise controls Bluetooth information center controls the 8.4 for C radio AM FM serious XM radio capabilities. This one has the nav on it So you're gonna get serious XM guardian for a year serious travel and the weather stuff for five years I believe Got all your climate controls here can do all your climate controls here Has the HD backup camera you get your front and rear Locking diffs your four auxiliary switches and your sway bar disconnect all with that Rubicon package, but with the recon you get the red seat belts, which is really cool This one also has the SOS and assist system and the home link buttons up there, but definitely one of the best-looking Wranglers we've had or that I've seen at the auto show Here has the top of graphical map Floor mats. I'll show you the back seats here. They made the backs bigger so that you get a Lot more room back here, and this is after one day of being at the auto show It's getting late here, but allows me to do some videos to get a hundred fifteen little hundred fifty watt plug-in two USBs Two USB C's back here as well You notice all those bars are color match painted Shred pattern on there. I'm gonna do one more final walk around of this Jeep since there's nobody here Directly in my view here, but very good-looking Wrangler Once again the 2020 Jeep Wrangler unlimited Rubicon Recon package Hope you enjoyed the video and to see more Videos and HD you can go to youtube.com Slash summit auto remember to like subscribe and share on this video and all the videos that you see there Really glad you were able to check out the video today And in fact in a second you'll see a link to subscribe to our YouTube channel on your left a link to more 2020 Chicago auto show videos like this one on your right And if you haven't been to our website check us out on the bottom there And thanks for checking out this video of this 2020 Jeep Wrangler unlimited Rubicon Recon package. Thanks again