So welcome to this session. Let us understand the following problem today. If sin of sin inverse 1 by 5 plus cos inverse x is equal to 1, then find the value of x. Let us write the solution. We have sin of sin inverse 1 by 5 plus cos inverse x is equal to 1. Let us name this equation as 1. And we know sin pi by 2 is equal to 1. Therefore, putting it in 1, we get sin of sin inverse 1 by 5 plus cos inverse x is equal to sin of pi by 2. Or sin inverse of 1 by 5 plus cos inverse of x is equal to pi by 2. Now add and subtract cos inverse of 1 by 5 to above equation. We get sin inverse 1 by 5 plus cos inverse 1 by 5 plus minus cos inverse 1 by 5 plus cos inverse x is equal to pi by 2. Now sin inverse 1 by 5 plus cos inverse 1 by 5 is equal to pi by 2. So, using this identity, we have pi by 2 minus cos inverse 1 by 5 plus cos inverse x is equal to pi by 2. Or cos inverse of x minus cos inverse of 1 by 5 is equal to 0. Or cos inverse of x is equal to cos inverse of 1 by 5. Hence, x is equal to 1 by 5 is the required answer. I hope you understood the problem. Bye and have a nice day.


Ah, thank you for your patience. It's some serious gear to get my computer connected to this conference. I'm here today to introduce the multi-toner. Zatula started last year. And I'm going to tell you the whole story of the multi-toner. I'm going to give you a short live demonstration of the current state. And then I will talk about the next steps and invite everyone to become a contributor. So first things first, my name is Lasse. I work as a freelance web application developer for different clients. And I studied visual communication at the Bowers University in Weimar, which is quite close to Leipzig, so I'm mentioning it. And I think often that I should do more design work. But I'm definitely not going back to in design because it doesn't run in my system. And I don't want it there. So my partner, Anne, is a trained photographer, and geographer, and writer. And it felt quite natural for us to start making books together. So we founded our label, Silver and Bly, which is German for silver and lead. Silver stands for silver salts using analog photography. And the lead stands for metal casting metal type. So Anne's part is the photography, the silver, and my part is the lead. We decided to make really good and valuable books using only open source software. And that includes that we provide tools by ourselves if they are missing. Our first book will be about angel statues and cemeteries. And there will be only black and white photography in it. When researching printing techniques, we found out that using only one ink for black and white reproduction is not enough for our purposes. One ink in this case for black and white photography can only display around 60 different colors. Gamut is really small, I could say. A lot of detail is lost. The image looks really dull and misses depth. So the solution used by printers is to use more than one ink to display the black and white channel. So what you do is to widen the gamut of the output device. Gamut is the set of different colors that the device can display, so how many colors the device can display. What you could do is using black for the shadows, darker green for the mid-tones, and the lighter gray to display the highlights. When you use three inks, it's called tritone. With two inks, it's called a duotone. And with four inks, it's called a quadtone. And in printing, there's no name like pentatone or stuff like that, but it's possible entirely to print with seven colors or so. I think PostScript has the limitation on eight colors, or so. PostScript does more. I'm not sure about that. So multi-tones stands for any of these options. And we are most probably going to use tritone for our book. So what we thought about is this workflow for our images. They're coming from camera scanner. We process them and create black and white images and remove some visible flaws. And then we create the multi-tone images and include these in our final document using context for layout. That's all. But turns out there was no open source tool to create these multi-tone images. So we had to create it ourselves. That's why I'm here, actually. The best known tool to create multi-tones is the duotone mode by Adobe Photoshop. And that one lets us export the multi-tone images as EPS files. EPS stands for Encapsulated PostScript. That is essentially PostScript with some comments in it. And PostScript is a programming language by Adobe. And it's documented very well. A PostScript file can be opened in any text editor. And there is a very good open source interpreter for PostScript. It's called GoScript. I think probably everybody heard about that. So everything, what we needed, is already available. I started with learning some PostScript enough to understand what Photoshop was exporting. Created to start a minimalistic multi-tone template myself. And then I had to enhance the Python GoScript C bindings for the GoScript CRP. Because an important part for the direct EPS preview that I'm using was not implemented then. The multi-tone is written in Python using GTK plus 3 as a toolkit. All color previews are rendered using GoScript directly. There are two bigger widgets that I implemented myself on top of GTK. It's the Curves Editor and the Preview window. Now I'm going to show you the tool itself. I prepared a multi-tone profile of calling it. So I don't have it to do this live, because that's boring stuff, right? Everything. This one could be smaller. And two images. You can actually display many preview images as much as your computer takes. There's no limit for it. A new file would look like this. Just empty. Here's an add button, where you can add new inks. Click the ink, and then you can set up the ink. Like the name, CurveType, and CMYK values for the preview. It's not going to be spot color colors at the end, but the color swatches of the swatch books are often giving CMYK values. So I decided to do CMYK first. In the upper left corner is the control panel, where we can arrange inks. The order is going to be important when it's being printed, because that's the order what the printer will use to print the inks. To change the properties of an ink, I told that already. On the right side, in this area, color previews for all inks from the output for white is on the left side. So when the input is white, this is on the left side, two black input on the right side. And this in the lower right area is the curve editor, where you can change the transformation functions that are used. So you can see how the disfunction behave according to the output very well. The final export of EPS files is done from the preview window, like this one. Just press Export and Save. And there is a command line tool that you could use in an automated process. So if you have 100 images and change them, sometimes it might be better to have some built process or so. And you can give these EPS files directly to your print shop or embed them in your PDF file. The print shop should know what you are about to do. So what's missing? Color management is really important for good food feedback. And what I'm doing just now is not enough for you don't know how it will look at the end. Actually, so GoScript has an option to load ICC profiles for multi-tone images. And I count on it. Unfortunately, it's broken at the moment. So I don't know how good it will work at the end. So I'm waiting for GoScript to fix it back. With color management, there will be new work again. The interface will need some additions. And some data structures will have to change, too. And after that, I'll need to organize a serious color profile for the first print job. I never did this, so I don't know how well print shops will help me with that. So if anyone can help me with that, please drop me a mail. It will be much appreciated. And then it's time to go printing and see if everything works out. We'll make a poster for the first test. And when everything is fine, I call it a version 1.0. After that, here are some possible features for version 2.0. This library of good color profiles could help people massively to get started. Photoshop has something similar. And it helps. Better previews without color profiles would be nice, too. The best fit Curves feature would help to get running as well. And including component-based separations as input would open the door for high-fidelity color, like hexachrome or more experimental ideas. So there's some things we could add to the program. If you like this project and need some inspiration to contribute, here it is. I won't add support for Windows or Mac myself anytime soon because I don't need this. But I would support you if you plan to do so. Packaging would be a good start to attract more users as well as writing documentation. And if anyone would start a Libre printing project, like to go to actual print shops and that everybody knows how to work with those tools we don't have at the moment, that would be really good, too. That's all. Thank you. So we've talked a few questions. So a few questions. Yes. So it seems to me that one obvious feature that's missing is the ability to save out a set of inks, the order of the inks, and the curves that you've produced. No, you can't do that. You can do that. If you can do that, the next thing you need is a standardized image that goes 0.1%, 0.2%, 0.3% little swatches like that, which you can then measure. And then you can bring that in. So effectively, you're not producing an appearance on screen and then using color management to help you get the same. What you're really doing is mixing up mixtures that you hope will give the end result and having a not very good visualization. Whereas if you did it like that, essentially, your visualization would be, OK, for all of the points on this curve, I have a measured swatch result there, so I can produce an alternate image derived from that, which can be color managed in the normal way. I thought about the profile would not include the curves, but would measure the whole device like under the inks in the right order. And so you could do color management for the whole device. But that would be more interesting, I think. Not just the curve set up would change between images, I think. Some more questions? Not right now. I'm sure people will come up to you and you will be here the coming days. I mean, I think all the time. OK, good. Thank you very much. All right.


Yeah, I think let's get started Alex. Okay, let's see this So in terms of some general housekeeping from the From the previous call the end user survey was was sent to Cheryl to be to be forwarded on to the end user forum We have an end user forum meeting Booked in for the night of July Which is after the keep going in China We will We had We had the CNCF sick charter forwarded on to the to the talk for discussion and My understanding is that they that the talk we're going to to review with a look on 15 on this prior to coupon Is that your impression to Clinton? Sorry, I lost my unmute button. Yes, I believe that it's all kind of happening And they yes that that is the plan. I'm not aware of any obstacles Excellent, okay, so So in terms of in terms of next steps We were discussing options about putting together some use cases Using cloud data storage We obviously We obviously wanted to get feedback from the end user community, which will be getting through the survey But in the meantime I just was to pull this group to see if there were any specific use cases That you fell sort of out of that you have the resources or or or some some basic information to To be able to move forward as well. So we could have a list of those for consideration Do you know when we expect to get some results from the story I put the deadline for the 16th so that we can get the we can get the summary of the results before keep going Oh, okay, it's good So I'll I'll kick off. I'll kick off This was it so the idea for the use case is myself. So I had an idea that you know even the Priorities of the end users assigned I think it would be useful to cover a Use case document for some common options for for volumes. So sort of covering the ephemeral volumes the the local volume support that's been recently introduced in Kubernetes and persistent storage from When using a storage provider to cover, you know, some some different use cases or different Applications that kind of fit to those three types of volumes and because this is quite possibly a Fairly common question we get asked at storage of us, but I'm open to sort of hear Comments from from anybody else on the team was to whether they think this is a good idea or whether they would rather we did something else I was just wondering. I'm sorry carry on Which is saying that The block Rob lock sport is available now. So maybe want to know some use case for that okay, yeah the the question I had was is information not perhaps already obviously publicly available I would imagine with the release of A local volume for example, there's presumably a blog post that explains what the use cases are for them or is that not the case? Yeah, there was a blog post with the with the feature release. I was I was thinking I Was thinking more something along the lines of you know, this is this is you know This is these are some examples of what you could use a federal storage for these are some examples for The local or the Roblox store and these are some examples of when you do when you want, you know an external persistent system or service To kind of help people with just general guidance on these things Again, I mean, I haven't actually been following the blog post associated with the releases of those features But I would have imagined that was already covered if it's not already covered and then yeah, probably falls upon us to clarify that Okay It would seem like a shortcoming in Okay, so I think there are some blog posts have more Use case cover like like for example the local voting I think there is a pretty good block covering use case actually from I think Uber's using that or something But for the Roblox for the blog post doesn't really cover Clear use case. So I think that one maybe we can See if we can you know add more to that At the risk of getting sort of overly philosophical. I wonder if we shouldn't I mean if projects are releasing features without a clear explanation of What the feature is intended for I did say that I'm just saying that I Was more thinking do we want to look at it, you know, if user, you know, actual users using that for something, right? Yeah, rather than just say, okay, this can be used for some okay No, that makes sense and actually I had a similar thought stroke comment when Alex was Soliciting agenda items yesterday. I think this word use cases kind of become Overloaded in the sense that and maybe we need to come up with a new with different words, but in my mind, there's like Specific user has specific, you know Project where they did something and other people might be interested in understanding like what exactly they did and why they did it And whether it succeeded etc Now, I don't know if we call that thing a use case or something else My sense is that we probably need to call that something else and then there is, you know In a more generic sense like what are the categories of use case for this feature? What kinds of things what are the properties that you're looking for, you know a more generic kind of description of something and That also I think currently gets referred to as a use case and maybe that's Right, so I think I think that's a good distinction. I don't know what to call The latter one where we're talking about something for from an an actual say and user. This is how we did something I was I was thinking about this specifically because it's something with It's something I've certainly come across and some of the the people I needed meetups have come across Where different classes of volumes, for example Are used incorrectly for different use cases. So for example A classic example of this is the official at CD operator Actually uses a thermal storage So if all the nodes have a power supply failure at the same time You will actually just lose all of your data because that's the way the operators designed and I think it would kind of it would benefit a lot of people to avoid these sort of stomach blocks and to sort of adopt some of these Client into technologies quicker If we just said These are the types of volumes that you can have just because you're you have a PV from from from a CSI driver or a native driver Doesn't mean it's going to be persistent doesn't mean it is going to fail over and all these sorts of things These are some of the the categories and therefore, you know, use them accordingly kind of thing and I was I was thinking about it in terms of You know short circuit in some of the pain points that that users are finding when they're using the official repos Yeah, okay that that that makes sense and I think that that kind of stuff falls very squarely in in our set of responsibilities in the sense that we You know across multiple projects and across multiple Yeah Things we need to sort of provide a comprehensive understanding to the end users However when it comes to a specific feature release for example local ephemeral volumes for Kubernetes or whatever one would hope that those are described by the Kubernetes project and and in cases where they're not where we Don't think they are sufficiently described. We should you know help them Fix that and and not do it again in the future. Otherwise, we're going to end up Documenting all of the CNCF projects, you know, which I don't think is our job and I don't think we want to be doing No, no, I can be keeping I'm not suggest and just to clarify. I'm not suggesting that we do that Provides kind of more about summary Compares and kind of docs so that people can choose to write a bunch of rather than sort of sort of Yeah, it's not the future releases Actually, just as a slight diversion. I don't know if I understood you correctly, but if I did then you were saying that that etcd using local ephemeral storage is a bad idea and I Think that's not true So I just want to make sure we haven't stood each other and and yeah, whether there was some confusion. Yeah, so so so This this came up for example The other day when I was discussing this at the meetup The the actual helm charts today installs It's kind of runs up its cd on ephemeral storage, which basically means when the pod dies that Instance of scd goes away now You know if you're running as in as many users do it say just three notes or something like that It's quite easy to get into a situation where you lose quorum or you lose all of the three pods at the same time And that basically means that there's no data to recover. There's no snapshots. There's no backups. There's it's just gone And so, you know, there are certainly arguments for whether you you know You want to take snapshots if you're using ephemeral or whether you want to use something else like for example the local volumes Because at least then you can restart the pod and recover the data. Oh I understand so your your distinction was between ephemeral versus local rather than Local versus remote Correct. Yes. Oh, I understand. Sorry. I wasn't actually aware that there was a distinction. Okay, that makes sense I was just the reason I asked was was There is a perception that storing at cd data on remote disks is a good idea and and That it's a bad idea to use local volumes and I'm not sure that that does Sufficient just as to the debate about the two options I completely agree. Yeah, okay. No, that's good. And now I understand your point Cool, all right So so in that case I can I can potentially maybe sort of put a Put a document outline together to cover together this and then maybe we can work Collaborately to split up different sections of the document. Hopefully this can be a nice short document that we can turn around That we can turn around quickly. It's not going to be something as meaty as the white paper Yeah, sounds like a good idea. So just to be clear is your is your thinking to do it like a kind of a Q&A, you know Should I should I use these kind of blockstores or these kind of blockstores? You know bunch of questions and answers or are you thinking of it as something? Something like that with a paragraph on each type and a comparison table or something and that's it so that it's kind of like a two-page Reference article if you wish Okay for a very specific kind of pair of alternatives Yeah Okay, that makes sense. Yeah, because the the white paper tended to sort of try and sketch all the possible alternatives and and pros and cons of each which tends to be a much more Difficult thing to give a clear answer to so it tends to be begging in a lot of areas by So you okay, that makes sense All right, so so we can get cut off the the Are there are there any? sort of other areas that the team feels strongly about or would like to to Think about or to try drafting Alex on the same lines that you mentioned about the local PV and decimeral storage does it also Make sense to write something on hyperconverged storage versus remote storage When it could be a good option to use one way or the other Yes, that is That is a good idea actually It is something we touch on in the white paper and and we have asked the End users about that in in the in the survey something we touch on there. So Yeah, that might that could be another good option. That's good idea And we can probably pull in the responses from the end users survey and feed into this Thank you kind of an article or the Same same format that you are using for a Yep, absolutely. I happen to to read that section of the paper recently and It's fairly brief, but I guess what I'm wondering is how much more there is to say My recollection of what the paper says is essentially The pros are that you have, you know, a uniform commodity hardware with a converged approach And it's sort of more flexible, but on the other hand your capacity provisioning and performance impact between your compute loads and your Storage loads gets mixed up So which I think was a very good summary of the My understanding of the pros and cons. So I guess what I'm wondering is what more do we think we need to add to that? It's not obvious to me that there's anything that's not answered by those questions, but maybe you have other ideas As more looking from the end user use cases, what are the scenarios where they are going for one option for the other? Is it more like an edge use case where they don't have Access to the remote storage or is it like an NBME? Kind of a devices which are available on the modes No, just trying to So I Kind of understand what printing is saying. I think we covered the basics in the in the white paper quite well What would be useful is to try and capture Some of the some of the motivations from From the end users and for some, you know specific difference Use case examples, I guess so so just by by example by word of example, sort of I I have come across different users have chosen different apologies for reasons completely separate to You know some of the technical aspects that we govern the white paper. So for example And Commercial reasons are one thing You know where that where perhaps they're looking for say optimized storage notes versus optimized compute notes But other reasons which are fairly more common are actually Operational and and sort of day two operations type thing. So for example They might have a higher rate of change on their other compute notes, especially if they're running with managed services in the cooking in cloud environments Then they may want on their storage notes, for example, I Don't know whether that's actually merits an entire paper or an entire documents. I Don't know open to date I mean, I guess you can we can we can put us on our we can put on our list And use the information from the survey to confirm one way or the other Yeah, that makes a lot of sense to me anything else Well, the other one is the the roblox that I mentioned as I think it would be good to see I know it's still relatively new. I'd like to see Who is using it and You know, if they are they think this is something they need That maybe helps them improve performance versus the the file system mode But this I think I need some user input Yeah Because we know in general what is why that is needed, but I'd like to see Someone is really using it and the same This really helps us Yeah So I don't think we have that specific question in the question. Now. Maybe this could be like a follow-up It could be do you know what actually Every time I sort of read about the roblox support one of the obvious things that sort of occurs to me is that this that's actually a neat feature for Storage systems to actually use right It's usually for some of the database applications like SAP actually they want to have the roblox or that's the We do that's one of the use cases, but I still won't you know Let to know who is actually using it since it's still relatively new right Remember whether we had that in the survey you do you think it's not I don't think it's I don't think that this is that We just say the general, you know blocks are you using blocks through a system or not because this is very specific This is like a one mode in Kubernetes Right so the for the for the PVC, right? So so we don't have something that's specific in our survey yet or our service kind of a general Well, it's certainly something we can bring up When we when we speak to the users Yeah, that's a good idea. So that is after they have done the survey and the after the keep calm. Yeah Good Yeah, one one last item Just sort of brain dumping is that as a sig as we transition into a sig one of our responsibilities is to Well, the two things that we have not historically done and that we will be responsible for in the future our project health So all the CNCF storage projects Just checking in with them making sure that they're happy and Healthy and and whether there's anything that needs to be brought to the attention of the TOC and then the other area is Identifying what we believe to be gaps in the storage Portfolio of of the CNCF So I don't think either of them are like super burning urgent like need to be done in the next couple of weeks But but we do need to make sure those get on our agenda and we have a plan going forward for how we can address them That was good. Yes degrees So, yeah, maybe you can just put that on the sort of agenda for the future Etc. Yeah, definitely Okay One other one other project or initiative That Like so it might be helpful, but I'm I am looking for guidance here because I don't understand that this could be potentially controversial so I'd be interested in putting together some Guidance for how we Actually measure some of the different attributes that we defined in the light paper So for example, you know something like performance might be a good place to start with something that's Quantifiable, but as we all know benchmarking is notoriously complex and subjective But I was kind of wondering whether it would be Useful to put together, you know something like a sample set of either Benchmarks or tooling or or even just maybe a standardized sort of automation framework for some of this stuff Such that end users can easily Use that framework as a as a starting point to stay for when they're comparing Different cloud providers or or or different Storage systems Within their environment and you know the idea being that we start a performance, but perhaps Move it to other areas where we have For example things like availability or something like that. So for example I know the opening the skies have have a Framework which which perhaps could fit the bill in terms of things sort of chaos monkey type testing with these type of storage systems But I was wondering if we could kind of maybe put an effort to sort of Standardize or bring together a number of people from the community to kind of put a I Think I think the general principle is very good I think having a place where users can go and Find a set of tools that we believe to be useful for this kind of stuff and maybe even some Performance results that we've done on you know either CNCF projects or general other projects other I mean in the extreme possibly even products although that gets Even more controversial My reservation is whether we have the bandwidth and the longevity to You know, I don't want us to build tools that we don't maintain over time So if we're gonna build tools we need to actually sign up for maintaining them properly over time And I don't think as a sig we have necessarily the infrastructure to do that properly And the same goes for performance, you know, if we publish performance figures for version X or product Y or project Z We we probably implicitly are sort of signing up to do that, you know periodically And then all the gaps and I'm a little nervous that we might be biting off more than we can chew there so so I I Completely agree and just to be clear. I'm not proposing that that's me the sort of published results because I agree with you they'll they'll get out of date Very quickly and it's apart from the fact that you know, we don't really want to get into Sort of a comparison of products ourselves. What I'm talking more about is this kind of providing Say for example, you know, there might be two or three tools like FIO and syringe and a couple of other things for example that measure performance for particular use cases But if we if we tie it together with Some simple automation framework with some sort of default known good values That make it easy for somebody to just say, okay, here's my storage glass. I'm gonna run up this this benchmark And I can do the same thing for some different storage glasses or some different cloud providers And it gives the users effectively capability to kind of try out this stuff themselves Then we kind of give An unbiased standardized way of people to compare numbers between different systems themselves And if they want to publish stuff, well, then that's kind of up to them to publish stuff But but we're not actually publishing it as a sake Yeah, that makes sense. I would propose that as a first step, we we just Survey the landscape of available tools and that can be used these kind of measurements and perhaps publish a white paper just saying, you know, these are the ones we Looked at these are the pros and cons of the various tools These ones measure these things and these other ones measure these other things and here's you know A couple of sample runs maybe of some of them just so that you know If somebody's wanting to do performance measurements on storage systems, they they can read this paper and get pointed in the right direction That that seems like a useful first step Again, the the idea of building automation implies that we maintain that stuff and keep it working and these things, you know, they Presumably that automation is going to rely on underlying tools which change over time and You know may or may not run on particular operating system platforms, etc And that's where I don't want us to like implicitly sign up to maintain a bunch of software that we don't actually have the people and with to do properly Let's in that context some of the users actually Gave links to some benchmarking tools to use maybe in the upcoming Meetup I can probably present some of those things and of them or how they use it Yeah, that would be great. I so I like the idea of starting small we can start small with With a survey of some of the tools and provide some So this guidance on which tools to use And then I guess depending on the uptake we get from the community we can We can make a call as to whether we want to take that further. I I completely sort of agree with you Quinton We should not and must not bite off more than we can chew But I kind of feel that with potentially dozens of Snorage vendors and companies who are sort of sponsoring this and could be interested in the SIG We might actually get We might actually get a bit of a community around this because I suspect that this is a common problem for end users and The sort of storage community alike. Oops. I think we last year Alex. Oh Can you hear me? Yes now we can. Oh, I'm sorry. No, I was I was just saying I think we may find that that's so there's a bit of a drive from both end users and The the vendors and the community around the SIG That that we may find is there's quite a bit of uptake in this So I like the idea of just starting small with the put, you know, a dock with a Landscape of the available tools and then kind of taking further based on uptake Yeah, I totally agree and I think the general principle of making real hard data easily accessible to Users is is very I strongly support that because I think as you point out, you know We wave our hands and we say that, you know, these kind of stores are faster than these kind of stores or more consistent or whatever But I don't think people appreciate quite how many orders of magnitude difference. They are between some of these things And when when we sort of make statements that when you use the wrong stack of storage Technologies or not necessarily the wrong one, but if you use a particular Combination of layers in your storage system This is the result in terms of performance or consistency or whatever the property might be and and it is like, you know X number of orders of magnitude different than than a different approach I don't think people always appreciate quite how big the differences are Yeah, cool, I'll stop by playing on that So So from from my parts that was That was all the things I had on the the agenda Does anybody have any other business? Not from my set. This is orders to Vegas. I was listening at the Colt and Probably didn't introduce myself. I'm Product manager at the EMC Oh, hello, welcome Hello, cool. Okay, so if if we've covered everything I think we get 20 minutes back on this Anybody has anything else they want to cover? No, sounds good. I assume someone's gonna write up some meeting notes. I think they're kind of empty at the moment Yes, get a note taker for each meeting in future Alright, thanks


Ready to go? Good morning, everybody. Thank you guys for coming out. This is the OpenStack A2Z talk or panel. We're here from Comcast. We're extremely excited to be here. And we thought we'd take this session to share a little bit about how we do OpenStack at Comcast. Specifically, we plan, build, deploy, manage that platform. And we have some of our awesome engineers here to kind of share with you on that. So we're going to kind of go around, do a panel format here at the beginning, go through some quick questions that we prepared. And then we'll also have time for Q&A at the end. So I think we'll just go ahead and kick it right off. So the first question is, can you provide an overview of the environment? Maybe Sheila, you can take that for us. So the Comcast Cloud team was formed in July of 2012. And we released our first proof of concept in November. I'm sorry, in July. And then we went into production in November. We were running Essex at the time. And since then, we are running all over the nation in several data centers. We are running Havana now. And we are moving to Ice House. One more thing I should add. We do have multiple customer-facing apps, internal and external, that are running on the cloud today. Parts of X1. We have Xfinity Share. We have our internal conferencing system. We have tools that we use, such as EtherPad, IRC, ZNC Bouncer. Everything's on the cloud. And much more that I just haven't listed. Awesome. Thank you, Sheila. So Scott, could you tell us how we have expanded our footprint and upgraded from Essex to Havana? Yes. Can you repeat that question one more time? Sure. How did you expand your footprint and upgrade from Essex to Havana? So we've had a lot of challenges with upgrading and expanding because customer demand has kind of outpaced us in a lot of ways. We still have some Essex environments deployed. And we've been moving people to Havana. Most of our environments are Havana. And the challenge that we've had with upgrading Essex environment is that you can't just do a live upgrade. You have to kind of nuke and pave it. So we're in the process of expanding our footprint to get those users off and absorb those into the Havana environment. We're also rolling out Ice House. And we'll have to do something similar. But with Ice House, we're experimenting with and Havana. We're experimenting with looking at live upgrade options so it won't be nearly as disruptive. Great. Thank you. Warren or Anton, can you guys maybe share a little bit about the other critical piece of cloud, which is storage? And what back ends we have implemented? Sure. So back when we started off our journey in the Essex days, we sort of had this idea that it was just going to be ephemeral and object storage. And there'd be no block storage. I guess looking back is probably a little bit naive. So we had more and more customers ask about block storage. People, for better or worse, also found Nova Volumes. So we started looking into SAF once Cinder materialized. We launched our first SAF cluster somewhere around the Hong Kong summit. And then since then, I think we've got over a dozen SAF clusters. And it's really been sort of exploding in size. Awesome. Maybe Anton, you can share some of the challenges we have of running SAF at that scale? Sure. So we definitely, I think, like most people that run SAF, at least the people that we've talked to at this conference, we started out with poor configuration choices with hardware, also with some of the config choices for SAF configuration on the software side. So just to give an example, we started out with no SSDs for journals. Then we went to really crappy SSDs for journals. Then we went to better SSDs for journals. And now we're actually switching to PCI SSDs. And then we're also looking and starting to switch to NVMe SSDs for journals. So that's been, so SAF is incredibly resilient. But we've had massive failures. And SAF just takes it and keeps going. We had one cluster that it was like six nodes. And we lost one node. Then we lost another node. Then we lost another. So we went from six nodes to three nodes. And we actually kept going. And obviously, as it rebuilds, it's a little bit slower. But it just kept going. So that's really cool with SAF. Awesome. Sheila, can you maybe describe some of the maintenance of the environment, the upgrades, the migrations, the nitty gritty that maybe we don't often get into here at the summit? Sure, absolutely. We go through maintenance, break fixes. And we follow a strict change management process that we put together. We make sure that our customers are notified fully prior to the maintenance. We discuss it with the team. And we create tickets internally so that we can keep track of what's going on. And that includes break fixes with incident management. If there is an incident and we need to change something immediately, we will. And planned and scheduled maintenance as well. So on the storage side of things, particularly with SAF, it's a little bit easier. We're relatively up to date on SAF. We're running the latest Firefly, basically. I think that we've sort of learned to hang back a little bit on the release of SAF. But we've been able to do things like add host, take away host, re-image them, update the kernels, largely without too much user impact. Great. What happens if our cloud goes down, Sheila? So the cloud has not gone down as of today, thankfully. But we've had individual data centers. We have. And we do encourage our customers to build in multiple data centers throughout the region, nation, and make it as redundant as possible. Yeah, we encourage pushing resiliency up to the application layer so that they can survive any single data center outage. Maybe Chuck, you can share about how we support multiple locations. Sure. So we do have multiple centers that we implement as regions. And depending on the type of the region, whether it's a national data center or a more local regional data center, we encourage people to bring up their applications in more than one site. Because with the local data centers, they might actually have a problem every now and then. They're not as redundant as the national ones are. Now, of course, this also brings a little complexity to the users, too. They have to deal with things like inconsistent IDs for images and so forth, too. So we try to make that as painless as possible for people by synchronizing all of the IDs and things like that. Or at least that's the plan. So yeah, there we go. Anton, can you share with us what are some of the default settings we had to change to make things work better in our environment, like our lessons learned, best practices? Sure. So I would say the biggest thing we went through was changing the RAM overcommit from 150% down to 95%. Just because in some of the higher loaded environments, as VMs actually start to use the RAM on the compute node, you run out of RAM. And then a random VM gets killed by the kernel. And so we were seeing strange behavior like that. And then we quickly figured out what it was. So we went from 150% to 95%. And actually now we're completely out of capacity in some locations. And so we're actually looking to change that a little bit. And I think we're looking at like 115%. But we're going to look individually into each environment and see what the actual utilization is. I think we're deciding that we're going to choose per environment. We're going to change that setting per environment. On some of the SEP side, so basically, if you guys didn't see it, if you guys saw the Time Warner cable talk on SEP, it's basically exactly the same thing that they are having problems with. We're having exact same problems. Closetively. So for example, by default, some of the default settings for recovery, so Max OSD backfills and Max Recovery, we set both of those to one. Because by default, if you have a failure, either a node fails or an OSD fails, the SEP cluster will basically DOS itself, trying to shuffle all the data around. So that was a huge thing. On some of our larger SEP clusters, we had an interesting issue where SEF, the memory utilization spikes up with SEF. The kernel isn't able to keep up with allocating the RAM, so we had to change the min-free k bytes setting in the kernel. We went to, I think we can mention, so we went from the default, which is like 90 megabytes to one gig. And we still had some issues, so now we're at two gigabytes. So that's helped us with that. There's obviously some other tuning things with Max PIDs and the Max port counts and stuff like that. We'll have to wait to change. Awesome, thank you, Anton. Chuck, maybe you can share with us how we add new customers to the cloud and how do we support them? Okay. So for us, we have an in-house tool to actually do all of the heavy lifting for adding users to the environment. Shout out to Tim Millick and our coworker here for putting that together, thanks, Tim. Now, as far as support goes, that actually begins before people even have access to the cloud, because we'll offer to meet with them, do what we call an intake call, where we discuss the cloud. And of course, a lot of these people have, they've never even heard about cloud before, but they find themselves being urged to go take a look and investigate. So we try to inform them of the ways of the cloud and get them to cross over. Now, as far as when they actually have access to the cloud, we send out what we call a welcome litter and it has lots of good information to internal and external pointers for resources, including some videos and so forth, very cool resources we've developed as well as ones that we found out on the net. And then of course, after they've had access to the cloud, we provide support via an internal IRC channel. It is, I have to admit, very, very disruptive sometimes during our day, but it is such a benefit to our users that we can't imagine not providing it now. And actually, I'll set a little bit of context. I know we mentioned this when we were sliding in on the couch show, but our cloud is serving internal customers within Comcast, so things like our products and services run on our cloud. And we have approximately 600 different projects or tenants running on our cloud today, about 1,500 different users. But we are internally focused. I just wanted to set the context for the room there. Anton, maybe you can share with us how we support custom code changes within the environment. Sure, so we keep our configuration consistent with tools like Puppet and GitLab, so we have our own internal GitLab. As somebody on the team makes a change, we go through a peer review process. So we need to have, I think, two or three plus ones from somebody on the team. And then those changes will get merged. And then as we're ready for a new release, we'll bundle those changes into a tag. And then the operations team will take that. We'll schedule a maintenance. We'll provide documentation on what are some of the changes. For example, after we launched Havana, we added live migration after the fact, some other stuff like that. And so we'll use Ansible to pull down the changes into each individual environment. And then Puppet will push those changes out to all the compute nodes in the environment. Awesome, Scott. Can maybe you share how our maintenance processes are evolving and changing to adapt to an ever-growing? Yeah, so when we first started, we had a small enough cloud that we could do maintenance and get through it. But we're running into the size issue. Bigger the environment gets, the more time it takes to get through a maintenance, like any one of these vulnerability maintenance that come out these days, a very large environment you could be spending weeks and weeks trying to push through. Our policy has always been one environment per day, no more, because doing more than one environment and introduces risk where if you didn't foresee the changes that you're making were going to cause some kind of issue with the customers on that cloud, then you've just kind of enhanced that by propagating that across multiple environments. We have recently improved our process for automating our maintenance, trying to be as hands-off as possible. We're using Ansible quite a bit more to create the playbooks necessary to make all the changes end to end, as well as the verifications and validation of the cloud after the maintenance is done. So what I foresee is going to happen is that because automation has improved for doing the maintenance, we will now have to go back the other way and start doing multiple environments per day just in order to be able to get through something in a relatively short period of time. We've doubled our environment in the last six months. We're doubling it again by the end of the year. Next year, it looks like it's going to be a pretty rocking year. Maintenance wise, from an operations standpoint, is something pretty much on the forefront of our mind. Yeah, so he mentioned basically doubling the compute environment, our storage environment. It's been growing extremely rapidly over the last really year and even more so over the last just three months. We're sort of exploding in size. I put up a little bit of a slide here just so you can get an idea. But sort of trying to manage that capacity growth is really difficult. So some of the things that we've done is we've looked into how are we going to build out stuff to manage some of this capacity need. So we diverged a little bit from typical recommendations. We're doing like four U boxes that have a lot of drives in them. I know the SEF community, even Sage himself, has kind of warned us about using the really big boxes for SEF. But we do make it work. It works probably. I think even the Red Hat guys, they're coming around a little bit on it. And they realize that it works a little bit better than expected. But it's definitely not insignificant to tune and run these. But it's also easier to build a house with bricks than it is with Legos. So we didn't want to kind of monopolize the entire time with preparatory questions. So I think we're going to open it up to the audience. We've set a little bit of context of what we do at Comcast. But maybe we can. The one thing I ask is either if you can come up to the Q&A mic here, or if you're on this other room, I can share my mic with you so that the questions are captured on video. So I think we're going to go ahead and open it up to questions. Hi. How big is the team that you have supporting your cloud? And are they dedicated to that team? So I'll take that one. I don't know that we can share exact numbers of how many we are on the team right now. But they are dedicated. We have three teams today, Dev, Engineering, and Operations. I would say that they're relatively small for the size of infrastructure we support. We also do support, which might be different than some cloud today, a lot of diversity in applications. So we're not just running a bunch of web apps. We run video apps. We run voice apps that are sensitive to jitter and latency and whatnot. We run video apps that have high bandwidth demands. Apps that have high packet per second rates. Now we're looking at doing some of the NFV stuff. And when you look at NFV, it's got all sorts of crazy requirements to it. So a lot of our engineering and focus goes into how do we support these things at scale? Part of our team led the IPv6 effort to upstream IPv6 and OpenStack. So we're doing some cool stuff and pushing the envelope. And so that also might add some dynamics to the size of our team. So we also work with a lot of other teams at Comcast that support us. For example, we have a good representation from our network engineering group here at OpenStack. So a good portion of what you saw in stage was from network engineering and other groups as well. Next question. We had a question over here. One sec. Other than your fame at the OpenStack Summit since 2012, what was really some of the value propositions you had beginning this journey? And as you've had your success listed up there, what additional value propositions do you encounter that you maybe didn't see before? So I'll try to take this and anyone can jump in as needed. And part two is how did you guys, what are some of the processes and role changes you guys had to really adapt to in making this move? So I'll answer the value proposition part, at least, or try to. So I think when we started this, there were a few options at the time in terms of what you were going to do with Cloud. You were going to go to a public Cloud, or you're going to pay some vendor to try and do it for you. And I think one of the nice things about OpenStack is that you can bring it in-house. You can be part of a vibrant community. So when we were looking at this, there were a couple choices available for running your own Cloud. And the thing is we were seeing increased velocity towards OpenStack. And so that sort of at least put us with the crowd. And we realized that we could build a community and build internal knowledge as opposed to sort of paying someone to do that. And personally, for me, it's great to be in an environment like this. It forces me to reach out to the community and talk to folks, as opposed to just being siloed. So can you ask the process part of your question again? How did we evolve over time, given? So that has been kind of challenging, because as with most people here at the summit, you have to start somewhere, and usually somewhere with practically no knowledge about the OpenStack and how to put together Cloud and run it. And we had to figure that out as we moved along the process. But the other piece of it, which has affected us quite a bit, is how do you get the user base there, you build it, will they come? You've got to really market it. And you've got to not only market it, but you also have to market it in the right way. That is a pets versus cattle. How are these different from typical virtual machines that may run on, let's say, VMware? Or how do you take a physical application and move it into the virtual world that works in the OpenStack environment? And how do you take an application that typically uses infrastructure that helps you provide that HA? How do you take that HA aspect and cook it into the application so that you're not relying on the hardware so much for your HA infrastructure? It's been an interesting challenge, and we've just kind of had to grow that along the way, kind of looking back, and I'm amazed at how far we've made it with as much as it's been going on. Next question? My question is regarding how you implement environment when there's still, I assume there's still some gaps with OpenStack and provide today versus what you want to do. And can you give us a list of those things and explain how you're going to cover those gaps? So I think one of the things that's probably a great example of that is IPv6. So we had to carry that internally for a while, but now it's actually upstream. So when you launch an instance in our cloud, you actually get a dual-stacked instance. You get an IPv4 and IPv6 address. So it's just an example of one of the things that we do. We either carry the code internally, or we have to set expectations with customers. And hopefully, with that expectation, there's some sort of timeline expectation that we may eventually get that feature. Maybe we could also share, maybe Anton, about how we had a gap by not having blocked storage for a long time, right? And then we actually worked with the community, found what others were doing, and brought in a solution. Well, you just answered the question. Thanks. Fail as moderator. I was just going to add, we don't offer load balancer as a service, for example. But we do provide great instructions for people and how to deal with that. And so we have some external tools for that. There's a couple of different ways how you can manage that, right? As somebody said earlier, we encourage people to deploy their resiliency on the application layer. And we tell just on every call, we basically repeat that with customers. And it's still not enough. But you can do that with the DNS load balancing. We do have some hardware load balancers. But that's not part of our open stack right now. So to kind of add on a little bit with that, and it kind of goes back to the whole process thing, one of the other things that we've encouraged all of our own internal customers to do is to kind of rely on themselves as a knowledge of information. And there are places within Comcast where users can ask questions, how do I do software load balancing? How do I do particular networking type things? And our user community has actually been pretty active in helping answer those questions. And sometimes they have the answers that we don't have, which is also very good. If you remember that IRC channel we mentioned, too, that it's getting to the point now where frequently other users will answer questions from newer users, too. It's kind of working out very well as far as we're concerned. Yeah, I was just going to add the same thing that Chuck just said. We have an internal forum as well that sometimes we push people to participate in. And it's just awesome seeing other customers giving feedback and advice to our other customers about how they're doing things. Cool. Yes, I'd like to hear some of the examples of onboarding process and how much freedom you give to the tenant or admin, project admin. And how do you control Cloud Sprouts? OK, what was the last part? How do we control Cloud Sprouts? Well, everybody uses Sprouts. All right, well, as far as how we deal with people, how we get them onboard in the first place, generally a lot of the questions that we have are related to how are our environment set up. For example, they want to do this particular thing in networking. And they need to know, OK, how do I make that happen in one of the clouds? Or is it even possible? And generally, though, it goes along those lines. And then there are also questions where we have to defer to other groups, too, as well. Because we're learning as our customers are learning as well, too. So we'll frequently refer people to other teams and people who have done things before. So it's interesting how users actually approach us to get on the cloud. And a lot of times, we'll see them actually come into our IRC channel and ask, how do I get on your cloud? I've talked to so and so, and they told me about it. I want to be on there. And then very often, we'll see stuff come down the management chain, which will then be passed off to this. But in all cases, we just generally push everybody to the ticketing system and say, this is where you start. And the whole ticket process that drives the intake will ask various questions like, what is it you're trying to do? How big is it that you're trying to do? What kind of resources are you looking at? Are there any special needs that we need to be aware of? And more importantly, do you want more information about the cloud? Do you want us to come talk to you about how to get on our cloud? So that's a very important aspect of the process. Well, one thing we do have right is a goal to automate a lot of that onboarding this year, right? So that it's not actually a ticket process just to even get set up on the cloud. Part of that whole process thing. Yeah. Do you want to address also the part about sprawl? Yeah, so we're still kind of in our infancy for dealing with sprawl in general. And I'm sure that's not helped with our capacity issues. It's so easy to get on the cloud that that's typical of any virtual environment. I'm sure there's quite a number of VMs that are sitting idle and not being addressed. But we do have quite a bit of things on a roadmap to try to address these, like looking at Janet or Monkey, and using Solometer to help identify the various VMs that may be dead weight or just kind of flat line and not doing anything. And the showback models will help encourage people also to make better use of their resources because they may eventually get charged back for that kind of stuff. Maybe we'll actually implement something like Cloud Minion. There was another talk, I think, yesterday on that too. And it was a very good talk. It's definitely something we need to look into. Have you noticed vendors approaching you to start using your cloud? Or to start using our cloud or to provide their cloud offerings? Because we did get that. No, well, like, give us an example.  Right. So our cloud is completely private to us. So we don't advertise anywhere externally for people to get that impression to come to our cloud. I don't really think so. So we do something a little bit different. So we have a lot of vendors in general at Comcast supporting applications and whatnot. And what we do do is encourage them to actually either build their own internal OpenStack clouds or use a public OpenStack cloud to vet their application on OpenStack before, so that when they come to try to run it in OpenStack at Comcast, it's already working. So we do do that. But we're not necessarily, our security models, everything are really focused around internal use today. We don't really extend that outside of Comcast or Comcast NBC and properties today. Hold on, I'll go to Mike over. Do you then, or are we? OK, sorry. Hi. This has been very interesting. I'm enjoying it. Could you speak to your capacity planning process and onboarding new apps and growth in your existing environment who owns the cap budget and things like that and what you brought into it? So I'll just start off with the storage capacity thing. So like Scott was mentioning, people will put in a ticket and they'll say, oh, I need half a petabyte of storage. And then we'll all freak out and we'll be like, wait, wait, wait, let's talk about what is it that you're doing so we'll get more information. We have a fairly lengthy procurement process. So it takes us a while to get things ordered, delivered, racked, cabled, networked, everything like that. So we do have to plan very carefully. We have to set expectations very carefully to let people know.  What are your IO requirements? Sometimes that's easy to get from people. Sometimes it's very difficult. And then usually like for larger projects, it's better managed on upper management levels. And we'll know well in advance that something large is coming towards our way. So we'll be able to order things in time. One of the things that we're actually just kicking off because we do struggle with this issue is we've actually been building out an internal CRM, customer relationship management, right? And actually reaching out to all our customers, trying to understand their capacity, right? We're almost trying to operate a little bit as an internal business unit, right? And saying like, hey, what's your projections? Where are you? What are the issues you're seeing? What features do you want? I'm starting to build that data so that it can better drive our decisions. I've really enjoyed this discussion. And you guys have really been a great success program for OpenStack. And some of the operational processes you've clearly thought through and have in place are really well done. So thank you. I had a question about as the enthusiasm for OpenStack grows at Comcast, how do you evaluate OpenStack projects and new functionality that OpenStack is releasing and how you determine when or if you'll incorporate those new projects into your cloud? So that's a really good question. And I think I don't have a really great answer on that one. But what we really try to focus on are the core projects. And then beyond the core projects, if people are looking into or asking us specifically about projects, we have a lot of folks here. And we keep our ear to the ground on what the current state of OpenStack projects is all the way from incubation through eventually being promoted. And so we do try to lay out a timeline for those customers or goal if we're going to ever accept the project into our own cloud. One of the cool things is the ecosystem that's bubbling up at Comcast where, for example, we actually had a database team approaches and saying, can we actually build and run Trove on top of your cloud and work with you on this effort? And we're like, hey, that's great. Because we don't have the operational capability to maintain a database as a service platform. We're more an infrastructure as a service team, this specific team. But this team that happens to be nearby us saying, hey, can we do this? Because they see this as a natural evolution of where data persistence is going. And so they're working hard. They're testing out Trove. They're doing all the proof of concepts, whatnot. And we're trying to figure out, what does this ecosystem model look like? And we have other teams doing similar models with things maybe up the stack or parallel to the stack. And working to integrate within the ecosystem to be able to deliver a service. And we also have a feature request page that we send our customers to. If they want to run something and they think it's really cool and there's a bunch of them, we just send them to the page. And then our architecture team takes a look at it, works with development, and eventually rolls it out. Hi, I'd like to know how your team built up the open-stack expertise. It seems that it's a very mission-critical applications. And obviously, there has to be some systematic way of developing skillset, including, by the way, a new hire, as well. I think being picky was one of them. The process of picking talent over the last several years has changed. We started off with just a few of us on the core team. And have had to grow just out of sheer necessity in order to be able to grow our cloud. We, at some point, got big enough that we were just one team. We had to split. And that's where operations and engineering and architecture kind of diverged a little bit. But despite that, we all work extremely closely. We're all in the same area. We talk to each other every day. And we act as still one big team, even though some of the stuff that we do is different from each other. From a talent acquisition side, I think that being picky about who we bring into our team has been very important. And it's not necessarily expertise, because open-stack expertise is actually very difficult to acquire as a general rule. All around us, we have lots and lots of Linux engineers. And a small fraction of them have heard of open-stack. And then an even smaller fraction have even played with it, or have much knowledge about how to install it, or do anything. So we do a lot of training internally. But we've had a lot of success stories, bringing on Linux engineers from within the company and network engineers. And we've also had success. I think our operations team is a maybe you're not tooting your own horn there a little bit. But we've had a great success program in terms of hiring green engineers, young engineers coming out of college, or just one or two years of experience, and then training them up as well on the operation side. And so we've pulled from all directions. Yeah, so it's kind of that evolving process. We're trying to find experts. And then we've come down to let's get some more juniors in there and train them up. And now we're about to kind of look at the other side again. Let's try to get some more experts if we can. But to kind of restate what Andrew said is that as our user base has grown within Comcast, and we've got over 1,500 users now using our cloud, we have been able to pull from that resource of users who now have experience as a user in our cloud, know the technology, know how to deploy their applications. And we have brought a number of them into our team. And they have been assets to us with regards to continuing to grow. And I think there's one last place for our source is actually the community itself. While it is competitive, the meetups we run, the operators meetups, the local meetups have been a great source of talent for us. And also being like, I think there is an attractiveness to a lot of engineers for working for somebody who's actually operating OpenStack versus being a vendor of OpenStack and having maybe a slightly different agenda or what might not be. So we actually are an exciting place to work because we really do care about OpenStack. We care about open source. We care about upstreaming the work we do. We care about our community participation. And I think that those values have helped draw in a wonderful team to kind of contribute back there. We got three minutes left. So a question about release scheduling. I understand not just upgrading instantly on what you've got running, but as you're planning out your next session, why are you staying so far behind? So we have internal patches that we have to carry and vet against the next version. And I think that's probably the primary reason that we get dragged down. Even after an OpenStack release comes out, we've got a dev cycle of I think we've evaluated it to be on the month's time frame to basically integrate our code into the new releases. It kind of depends on where we are. But primarily, like Neutron, we have a patch for Neutron that's been pretty painful for us to carry along. I mean, I think one of the things we do try to upstream our patches. But one of the things, too, is we didn't focus on building out CI, CD early on, continuous integration, continuous delivery. And because we do carry custom patches, right, not everybody needs to carry them. We had some stuff that we were doing with QIS that we hadn't upstreamed yet, or hadn't been able to get upstream yet. We're working to get that upstream. But we've also built out a good CI, CD pipeline now, which I'll hopefully enable us to do those things faster. I think maybe time for one last question.  Anybody else have a question? Hi, guys. It sounds like you're well on your way towards hyperscaling OpenStack. If you could give one piece of advice to maybe some of the people here who are just starting their journey, what would it be? Just what we were talking about earlier, don't write custom code. No, let me rephrase that a little bit. Upstream your code, right? Work with the community. Upstream it, right? We did that with V6. We no longer carry anything with V6. Works great, right? So I think that's a great example. There is investment to work with the community, but we feel it's well worth it. And kind of to the community, one, don't give up. Because OpenStack is an extremely challenging thing to set up, especially if you're coming into it new and you don't understand what OpenStack is or how it all works. It takes a lot of effort to get it up and going. Don't give up and rely on the community. Come to the summits. Come to the meetups. And take advantage of the vast amount of knowledge that happens to be everybody else has already had that kind of pain. Yeah, and I think the community is key there that you can kind of get that help leverage it. We have one, I'm going to kind of use a moment to pitch our favorite event, which is the OpenStack Operators mid-cycle meetups, right? So in between the summits, the big summits, so every six months, three months from now, right? So probably first week of August or late July. Somewhere, I think, unfortunately for our international, it's usually even in the US, we get together with just operators. There's no salespeople. There's no vendors, that type of stuff. It's people who operate OpenStack Clouds. Everything from people operate like a little small 20 node cluster to close to hyperscale-sized OpenStack. And we sit there and talk about our best practices, what things work, what pieces of equipment work well, what ways work well, how do we solve keystone scaling, how do we solve that, all these type of things we learn there. So we're not in this on our own. There's the mailing list, there's IRC, but the meetups have also been great for us. So I think we're out of time. So thank you all, and we're going to hang around here for a few minutes or a question. Thank you, guys. Take a bow.


Good morning, everyone. We're going to start our next session and I am delighted to introduce our next speaker who is Craig O'Connor with the Office of Renewable Energy and Environmental Export with the US Import Export Bank. This is a really important piece of how the US government assists businesses, American businesses in terms of really building their markets and and really deploying technology around the world. Thank you very much and thank you for the opportunity to be here with you today. I'm going to talk very rapidly about Exxon Banks. I'll give you a quick overview of our programs and more importantly and interestingly for you some of the deals that we've financed. See my colleagues from the Department of Energy in the back. So and I'll try to allow time for a couple questions. So actually it's the Export Import Bank of the United States and we were started in 1934. We're located on Vermont Avenue and basically set up in 1934 during the height of the Great Depression to provide financing with financing with scarce. It was challenging. Obviously there's many markets that lack the financing necessary to rebuild and buy US made goods and services. So originally we're set up to provide some import financing, but we dispensed with that early in our years but never bothered to change our name. So we only finance the export of US made goods and services. Untership dust doesn't matter. I'll tell you about Gamesa builds US turbines, wind turbines in Pennsylvania. We finance the export of those. So Exxon Banks had an interesting history. We financed the Pan American Highway, the Burma Road during World War II, some of the first steel mills, the rebuilding of Western Europe. So our basic mission again is to finance the export of US made goods and services. We do look for a reasonable assurance of repayment and so I like to tell people we're not in business to make money, but we're not in business to lose money either. We're a congressional charter organization so every four or five years or so we get a new charter from the US Congress. The reason I'm in my current job, I've been at the bank over 20 years, started life as a loan officer there. I was talked into being the bank's environmental liaison officer back in 94 as a part-time job and now it's a full-time job for me and five other people as of 2009. And so the sense to increase our support for environmentally beneficial exports was actually written into our congressional charter. And so increasing our support for renewable energy is a congressional mandate but it's also a top priority for the administration. So how do we do that? Well, first we provide direct loans and that's very critical, particularly in times of financial turbulence. And so for renewable energy, Exxon Bank is able to offer repayment terms of up to 18 years. That's a long time. And as my esteemed speaker, Scott Scalar and Carl Gaywell mentioned, with renewable energy you're buying your fuel up front. And I tell the international customers, you know what, you've got a 20 to 25 year fuel price hedge. I was at a meeting on Vietnam wind developers, a lot of government officials on Monday at the US Trade and Development Agency, another agency you should get to know. And I always do that, ask for a show of hands. How many people think the price of oil and gas will be lower in 10 years? Nobody? No bulls? Well, you heard from our friends in solar. So it's obviously the price of solar and wind are going to go down. But notwithstanding that, if you've got your energy costs locked in for the next 20 to 25 years, that's a heck of a fuel price hedge.  Solar is already a grid parity. Solar is already a grid parity in a lot of areas in India. So I'll talk a little bit more about that. Direct loans up to 18 years. Our current interest rates based on 1% over comparable maturity treasuries is about 2.96%.  I can't get one either, so don't feel bad. But why is that relevant? Well, if you're talking about a market like Brazil, Brazil is an investment grade rated market. So CalPERS could buy Brazil bonds if they wanted to. But the interest rates in Brazil are very high. The average rates that banks charge to customers is probably 25%. And the longest loan you can get is probably three years, and that's only through a development bank. And so interest rates are very high throughout the world. Terms are very short. We supported a project in Canada where the banks were only going out eight and a half years. Can you imagine trying to pay a home mortgage off in eight and a half years? No way. So this is not so much different. So if you've got an 18-year term, it enables the per unit costs of renewable energy to be much lower. Now we also provide loan guarantees, and that's 100% full faith and credit guarantee of loans that commercial banks would make to credit worthy buyers and credit worthy projects to buy U.S. made goods and services. That's another reason that we exist is because basically banks are limited in their ability to make cross-border loans. Every time a bank makes a loan, it has to hold back a little bit of reserve requirements. Part of the financial crisis is that those that were doing CDOs didn't hold back that capital. But that's another discussion we can have. So banks have to hold back capital, okay? Well, they've gone through a lot of capital in the recent experience. And so if they can make a loan, have it 100% guaranteed by Exxon Bank, we find a reasonable assurance of repayment. It's a win-win. So we're crowding in private capital, private financing to support renewable energy. We also have working capital guarantee. So if you're a company like TAS, Turban Air Systems, a company that Carl Gaywell knows very well. It's an emerging geothermal company. You're in Houston. You go to your bank and you said, listen, great news. I've just got $50 million water from overseas, but I need working capital to build my systems. And a bank looks at them kind of funny. And what? Your source of repayments overseas. And again, their ability to take a long-term cross-border risk. And so Exxon Bank provides a 90% guarantee to a bank like a Bank of America or Silicon Valley Bank to make that loan to the exporter to support their build-up to producer goods for services. Just in case you're wondering, over the past five years, since 2008, we've returned $1.6 billion to the U.S. Treasury. And that's because our loss ratio is very, very low. Some might argue too low, particularly those of us in business development, but that's Craig's opinion and not for attribution to the bank. But no, we have a great loan officer core. And so they do a great job at structuring these loans, and so we do get repaid. We've really assumed our historic role when financial markets were under some difficulty. We stepped in, and so when I joined the bank, it's probably $8-9 billion a year, and financing is probably a big year. Well, last year, for the third straight year, we broke a record $35 billion, and that supported $50 billion in exports. Now, this is with the same number of people, so our productivity is really high. People are working a lot longer and sweating as well, but that's another story. But we're achieving our mandate, but hopefully that financial markets will come back to normal.  They needed it. Well, Germany's economy outgrew Exxon Bank. They developed their own savings, and then we focused on other markets. So we're kind of, one of the kind of the sub-lines of Exxon Bankers were a lender of last resort in many areas. I like to be the lender of first resort for solar because I want to grow this portfolio in case you're wondering. We started the office in 2008, and our growth has really mirrored the growth of the industry. So we had $30 million the first year we started. That's not much. Next year we had $100 million. Next year after that we had $330 million. Next year, the last two years, our total has been $1.6 billion. So we're demand-driven, right? So what I try to do is increase the probability goods and services will be demanded by saying, well, this is a great financing package if you buy U.S.-made components. So I'm proud to say that we're probably the number one, number one or number two lender for solar in India. And so that's supporting over $480 million for the projects. We're probably... The Indian government has a national solar mission, so they've committed to get 20,000 megawatts by the year 2022. And so in this first, they've hit the first gigawatt, the first 1,000 megawatts. We've financed probably over 30% of that. And again, because of the challenge of short bank-term tie-interest rates, we financed the first commercial concentrated solar project, CSP project for Arriva. Now, Arriva is, yes, their French company, but they bought a linear Fresnel company in Mountain View, California. So that export and the goods and services that go into that export were used in a project in India that's the first CSP. We financed recently a rooftop project in Mexico, a $780,000 loan guarantee that we provided to UPS Capital to a business in Mexico for 10 years to buy modules made by Cineva in North Cross, Georgia. So we don't have a minimum project size either. And so we can either... Echo Heat Pumps is another one of our customers at very, very small loans. Some of the other interesting deals I've mentioned can be made a couple of times. So we financed a project in Honduras, 18 years because it's renewable energy, we also support local costs, site preparation, grid connection, up to 30% of the U.S. contract amount. That project, Serra de Hula, provides 6% of all Honduras' power and does so at a rate that's half the cost of existing costs of generation. Why? Because they're importing bunker fuel. They're importing diesel for generation. Now they've got a 25-year fuel price edge. That project is the largest wind project in Central America. We've got a few more as a follow-on. So really the entire bank is created to supporting renewable energy. We have my unit that's really business development. We try to structure the deal so they look like they're bankable. Then we have a project finance group that they're really experts in their field and they're really committed. So they've done some of the first independent power projects around the world. Independent power projects means a government agency doesn't own it. So it's privately generated, privately financed, privately owned. Maybe a government utility like CFE in Mexico agrees to buy the power. But again, it's crowding in. A lot of investment that can be used to support infrastructure development. So this whole idea, you know, it's a policy form. So the whole idea is, you know, in 2004 you probably had one U.S. wind manufacturer. And that was GE Wind, right? Because of the PTC, now you have probably well over nine wind turbine manufacturers. These are high-wage, high-skilled jobs. And guess what? You've got this production base in the U.S. Now these guys are ready for export. So the first time I ever met Kameza, I said, well, you guys are a U.S. company now. They kind of looked at me funny. No, we're Spain. As far as I'm concerned, you're a U.S. company. And think about the customers in the Americas that want to buy U.S. made goods and services. Nobody can touch Exxon Bank and the financing. So we can be very important to you. So following project, one of the first wind projects in Uruguay. Uruguay's government provided special incentives for wind. So we're there. Obviously we're looking for GE. Again, we're demand-driven. Look for credit worthy source of repayment. Some of the other projects you may be interested in. We financed a rooftop project in Barbados. So our engineer wanted to go inspect the project. We said, no, you don't need to. This technology works. Nice try. So that was an individual that did that. It was Williams Industries. And he put solar panels on a recycling facility. Why? Because it saved him money. All right? I mean, he wasn't necessarily green. The green was, hey, it's going to save me a lot of money. And I've got a fuel price edge. So that was another deal that we did. In the geothermal side, this is interesting that Carl made the point about geothermal technology and how increasingly it's becoming modular and you can use geothermal with lower heat. So we financed a $1.6 million small, $150 kilowatt project in Turkey. So it was three modular Pratt and Whitney units. And that was a loan guarantee to one of the banks that made that possible. So I think I'll stop there. I think I have a few minutes left, right? So one or two questions. Yes, sir. It's Morocco, isn't it? A bold phenomenon. Absolutely. 20 years ago, King Muhammad was doing that two tables and making Morocco the same. They believed that 200 gigawatts of renewable energy in Morocco were having, we've had substantial problems getting maximum pay attention to this project. Cool. Cool. Please do, because nobody's talking to you about it. So trust me, I'll get my teeth into it and won't let go of it. Yeah. I think, well, one issue is local content in Brazil. A lot of countries have local content restrictions. And so Brazil is one of those. So we finance wind blades. We didn't finance the turbines because Brazilians are saying that for projects they support, 60% has to be made in Brazil. So that's probably one issue. The other issue is the government's pretty, they jealously guard the electricity sector. So it's hard for new companies to get involved in there. But let me tell you, since you brought it to Brazil, one of the speakers you're going to hear about in your upcoming sessions is Steve Wilburn. And they, the company Firm Green, and they had a project to capture the methane gas from one of the world's largest landfills in Rio de Janeiro, the Novo Grimacha landfill. In fact, there was a documentary film that was made about it. And so they were selling, they were cleaning the gas, they were selling the gas to Petrobras, which is a state oil and gas company, for use in their facility. And so we provided a 12-year direct loan, and that project was our project of the year, this past year. So that's a project we're very proud of. Firm Green's a small company. They had the proprietary technology. This is good use of outsourcing. They used the outsource manufacturing of the systems to various, you know, metal shops all throughout the U.S. Midwest. And obviously they're hoping to replicate that. Okay, one more question. Yeah, great question. I mean, for the bank as a whole, it's in the thousands. For renewable energy, it's probably, it's probably firms that we know, probably less than 100. But I mean, you look at the solar manufacturers in the U.S. and you could probably count them on one or two hands. When one or two hands. So, again, we hope that more companies will get engaged. And look, companies are born global.  You've got to think immediately about global resources. Department of Commerce has a system to help you find a distributor. They'll do a customized market search for you. You know, they'll help you with your issue about market access. We've got the financing both for the international customer as well as the U.S. company to get ramped up. So there's no, and I tell you, we have the venture capitalists and we're locked in with some of those. Basically the pitch is send us your new companies and this is how we can help them export. They're born global. And the VCs will ask right away. It's like, okay, what's your plan for going global? And there's, there's no, we've got great technology in the United States. A lot of it's automated. A great place to manufacture. I don't have to tell you that. If the policies are right and you, and you get an industry like we have with solar, like we have with wind, that's an industry that can go global and be successful. And the financing is a key part of that. We hope to keep growing with the industry. So thanks a lot and look forward to hearing from you. Great. Super. Thank you. Thank you. Dr. Danielson is going to be speaking now in the caucus room, but I want to just say a special thank you to Craig. I think that it was great that you were here today because I think that we are really on the cusp and that this is a very, very exciting time. We are looking at renewables exploding across the world in terms of thinking about market. The opportunities are endless. The new technology and companies are so fascinating with many, many terrific stories.  So thank you very, very much for being here.


So first of all a little bit of background. What is the preload node? What is a preload node, right? So the preload node is a node, a GoFPS node that we set up to help out JS IPFS. So JS IPFS, if you're running in a browser, you cannot establish a direct TCP connection.  I mean, so what we did, what we have is we have some some nodes that on the one hand, they will actually be work as a DHT proxy. So you are in JS IPFS, you want to load something from the IPFS network. Well, you cannot connect to the DHT because you cannot do TCP connections. So it acts as a proxy for JS IPFS and it also serves as a place to persist the the storage of the JS IPFS node so that if you reload the page, your content is still there. So what is a problem with the preload nodes? Well, it turns out that the preload nodes also need to act as bootstrap nodes. We'll get into why in a moment. And the bootstrap node addresses are hard-coded in JS IPFS in the library itself. You can overwrite it if you're running a node app or if you're writing an application on top of it but people usually don't do that. And since the peer IDs are hard-coded, you cannot just, you know, add more bootstrap or preload nodes because they will not connect to them and you cannot change them because you cannot change to a different node because, of course, the peer ID is hard-coded, right? And we to start with PL had two preload nodes, node 0 and node 1. Node 2 and node 3 were added later, but the problem is that older versions of the library only know of node 0 and node 1 and people are not upgrading because upgrading is a pain. Basically, not enough care has been taken to ensure compatibility between versions, API compatibility between versions. I think things are getting better, but older versions, so upgrading from an older version to a newer version can be a hassle. So people don't do it, which means that node 0 and node 1 are basically getting hammered. And I discovered this during my first, the first thing I did when I joined protocol labs, which I had to scale up the preload node, node 0 and node 1. And even after the scaling, because they were crashing, I don't know, every half hour to one hour, basically, the Go IPFS was running out of memory. So we scaled them up and now they're crashing only twice a day, more or less. So it's better, but it's, of course, not ideal. So how do we solve this? Well, we basically do a man in the middle attack. So we create something kind of a reverse proxy. I mean, that's the way you scale things in, that's the way you scale things in web 2. Well, let's grab a page from the play to one of the web 2 playbook and let's create some sort of reverse proxy and load balancer, which would impersonate the node. It would basically take over its PRID, its private key, its private and public keys, and it basically uses a bunch of Go IPFS nodes in the background to downstream to actually get the data, right? I called it Katsune, which in Chinese mythology is the magical, many-tailed fox. So it's one head, which is where the upstream nodes, the JS IPFS nodes connect to, and many tails where are the other Go IPFS nodes. So I had to figure out what does the, how JS IPFS talks to the preload node, right? So basically, yeah, I mean, it connects via WebSocket, to BitSwap via the WebSocket transport, right? So it uses BitSwap, but it also uses the API port and it calls this endpoint called API v0 refs. What that does is it forces the, what the API does is it forces the Go IPFS node to go under DHT, get the, get the block, get that SID, and you can actually tell it to get it recursively. So it will actually get all of the nodes that this node references through IPLD, okay? So it has, basically it has two connections. It has the HTTP port and it has the WebService port. So I dove into how this works, right? So on the one hand, for proxy in the DHT, the application will call IPFS get, for example, and JS IPFS will make a post and request the reps recursively. It does a recursive get. Then the preload node will fetch the content, and then JS IPFS will send a want over the BitSwap channel, and then the Go IPFS node, well, the preload node will send the blocks back, right? So I had to dive into how BitSwap works. For persistence, it does the other way around. So the application calls IPFS add, which adds it to the node's local storage. Then it sends another IPFS reps with that CID. And what that does is the preload node will query its peers for that CID. And of course, the only peer that has it is the JS IPFS node. So the JS IPFS node will respond to that want with the blocks. So we have, on the one hand, we have a want coming from upstream and going to the downstream peers. And then we have, we can also have the want coming from the upstream peer going to the downstream peer. Yeah, anyway, we have wants and blocks going both ways, right? So we have a couple of challenges here. First of all, I did not know Go. So I took this as a chance to learn Go. And then, of course, diving into how to use the P2P and how BitSwap works. There was a couple of challenge. So one problem is that BitSwap actually, when you send a want, it will not send the blocks over the same stream that it got the want on. It will open a new stream. So I get this stream from this, from that upstream node, okay, who requested this, excuse me, because I do not have the association. So I had to make it to keep track of who wanted what, so I could send them what they wanted. The other part is that when you do a request to API these zero refs, you do not have the peer ID. So you do not know who requested that ref. So you do not know who to set the blocks to again. Sorry, who to set the wants to. So the downstream node sends a want and says, okay, who requested this. So right now, what I just, what I did was, okay, just send it to all of them, kind of like Go IPFS does, but there's an optimization that can be done later by the, using the Pierce IP address. There was one unforeseen consequence or effect of this. So on the one hand, Kitsune does not participate in the DHT. Basically, it's only doing BitSwap and of course the identify and pink protocols, the basic to the P2P protocols and it's a bit swap. Which is interesting is that I'm getting the, when I get a don't have message from the downstream peer, meaning, yes, I do not have that. I do not have, I do just represent me a want and I don't have it. So Kitsune will call API v zero refs and actually have the, have that, that the go IPFS node query to hold the HTU, which is basically what also needed for JS IPFS. But the effect of this is that this can be used as a generic one to one to one way P2P to P2P to P2P beer bridge. I might have done a one to extra to P P2P to P2P bridge. Meaning that, for example, you're in a private network, right? You have, you have a private swarm, a private network, but you want to get some content from the public network that happens. For example, when you upgrade a go IPFS and go IPFS needs to upgrade the repository and it does it does a query to get the repository upgraded by of the binary. And I had that, we had that problem in our previous job. So now you can actually peer in with your Kitsune instance. And that will actually get the information from the public swarm without exposing the information from your private swarm. You can also even if you have your private swarm, then you can also do a post to API v zero refs on Kitsune. It will get it from the, from the upstream nodes from your internal nodes and expose it to the world. And it's also, I mean, yes, we're all peer to peer. Yes, we're also supposed to be connected to connect to each other, but network administrators beg to differ. They, some network administrators really want to control what traffic is coming in and out of their networks. So Kitsune will allow them to have computers that will host that participate in IPFS, but without having to allow everyone to connect to everyone to everyone on the internet. That's just some of the ideas I got from possible consequences from from what we're doing here. So next slide, please. So what's our current status? Well, we do have a GitHub repo. And I have actually, it has actually been able to transfer content both between to go IPFS and between to JS IPFS node. We actually have a test test that will check that the preload node works. And that's what I've been using to test this. And a few next steps. Well, the one optimization I mentioned earlier about just sending the one depending on the IP address that the rest call came from. For production use, we use, we will need to add metrics. I would really like to add configuration because right now it's just configured via command line flags. And again, for production use, it's more convenient to have a file or environment variables. Yes, tests, we need to add some tests also loads testing. I have no idea how this will behave under load. Some cleanup of the code. And right now the private swarm thing is not yet a thing because we do not have, have a swarm support. A big thanks to Lydl, to Max, to Adene, to Horopo. They were all extremely helpful in developing this. And thanks for, to everyone in the cohort because it was a blast. It was an awesome time working with you guys. First of all, for web sockets, you actually need to have an SSL certificate. So I was thinking of getting my own SSL certificate. And then I discovered Ngrok. So what it does is it creates a proxy, an SSL proxy. And my proxy is right now, this host here, FB92, blah, blah, blah, is being proxied to 428080. Now here is where Kitsune is running. And I'm telling him, telling, telling it there is one downstream node, which is my, my Go IPFS node on my local machine. I am telling it to listen on, to listen for bit swap on 24001. And I'm telling it to listen on web socket on 28080.  So if people want to actually use it as proxy, they don't have to enable the proxy functionality. So Kitsune is starting. And now I will just run the, the preload tester. Just need to change so that the bootstrap address is F9B2, which is basically the same address that we have down here. And so I'm connecting, as you see here, I'm connecting through WSS to this node. And this ID is the same ID that I have in Kitsune. So nothing in my sleeve. And basically this, what this does is it starts using puppeteer. It starts a browser, a node inside the browser, and it starts another node in Node.js. And then it adds some content in the browser. And then it tries to get it from Node.js. And hopefully it will work because, of course, you know, the demo gods are the demo gods. It is a bit slow just because of timing. And yeah, the preloader is working as expected. It basically published this content, test content created blah, blah, blah, blah, blah. And it says, yeah, it's working. I mean, if anyone wants to see a little bit more about how it works, then just ping me at any time.