Shipmates, it's the 30th of March, 2020. Mick Pondsmith and I wanted to provide you and your families an update on COVID-19. During this extraordinary time, what remains constant are our top three priorities, taking care of you and your families, being mission-ready, and supporting the whole of government effort. This past week, we've seen a rise in numbers who have tested positive for the coronavirus throughout the fleet. We continue to take this threat very seriously and are working aggressively to keep sailors healthy as well as to prevent further spread of the virus. We have to think, act, and operate differently right now to both protect sailors and to remain mission-ready. This is not business as usual. That is why many commanders have implemented a 14-day fast cruise for units preparing to get underway, which will conduct important training evolutions, exercises, or deployments. Additionally, we have implemented a 14-day restricted movement policy for new recruits at RTC, and the entire RTC staff will also remain on base for up to 90 days to minimize potential spread of the virus. We are also supporting the whole of government approach in many ways. USNS Mercy arrived in Los Angeles last Friday, and USNS Comfort arrived in New York City this morning. We also deployed two expeditionary medical teams, one to Dallas, Texas, and the other to New Orleans. 2,200 Navy medical professionals are onboard these ships, which will serve as referral hospitals for non-COVID-19 patients. Another 1,000 medical personnel are awaiting orders to be deployed. In this fight, our Navy medical team is on the front line, fighting to care and treat the American people. Other sailors from our reserve and engineering communities may join the coronavirus fight soon. You all have our thanks and our gratitude. People are the lifeblood of the Navy, and we are counting on every sailor to take this outbreak seriously. While we recognize some new COVID-19 policies place a burden on you and your families, these actions must be taken to ensure your safety and also to maintain mission readiness. To the families and loved ones at home, thank you for your support and understanding. This is a hard time, and we could not do this without you. I'll end with this. As military professionals, we prepare daily to deal with adversity, uncertainty, and conflict. Our sailors and their families are resilient. We know you will set an example for your friends, for your neighbors, and in your local communities on how to make personal sacrifices in service of the collective good. So stay safe, shipmates. Americans depend on us for security, and we will not let them down.


Okej, hello, everyone and welcome. I think we'll get started. I think it's about time, so welcome again. I hope that you are enjoying the Drupal console far. Here today, we're going to talk about revisions everywhere and specifically in Drupal core. More exactly, what are we going to talk about? We're going to start with a problem statement and going to explain and lay out what I think is the current problem. We're going to look at how other systems approach this particular problem when it comes to revisioning. We're also going to look a little bit about some Drupal history, past discussions on this topic. And then we're going to look into what we can do about this, how we can approach this moving forward. We're also going to cover a very quick demo of how we approach this currently in Drupal 8 contrib. And then hopefully at the end or even during the presentation if you want, we can have a discussion about these things. So again, the goals to discuss some improvements to the entity API, discuss inclusion in core over a long term perhaps. And again, feel free to interrupt with any questions.  It's a core conversation, right? So if you have any questions, please step up to the mic and feel free to interrupt at any point. Okay, first, before we start, quickly about myself, a long time contributor to core and contribute space as well. I'm from Sweden. I'm working for a pharma company trying to make Drupal work essentially. So it's really quick about me. So what is the problem? I think we've all been here just trying to update the node and suddenly it's gone or you deleted it. You want to undo, you can't, you're stuck. And we all know that this is not necessarily the solution to the problem when it comes to undoing a delete or regretting, right? This can't be undone. And another problem that we have since our revisioning system doesn't support concurrent editing, we have to do things like this. That we see if you open up or if two users modify the same node at the same time, we simply have to let one of the users know that, sorry. We can't save your content. Someone else already updated it. So now you have to go through hoops and copy paste or redo and it's a little bit messy.  We're supposed to manage our content. For instance, as we just covered, it's not possible to undo a delete. At the moment we are blindly overwriting content when we're updating your changes, they can be gone in a second. If you do the wrong edit, if you edit the wrong node, if you don't have revisions, turn on your node, they are gone forever. Your changes could be big changes that someone has worked on for a long time. And the answer is not to revert to your backups. We should do better than that. And again, concurrent editing is not supported. Again, we're content management system. These are sort of the things that perhaps Drupal should be able to support. Don't get me wrong, our entity API is good. It's really good. But we don't really use it enough in core. We don't even have it turned on by default for the default content types to come with the standard installer. Perhaps we should do that. Another thing and another problem I should say is that Drupal is distributed. But we haven't realized that our content is also distributed, right? Content might be coming up from local development if you're preparing a campaign, if you're preparing some content. Content might be coming from an editorial staging environment. It might be user generated content in production. We might integrate with other systems where we pull in content from. And there might be things like API clients writing and pushing content. So our content is distributed. It's coming from everywhere, okay? And we don't necessarily consider that when it comes to core. So some of the use cases that are impossible or at least difficult to deal with at the moment are simply being careful about our users data. That's difficult because we don't do a very good job at taking care about our data, not losing it. Concurrent editing, workflows, content workflows, content staging, konflikt management. When it comes to conflicting revisions, etc. These are some of the use cases that are really difficult to solve. Yet, I think very, very important for a CMS system like Drupal to deal with. And these are really things that matter a lot to our end users. We simply want to manage our content, manage our workflow. We want to be able to move and share our content around different systems. Or at least within the site, even. And it's about letting our users know that Drupal actually cares. We won't go and lose your data just in a second just because you update or make a wrong update or happen to delete the wrong node. We need our users to trust Drupal that your content is safe. We simply want to relax and just know that everything is going to be fine. That's the bottom line, really. And a CMS should do this, I believe. So before we just go and revision all the things, let's talk a little bit about systems that do care. How have other systems solve this? Other systems that care a lot about your data that just don't want to lose it. So two types of systems that I'm going to look at. Distributed version control systems and databases. There are capable of true multi-master replikations. These are two types of systems that have great capabilities to deal and take care of data, not losing it, revisioning it, and so on. So we're going to take a little bit if we look at this. So to start with version control systems, we all know Git. Git assumes that your content is distributed, that the content can come from everywhere. Every change in Git is a new revision. Even when you delete the file, that's a new revision. Because you don't lose files with Git. Every revision has a universally unique hash so that we can identify it across our systems. Okay, very important, because again, our content is distributed. We need to be able to not only identify the general content, but also what revision is in what system, etc. And what's also unique with Git is that we're tracking a revision history, a revision tree. Every revision has a parent. It's just not a coincidence that the revisions are after one and each other in a database table. They actually have a parent. Okay. So that's some of the capabilities and features in Git that makes it a good system to take care of data. The same. We're going to look at multi master databases. There's a list on Wikipedia, on databases that are capable of multi master replication. I'm going to strike out a few of them here. The two ones in the bottom, they are proprietary databases, so we don't care much about them. MySQL and Postgres, you can do multi master replication, but again, the revision management and conflict management in those implementations are not really good. So we're not going to look at them. And I'm also striking out Cloudant here, because it's just a enterprise version of the first database that we're going to look at. So Apache CouchDB. What does CouchDB do to take care of your data? Well, very, very similar to Git, it turns out. Again, we assume that our content is coming from everywhere. That content is distributed. Every change in the database is a new revision of your document. It's a no SQL or a document store. So even deletes or a new revision. And every revision can be identified universally across systems. And every revision also have a universal or a identifiable parent. So very, very similar characteristics to Git, as we see. Another thing that I won't touch on so much in this session here today is that Apache CouchDB also have a very nice and reusable HTTP protocol for doing replication between systems. Quite useful. We're not going to touch too much on that. We're going to see a little bit, perhaps in the demo later. Yet it's a very interesting piece for many reasons. So just a little bit of Drupal history and some past discussions on this on this very topic before we go further. We've been discussing this quite a lot for a long time, actually. Even back in 2011, when we started taking notes about how we should design the entity API in Drupal 8, there was a buff with very interesting notes there. And there was a feature in Watchdog that touched on a lot of concepts that we're going to cover here. We also have Steve Ekter here in the audience. A very interesting session from Denver, where it was discussed about revisioning and workflows and struggles with editing notes. So these are things that touch on the same topic. I also had another session back in Denver about content staging in core, which is one of the use cases here. Not the only one, it's just one of them. All the discussions that we've had, they touch on sort of the same concept, which is a quite interesting concept called CRAP. What is CRAP? I want to quote chicks from the Watchdog feature where this was discussed. So in Drupal 8, and this is just for some context, this is when we were designing and discussing how to build Drupal 8. So in Drupal 8, there are two competing approaches, CRUD, which is create, read, update and delete, and CRAP, which is create, read, archive and prune. The latter would solve our issues with revisions, and thus is my preference. So that's a quick quote from the Watchdog feature there that chicks put out. So just to summarize the discussions very quickly. We do want to manage our content. We want to be able to manage our workflow well. It is a very complex topic. It has a lot of complicated use cases. However, the need for revisions, that's the lowest common denominator across all of these use cases, concurrent editing, content staging, distributing, moving, sharing content. The need for revisions is the lowest common denominator across all of these. Other systems have solved this problem already. We touched on git, we touched on couch to be how they are very good at the revisioning and managing content. And then the concept of CRAP versus CRUD. CRAP is a preferred way to do this type of revisioning. It's a append-only type of way to work with data. So create with archive prune instead of create, read, update and delete. So what can we do about this? What can we do about this in core and in Drupal in general? Well, first we have contrib. We can play around how much we want there really. We can iterate and mature the ideas that I'm going to show here. I'm going to give a demo on three or I should say two modules here. We're not going to demo the later one here. We're going to demo a implementation of the multiversion module in Drupal 8, which implements a CRAP way of storing entities. And then we're also going to demonstrate the relaxed web services module here, which is used as the transportation mechanism. We want to focus on the first part here, the first module. That's what's central to this topic really. The transportation mechanism isn't just a side effect in this case. So without further ado, we're going to jump over to a quick demo here, where I'm going to show how we could work with this and how it could be implemented. So in this use case, we have two sites, Drupal 1 and Drupal 2, as we can see here. We don't see the video, I can see. So let's do like this. So again, we've got two sites here. Drupal 1 och Drupal 2. Both sites have the multiversion module installed and the multiversion UI module, and these are what we want to focus on. We also use the relaxed web services module, which is going to be used just to transport around the content and show off some functionality around revisioning. So we're going to start by adding an article on our first site. Hello world, simply, we put in the title, and we save and publish this one. We also then have a simple drash command that we're going to use to just replicate from site 1 to site 2. We're going to move this content around. So we have drash replication start that we're going to run, and we have the URLs to each endpoint here. So we run that command, the replication is successful. So we open up site 2 and update that here. And we can see that the content has now moved over. It's been replicated over. And just to show off the sort of revisioning capabilities here, we're going to do bidirectional replication. We're going to do an edit now on site 2. And we're going to replicate that back to site 1. This could be, it could be on the same site, it could be concurrent edits. This is just one use case where we need revisions, where we need a better revisioning system. So we replicate this back to site 1, and we can see that the title has now been updated, say hello LA. We now have two revisions of this, right? So we can go in and look at the revision tree, which in this case is quite simple. It's just two revisions as we can see there. And we have the same revision history on site 2. It's the same revisions and we have a global revision hash here that we can identify across both systems. I'm sorry that the colors, they seem a little bit vague there, but I hope that you can see. We now make two incompatible updates on site 1 and site 2. So we're going to create a conflict in our entity here. So we make one update on site 1 and one update on site 2. We're going to introduce a conflict here. And then we're going to replicate from site 1. To site 2. So we're going to introduce the conflict now on site 2. And then we can open up the revision tree for this, and we can see that we have a conflict. We have a split in the tree here. I'm not sure you can see the lines, but we have two. We have splits there in the tree. We can of course continue editing. So we go back to site 1, make further edits, and we replicate that over to site 2, to continue the revision tree for our edit. And we use the drash command to replicate our change over to site 2. And we can see now that the revision tree has been continued from site 1. And we have an open revision. That was the conflicting one. Further, we're now going to delete the entity on site 2. We can see that it's gone. And then what we're going to do is that we're going to go back to site 1. We're going to continue editing this and show that the node is actually not gone. It was just on site 2 just a new revision that was flagged as deleted. But we're going to continue and editing it on site 1, replicate it over, and thus undeliting, so to speak. So now we're replicating further changes from site 1 over to site 2. And we can see that the entity is actually not gone. As in git or any other version control system. We've now continued our revision tree here for the entity. We can see that we had the first split further up in the tree earlier. And then we have another split where we actually deleted the entity on site 2. But then we continued editing the node on site 2 and replicated that. We can see here also that one of the revisions are missing. And that's because we only replicate open revisions. We don't need the whole revision tree when we replicate. So this is just an example where a better revisioning system is very capable of handling these types of things. And again, this was content staging. It could be concurrent editing. It could be multiple API clients doing conflicting changes. We're going to show another system integration now. Another use case. So what we're going to do in this case is that we're going to replicate our content. We're going to use that HTTP protocol that I talked about earlier to replicate from Drupal to another system, because it's all an HTTP API. And there are multiple different kinds of systems that use this same protocol. CouchDB is one of them. We're not using CouchDB in any way in our Drupal site. We're just using the same protocol and the same HTTP API. So we have our CouchDB here for instance. So what we're going to do now, since we speak the same protocol, we're going to replicate from Drupal 1 to our CouchDB instance. So we're going to move the content around here. And again, as I touched on earlier, CouchDB is also a very capable database to deal with revisions and conflicts. And we're using the same protocol here. So we've now replicated the content over the CouchDB. We have adjacent representation of our entity here. And to further again show our bidirectional replication capability. We're going to update the entity in CouchDB for instance. And then we're going to replicate it back. To see if that works. So we're updating the content here. Just updating the title and replicating then back from CouchDB back into Drupal. And then we're going to propagate these changes over to site 2 for instance. Just to show that we can move around and we can sort of shift around the content. And we can relax because we know that our revisions or changes won't be overwritten. We take care of the content here. It's a revisioning system. So now we have the new revision and we're now going to propagate this over to site 2. Just to show how capable this revisioning system is. So we move it from Drupal 1 to Drupal 2 under command. And when we reload the page here we have our seventh revision on site 2. Förder, we're going to touch on another use case. Another use case here is frontend applications. So if you decouple your application you have a separate frontend application. And using Drupal as your back end. Perhaps you're building an offline capable website. A website that you can use offline when you don't have a connection. That's a prime candidate for needing a revisioning system. Because you will have clients, you might have editors working on the site concurrently. Everyone being offline editing the same article.  Then they upload their changes automatically. They might conflict. We need a solid revisioning system to handle these conflicts. So I'm just going to give a very, very quick demo with a library called PouchDB. Again using the same revisioning protocol. The same replication protocol here that is capable of dealing with revisions in this way. So we have a super simple frontend application here that will store entities in IndexDB, in the local store. So the app is just a blank page at the moment. And I'm just going to show the IndexDB store here. So when we reload this page, it's now going to pull down the content from our HTTP API. And there we have the content. It's just replicating down. And now we can edit it. We can manage it however we want. I'm not going to show that particular thing in the demonstration here. But we have the full entity with all the fields. And again, if we push back changes, we can deal with that in a very elegant way. When it comes to revisions and conflicts and these kinds of things. So that's about it when it comes to the demo. Let's see if we can switch around here. So I just want to, before I continue, just want to give some credit here to the work that's been done for this demo. I want to credit Andrei here. Andrei is a guy who's been working on a lot of the code here. He's also a new core contributor. He's been submitting and getting in multiple patches in the course. I'd please give a round of applause to Andrei because he's done great work on this. Andrei is not here today, unfortunately, but he's doing awesome work on this, on this module, alongside with me. OK, so what can we do? When it comes to core, inclusion in core. Of course, baby steps, a first simple thing that won't bring us this whole system, but they will get us a little bit along the way. Perhaps we should just enable revisions by default in Drupal. We have a good revisioning API in core, but we're not using it by default in the standard installer for the content types. Perhaps we should also set tick the check box for enabling revisions by default in the entity type form. Such a simple thing just to use our revisioning API because we're not using it by default anywhere. Once more baby steps that we can do in the longer term, being a content management system or framework, whatever you want to call it. I think we should assume that our content is distributed. We need to assume that content is coming from everywhere and that there will be conflict. So we need to have a revisioning system and entity API that can deal with that. I would even say that we should enforce all content entities. They always have to use revisions. There shouldn't be a way to opt out from that. We shouldn't just be able to blindly overwrite or delete content with no turning back.  Then we can, of course, prune away deleted old revisions in background jobs or things like that. We don't need to store everything forever. That's not what I'm saying. We can prune away old deleted revisions. We can still do that. We should treat all changes as a new revisions, even delete.  I think that we should assign a universally unique hash to every revision so that we can do these things that I just demonstrated so that we can identify revisions and conflicts across multiple systems. We should also give our revisions a parent, not just coincidentally put them after each other in the table. We should make sure that every revision has a parent so that we know where it's coming from. Then, perhaps, as another step, we should expose revisions over a RESTful API. All these steps you are mentioning here are supposed to be something that is not going into the Drupal 8, or maybe some of them could be possible even in Drupal 8. For instance, I think the steps that suggest to assign a parent to revisions should be fairly easy to achieve right now. I do think so. I agree with you that there are definitely pieces of the system that we can bring into Drupal 8. So, for instance, the parent is something that we can do. If we do the parent, I also think that we should bring in this additional revision field. That I've introduced in multiversion, the revision hash, so that we have sort of a UUID for our revisions as well. These are two things. So, the additional revision field and a parent, and then perhaps the deleted field as well. That might be a little bit more invasive because then we suddenly need to start changing the logic around in our save and delete operations. But perhaps the revision hash and the revision parent is something that we can do. It's really not that big of a change, to be honest, what we've done in multiversion. We're not changing around the APIs too much. We're just using our APIs differently. We're just instead of deleting, we're just marking the deleted field, set that to true, and then we save the entity. It's not an API change. Necessarily. And then, of course, there are some API changes we need to do here, of course, so we don't necessarily need to move the whole system. Perhaps the whole system can move into Drupal 9 or Drupal 8.1 or 8.2. This is something that I'd love to discuss moving forward. And with those words, that's all I had to say, actually. I'm really looking forward for your feedback. Please step up to the mic. If you have questions or feedback or ideas, I'd love to discuss these things. Thank you. First of all, this is extremely impressive. It's great what you've been able to do. I'm thinking through applications here and what I've come up with so far is the use case of single node workflow or approval steps, basically the workbench moderation use case. There's the use case of needing to preview a group of nodes together. Basically the magazine issue publishing all at once use case. There's the election night use case where you're trying to stage what does the site look like if this person wins the election? What does the site look like if this person wins the election? That could be a whole bunch of things. There's the offline changes use case. There's the site to site syndication use case. Basically the network of affiliated sites use case and the synchronous editing use case, basically the Google docs use case. Are there any others that you have in mind? I think that covers it pretty well offline. There was a long list. The follow-up question is probably not possible to, or necessarily a good idea to try and solve all of those use cases in one big UI. How can UI tools be built on top of this API in such a way that these different use cases which will get built on Drupal 8 coordinated or not coordinating with each other? How can the UI tools be built in such a way that they're using the API well and not reinventing the wheel each time? It's a very good question and I don't necessarily have the answer to it, but what I definitely know is that we have a common lowest common denominator here. Using the same revision API here makes a ton of sense. How can we build the UIs on top of it? I think they can be fairly separate. And from an implementation point of view, you just need to use the regular entity API. You don't need to do anything different. You're loading, you're saving entities. If you try to load a deleted entity, it won't get loaded because we have introduced an additional method of loading deleted entities. So that's perhaps the only addition that you might need to take into consideration there. It's a good question. I don't necessarily have all the answers there, but it's not that different. You can just use the entity API as is today. You don't even need to know that the underlying system is using a multiversion way of doing it. OK, one more detail. If you don't mind, is the entity's base table the default? It seems like in each of these use cases that you need to know which revision to grab. If you're asking for node one, two, three, well, which revision of node one, two, three? How do you know which revision to grab? So when there is a conflict, obviously only one can be the default one. By default, a new revision becomes the default one. When there is a conflict, that's a trickier situation, and how the system deals with it and how the replication protocol that I talked about, how that says that you should deal with it, is that you need to have a deterministic logic. It needs to be, if you replicate the same conflict to multiple systems, the result should be the same in all systems without them needing to communicate to each other. So there is an algorithm, which essentially just do an ASCII sort on the revision hash, so the longest revision tree wins, or if they are the same length, the one that sorts the highest. It's quite simple as that, so then it's at least the same deterministic logic applied to all systems, and then it's up to your application to say, was that right or wrong? You might want to create another revision to solve a different conflict if you want to. Great. Thank you. Thanks. Larry. Hi there. So this looks great. You mentioned in terms of things to add, adding rest support for revisions. When we were specing out the rest system in core, we actually did include how we would handle that, because that's actually an area where the ITF standards have a very clear set of links and standards to leverage there. That's just a matter of we never got around to it, so that is absolutely safe to do an 8.1 as one of the maintenance for the rest module. Yes, please, someone write the patch and I'll review it for 8.1, not for 8.0. But definitely integrating that kind of stuff into our rest support is pretty straightforward, let's do it. Yeah, I agree. Thanks Larry. Hi. Hi. So a couple of questions about the whole headless Drupal thing where you might have an offline application, your example, and it may store the data in a different way, not necessarily using node IDs and stuff like that.  How would you handle that? So that's a trick or what, right? Because if you push back things to Drupal, they need to look and work like a content entity, a node in this case. So you would need to have some sort of transformation if you have a need in your application for it to look different. Yeah, it needs to look the same when you push it back, right? So the easiest way is perhaps to use it as is if you can, otherwise you'd have to have some custom mapping, transformation logic there. We don't push the local entity IDs, we only push the UUID and the revision hash.  You can work like universal IDs for all your applications. Okay, so another thing about the Google Docs scenario where you have like a really cool YC week that accepts all of that, would separate revisions still make sense for like, you know, two people are editing the document at the same time and each keystroke would generate like a revision potentially that's maybe to wasteful or how would you? It's a good question. I haven't thought of exactly how that would be implemented. I think I'd leave that to whoever wants to pick up the ball on that. But a new revision per keystroke, I think that would work. The default storage, which is mySQL, can handle loads of rows, so I don't necessarily think that will become a problem. And again, you can prune away all revisions if you have the need to. Because whenever there might be conflict, having two revisions is a fundamental requirement, otherwise you won't be able to solve conflicts in concurrent edits. It's a great question. Perhaps there is a middle way somewhere. Right, so about conflicts, say you have a big body filled with a bunch of text that may be two different edits at the same time and two different systems, only edit like okay, I'm fixing this paragraph and the other guy's fixing that other paragraph and they could be merged like Git does. Is there like a way to implement an algorithm that merges that in a smart way? The system that we have implemented at the moment don't deal with merging at all. It's up to your application or your business rules to decide how you'd like to merge. The system can't know how you want it to merge. It's the same thing in Git, right? You can do simple merges, but when there's a merge conflict, Git simply tells you open up the editor and manually merge this and then commit back, create a new revision with your fix. So it's left up to the implementation to decide how you want to do the merge really. We just flag the fact that there is a conflict. Thanks. So any ideas or plans for handling revisions for groups of entities? So for example in commerce we frequently need to care about revisions for a product and its variations or for an order and its line items. So how do you mean you want to refer to, if you have a collection of entities, you want to refer to a specific revision? Is that what you mean? We want to track all of those entities as one. I see. Git branch. Yeah, basically. Yeah, it might come in layout as well. Just to repeat the statement here. Haven't thought too much of that. I think that you could just implement some sort of event system, hook system and just create new revision whenever there is a change to parent entity of some kind. Yeah, but you're creating the same revision then. For example, if a variation changes and the parent product didn't, but you create a new revision for it and that revision tells you nothing about what's changed in the child. Yeah, I'd have to take a rain check on that one. Yeah, baby steps, exactly. In any case, great work so far. Thanks. More questions, thoughts, ideas. Yes, hello. Can you just work at all with the concept of forward revisions and default revisions? If it's working with forward revisions. Was that what you said? Yeah, so can you say forward revisions with all of this matter? So you can definitely change what becomes the default revision. When you create a new revision, it doesn't have to become the default revision. As long as it's just a new revision, it doesn't have to be the default one. The default revision functionality is essentially what you would use to do forward revisions. And you could, with the normal entity hooks, just say no, actually this should not be the default revision. So it's up to you, really. There is logic, of course, to say that we suggest that this is going to be the default revision, but you can alter that in a normal entity hook. If your module don't agree with that fact, and that this should be a not forward revision, for instance. So yeah, it's possible, definitely. My other question is, it sounds like some of this may make it into 8.1 to something like that. Possibly some of it maybe is in a little bit more of a gray area. So if this is something that ends up living in contrib for a long time, I guess my question is how hacky is it? Like, is this something that's going to cause lots of problems with other things, potentially? Or is it something that generally should work pretty smoothly with the rest of the Drupal ecosystem? So, parts of it, I definitely think that we should move into core, definitely. I think there are, as we just discussed here earlier, there are parts that make sense to move into core, perhaps already in 8, or at least 8.1 or 8.2. I think it might be too late for 8.0. But I don't think over a long term, sure, this whole revision system could definitely move into Drupal 9 or future. How hacky is it or will it cause troubles? It's surprisingly clean, I would say. It doesn't do any strange thing. It's only using our awesome entity API that we have in Drupal 8. We are essentially swapping out a storage handler to another MySQL storage handler by default. We can create, it's all captured in a non-storage specific trait. So you can apply it to a MongoDB storage handler if you want to, to achieve the same system. And it's enabling revisions, the way that you would enable revisions using the entity API. So it's just creating additional tables with the schema handler that we have in the entity API. So we're not doing any trickery or hackery to achieve that. So it's very clean. As long as you use the APIs as they are meant to be used, then you're going to be in good hands. So for instance you need to use entity queries for all your queries. Because the entity query API then will take into consideration that this entity is using revisions. So we need to query the tables in a slightly different way compared to if you wouldn't have revisions. So as long as you use the APIs as they are meant to be used, then it's going to be good. And of course if modules assumes for some reason that your entity would not have revisions, then it will cause trouble. But I can't come up with any sort of reasons or cases that would actually cause trouble in that case. I can't see how you would make those assumptions to be honest. So I think it's good. Okay, no further questions? Thank you everyone.


Welcome to dealing with materials data, in this course we are trying to understand the collection analysis and interpretation of materials data. We have done two modules so far, one is introduction to R, the second one is descriptive statistics using R, this is the summary of the second module on descriptive statistics using R. So we have learnt how to visualise data, so we have learnt how to use scatter plot, dot chart and stem and leaf plot for visualising data. Then we have learnt how to prepare rank based reports of data, which includes cumulative distribution, histogram and box and whisker plots. We also have learnt how to prepare summary reports for data, mean, median, variance, standard deviation and quantiles are the quantities that one calculates in the summary based reports. One has to learn about the significant digits in a given data, sometimes when you do this analysis computer returns a large number of decimal points, but beyond the point some of these numbers are not meaningful and so we should not report them. This is like for example, if you say that price of some three things is 100 rupees and each one costs how much, so we are not going to report the resulting number beyond second decimal place because below paisa there is nothing, there is no meaningful number that you can quote. So this is similar in all cases, for example we have seen that conductivity measurement itself is up to first decimal place, then it makes sense to report the means and standard deviations etcetera up to first decimal place. This does not mean that when you are doing the calculation you will always do only up to first decimal place, it is a good idea to keep the extra digits and round off only and the last step when you report the numbers, so but it is important to know the significant digits in any given scenario and report data only up to that. We have also learnt how to report errors, you can report it in absolute terms and in relative terms and we have learnt how to present data with error bars because in order to understand trends in any given data it is not sufficient just to look at the mean values but you should also look at the error bars in the data. And we have learnt how to classify errors and we have learnt that error propagates and we have learnt how to quantify this uncertainty propagation, either by using some analytical calculations or by using Monte Carlo simulation. So either way you can find out how the error propagates. So these are the things that we have covered. We have also learnt a few important things in doing this course in this module. First thing is you have to understand your data and the errors. So visualizing the data is a very good way of understanding the data and visualizing with error bars is a nice way to understand the errors in the data and it is always important to pay attention to the outliers, some amount of effort is needed to understand why they are there and it will help you improve the experiments or understand what is happening better. We saw one example where there was an outlier in the electrical conductivity measurement and little bit of analysis showed that the measurement methodology was not applicable for that scenario and that is why we got some meaningful and numbers which were not in tune with the rest of the numbers. So they were not consistent with the rest of the data. So we have also learnt that while analyzing trends it is important to incorporate the error information and we have understood the importance of propagation of errors and so if you measure some quantity and if it has some uncertainty any subsequent analysis that you do using that data also picks up the uncertainty from these quantities and we have also learnt that the materials data is broadly of two types, one which can be represented by summary reports like conductivity for example. So it follows nice normal distribution so it is sufficient to give the mean and standard deviation that completely describes the data. This is because every measurement gives one number and the fluctuation that you see is random, it is a noise so it can be very well described by the normal distribution. On the other hand some materials data can only be represented in rank based reports. You need to give things like histograms to describe these data and if you just use rank based reports like we found for example for phase 2 it looked as if there were so many outliers, it was as if you assume that the data is normal then obviously it is not normal because of which you find that compared to the mean up to 6 sigma you found data points and they were all only on one side. So it is obviously not a normal distribution. So it is better to describe such data using the appropriate distribution, it is not a good idea to assume that it is normal and assuming that it is normal or not can also have a say on understanding the further results. So we have found an example that if you are trying to calculate the uncertainty propagation using grain size data whether you are going to assume that it is normal and the errors are normal or it is not it is log normal for example is going to make a difference to your analysis. So it is important to know, it is also important to report these quantities sometimes in the literature these quantities are not reported then it becomes very difficult to understand or analyze the data. And finally wherever possible of course we should use both rank and summary based reports because that is the most complete information that one can give. The best is to actually give raw data and it is recommended to give it as supplementary data but in addition when you do any analysis we should give the analysis methodology and we should give as much of information as possible in terms of summary based and rank based reports so that a complete picture of data is presented. Thank you.


So, hi everyone. I'm Christopher McGowan. I'm the co-founder and chief scientist of Piston. With me, I've got Mandel Dejrenus, Jonathan LaCour, and Jordan Tardiff from Dreamhost. Mandel's with me at Piston. We started this talk as the idea we were going to have a panel on how we use Ceph and Cinder together. To that end, there's no slides. We've got a Google Doc open, and we're drinking beer. So, the rules are, if you guys have questions, you're free to interject at any time. It's going to be a moderated, or a moderator-less panel, but the rule is you have to have a beer in your hand when you're at the microphone.  And I'll let everybody go through and make their introduction, so we'll start with Jordan, because we promised him he wouldn't have to talk until he was called on, and I don't believe in that sort of thing, so. This is gonna work? Cool. My name's Jordan. I've worked at Dreamhost for 11 years, I think, now, and I'm one of the two cloud engineers on our team. That's right. I'm Jonathan LaCour. I'm the vice president of cloud. It's my fault that there are only two cloud engineers on our team, which is why one of them is not here, because he's tired, which is pretty much true all the time. And our product is called Dream Compute that's built on top of OpenStack, and it's a public cloud, and we have a multiple petabyte storage cluster running on SEF for block storage. And I'm Mandel Degrenis. I'm working with Piston Cloud, and I've been working on their storage side since it started, and pretty much having to do it myself until recently, we finally hired someone to work with me. Hi, I'm Christopher McGowan. As I said, I'm the chief scientist and co-founder of Piston. It's actually my fault that Mandel's the only person that's been working on SEF and Cinder and all of our storage for the past three years. So, yeah, if there's anything wrong with what's going on, blame us. So I think we were going to start by describing our kind of, you know, we have very different use cases for the, for SEF and Cinder, maybe describing a little bit about the architecture and kind of what the footprint looks like for each of us. Right. So you want to kick it off? You want me to kick it off? We could rock, paper, scissors, or we could just have Mandel do it. All right. All right. You go first. All right. So we use Cinder as both block and SEF, sorry. Yeah, that'll help. Yeah, we use SEF for our block storage and for our object storage. Closer. Does that work better? Oh, there we go.  Anyways, describing our architecture is a little difficult because all of our customers use it differently. However, I understand USDA is giving a talk this afternoon and talking about how they're using us. So that might help. Basically we've got a configuration manager that allows you to define how you want to use various different types of disks and put some into multiple pools in SEF. And one of the things that we do is since we have a converged infrastructure distribution of OpenStack, we actually run compute and storage all on the same hosts. Because of that, we have a very different use case than most people who are deploying Cinder. So we operate with monitors and OSDs on the same hosts, and those same hosts are also running virtual machines, which puts us into a kind of an interesting... We discover a bunch of edge cases that Ink Tank and SEF, the SEF community wouldn't otherwise have found because of assumptions they've made and assumptions we've made that don't always necessarily match up. Cool. Yeah, our use case is completely different. We are a public cloud use case. And in our case, we have a multi-pedabyte cluster with 126 Dell R515 storage nodes. Those have 12, three terabyte disks in them. So you can do the math. That's Bukku data, a bunch of it. And each one of those has 32 gigs of memory. In our block storage, we're using 10 gig networking everywhere. And that helps a lot. And we also have three monitors right now that are Dell R415s. And we're going to be adding two pretty much right after the summit because we're opening up our beta quite a bit. So in our use case, we don't have the converged infrastructure. Our hypervisors are separate. And those are our Dell C6145s with 192 gigs of memory and 64 cores. And we have 48 of those. So we're supporting thousands and thousands of VMs on a single storage cluster. And we use Cinder for the block storage. And we use Glance to store our images in Cep as well. Yeah, and we're doing the same thing for writing our images back to Cep. And we have some other extensions that we write into Cep as well. We use it as the backing for our memory page sharing service that we have in our piston open stack. Also, MySQL database sits on Cep. Oh, yes. MySQL sits on Cep. And that is hosted for the infrastructure services for Cinder, et cetera. That's interesting. What in particular got you running MySQL on Cep as well? We don't want the MySQL database running outside of the cloud. Basically, our installations turnkey. So we had to be able to orchestrate the services that the services depend on. And since we use master election to do that, we needed to be able to ensure that the service could run from any of the hosts in the cluster. And that gave us very few choices. I mean, we could have used Gluster, but we were already deploying Cep. So it made sense for us to orchestrate Cep and then at the same time deploy the services on top of them. So we have a question. He's got a beer. He went through the whole thing. So please. And introduce yourself as well. We haven't got this, so I could ask this question. I haven't been drinking it for a while. So you've got MySQL on Cep. Are you having MySQL right to a raw RBD volume? Or are you actually using Cep FFS to share that out? We are not using Cep FFS. When we started off, we were using Cep FFS. And when I mean starting off, I mean when we were doing the initial development. And it turns out that Cep FFS at the time in 2011 wasn't performance enough to be able to handle a single concurrent write or a single read. We discovered this when we put it on there and we're like, what the fuck's happening? The whole thing's going away. This doesn't make any sense. And then in 2011, this has changed. We haven't tested it since then. But if you created a directory tree four levels deep, so A, B, C, D, and then CD into D, an LS on an empty directory would take 30 seconds to return. Which is not ideal. Yeah. So since then, we've tended toward the Cep community's recommendations not to use Cep FFS. And so we really haven't evaluated it since then. So it's in ODB using an RBD volume as a raw volume? Yes. Thank you. And my understanding is Cep FFS has come a really, really long way since then. So it's definitely something that we're going to be looking to it a lot more very soon now. And also one of the things that we're going to discuss later on is how Cep has rapidly matured since they've started and since we started using them and since stream host has come aboard using them as well. So I think we should describe our sender setups. I think that might be a valuable thing. And I'm going to kick this one to Jordan because he is, I'm amazed he still has hair. Since he's been battling with, what was that? It's actually a wig. Oh, it's actually a wig? Yeah, I wouldn't be surprised. So our sender setup is something that we've been tinkering with a lot lately. And I can talk a little about the latest stuff that one of our guys, Ryan, has been working on. You can go through what we've been bodging together to make things work so far. Yeah, so we've been looking into a couple different ways of doing things. With sender running multiple volume services, it's not necessarily a supported way of doing things. So there's a hack that a couple of people use that we learned from CERN. And it's to change the host in your sender.conf to all of the same hosts.  But apparently that introduces some race conditions that we haven't actually hit yet. And I don't think many other people have hit. So we do run four sender volume services on separate hosts. And then we run also four, I think about four API services as well right now. And so yeah, that's where we're at right now. We're also researching into some way to actually split the rabbit messages out so we can separate deletes and creates off to separate sender volume nodes too. Yeah, and the reason that's important is I think the interesting thing about sender is sender was really designed for use cases where potentially the driver is going to have consistent performance and very rapid. Yeah, I'll finish my statement and then we'll call you. Really rapid responses to each operation where they're fairly consistent. When you're dealing with a large kind of multi petabyte cloudy storage cluster, it's a little bit of a different situation. And so not all operations are always going to be fast. And it's not that sender is dumb to do it this way. It's just it's a newer use case, it's a different use case. And so we're working with a lot of the sender devs and they're doing a really great job of helping us. And kind of the approach we're taking right now that Jordan was alluding to is we've implemented a RPC driver that allows us to send and shard the sender operations that are going to go to sender volume to different topics. And then we fixed a bug in sender volume where if you configure it to listen on a different topic it actually listens on that topic. And then we can shard like certain operations that we know are slower off to a separate pool of sender volume workers. So deletes for example are pretty slow in SF and RBD compared to, you know, if you're using something like LVM, right?  So that's kind of why we're doing that and what we're working on. We will upstream that once Ryan finishes. This guy right here, you can hand him. And I've also volunteered Mandel and Lyssa to work with Ryan to get that so it's able to be upstreamed. So Lyssa's in the front row. So yeah, grab her and go ahead. Yeah, so this is a question on the convergence of structure. When you're with the convergence of structure, did you end up separating the compute traffic versus the storage traffic? We do. We do it kind of interestingly. We have some in the configuration that we use that deploys the cloud. It's sort of a meta configuration. We specify a bunch of different VLAN networks primarily for things like dealing with management traffic because we have a distributed control plane and the storage traffic is on a separate network than the compute traffic which is coming from the virtual machines. And those are specified and in the neutron deployments those are actually on completely separate networks that are user defined. So we have a host network that handles the Paxos data for Ceph and at the same time for some of the other less chatty Paxos implementations that we use. And actually this is something that maybe Jordan could chime in on as well. We actually have multiple networks set up as well to ensure that Ceph has, so basically the tenant traffic is separated from the management traffic for Ceph. So when Ceph is peering and shuffling data around it's not saturating the networks that customers are using for their actual operations. So I mean, could you expand on that a little? Yeah, I mean there's not much more than that. We basically have two networks we're basically both all on 10 gig each and so we have a 10 gig Ceph front end network where all of the tenant kind of traffic is gonna happen and then we have a Ceph back end network that's all 10 gig as well that all the migrations and stuff like that happen on. And then we also have another management network that's just gig for actually talking to Cinder and stuff like that. You wanna interject Mandel? You haven't talked in a while, you've been drinking. Yeah, well drinking's more fun. Anyways, the architecture, right, Cinder? Just checking. We use single master elected scheduler and volume at the moment. API is spread out across the whole cloud and load balanced. We use multiple Ceph pools as I mentioned for different storage. Close, close, close. There we go. Sorry about that. So multiples have pools backed onto different types of disks for our multiple volume types. Everything else is an operation of the install, so.  Yes, we are. Yeah, and the benefit to that if you aren't familiar is that Ceph supports copy on write.  So you get some operational efficiencies there. And we've actually had really good luck with that. That's been something in terms of our war stories. It really hasn't been much of a war there. And that's allowed us to get boot times in like the 42nd to 72nd time range right now. It spikes up higher than that sometimes, which is why we're working on the sender sharding. That's really the big reason that it can spike up. Our goal is to get around 30 seconds, which we think would be pretty spectacular. And that's a lot of that is to do with Ceph and copy on write. So I would say that in the community as a whole that people at Piston and the people at Dreamhost are the ones that have deployed Ceph the longest and have probably the greatest amount of expertise with building the scalability and dealing with scaling out their clouds. Could you guys describe how you guys are doing the scalability and how you're doing that orchestration and then I'll have Mandel do the same? So you wanna maybe, we're using Ceph a lot to do all of our kind of management of our Ceph, configuration management of our Ceph cluster. We have had some interesting fat finger moments there. And we've had some fun upgrades and such. We'll get to that a little bit later when we get into the horror stories. But Jordan, you wanna expand on that a little bit? Yeah, I mean, we've been using the Ink Tank Ceph cookbooks I think for a while. We've had to make a few kind of customizations to spread out our OSDs across racks and different power zones because we've had a few power issues that have actually taken out the whole cluster. So we had to actually make sure that we can lose certain parts of racks by setting up different roles for each of our Cinder OSD clusters and make sure that we can lose those. Before I have Mandel answer, could you like describe some common scaling problems you guys have run into? I mean, we could talk about the one where the configuration issue is one of our major ones. Well, let's hold off that one until we get to our section on the Ceph Pocalypse. We can get there. So scaling for us basically means that we have to add new servers because we've got converged servers. So that's our Moxie scale out works great. We've got disk profiles which define what each disk looks like based on profile, we profile disks when they come in. So we look at them, see the size, see the speed, et cetera. And then we based on that, we make a decision on is this going to be used in model A, model B, model C, get divided up and it just gets added into the Ceph Pools. Our crush map has been, was fun. That was figuring out how to make sure that we ended up with data actually going to multiple... Yes, that's important, making sure data goes to the right places. Yeah, it's really easy when you set up a single crush map where you've got one root and everything goes. The default works great, but as soon as you've got two different classes of storage, then you have to build your crush map individually.  And so you have to be really careful and sure you do a lot of auditing. We've written some... One of our cloud engineers, Jeremy Hanmer, has written some Nagios plugins, which are really helpful for actually auditing your crush maps that ensure that you don't screw it up, basically, because we're humans and every human makes mistakes. We have errors, so we monitor ourselves, especially the VPs. Pretty sure all of those that we've created are actually up on our GitHub, too. Yeah, they're up on GitHub. And if you need to know, talk to me. Happy to share. So, are you guys using the current interface to the Chef Crush map? Or are you doing it by hand? Oh, the Calamari stuff? Yeah. Not... Okay, so the funny thing is, the guy who wrote Calamari, he's got it running on his laptop and looking at our Chef cluster, but we're not using it, so... But we plan on it, especially now that Calamari is gonna be open source. Thank you, Red Hat. That's fantastic. We're excited about that. Yes, thank you very much. I would clap, but I don't have a hand because I would drop it. That's the sound of one hand clapping. We actually, because there was no interface to the Chef crush map API when we first started, we actually have a bunch of really ugly, ugly orchestration around that, and we really hope that once Calamari is open sourced and available to the community, that it's something that we'll be able to get rid of all of our ugly manual... It's automated, but it's still manual, like file editing and munging out of that to use the supported API. Shall we talk the Chefpocalypse? Yes, let's. All right, so I'm going to start with the introduction to set the stage for the Chefpocalypse because we've both had Chefpocalypse I. Hours didn't end quite so happily as Dreamhosts did, and it started off even sadder. My grandfather in 2012, he passed away, and I had to go back to my home for the funeral, and when I landed, my great grandmother was in a nursing home, and she was also going, and she passed that night, so I'm in Maine, I'm attending two funerals in a week, and I get a call. Oh my God, you won't believe what happened. And I go, what do you mean? I won't believe what happened. And they go, well, you know how ButterFS says you can never lose data? You can. It turns out that ButterFS is the only world's first production-ready, write-only file system, and what happened was we got hit by a bug where the ButterFS file system for the Cep OSD got corrupted, and there was a horrific Cep bug where there was an off-by-one error for all the replicas, so replica one was written perfectly, replica two was here, and Cep thought it was here, and so the data disappeared. You want to go into more detail? Probably not. Not too much, but anyways, the painful part with ButterFS is there's no FSCK because you don't need it, it's a journal file system, that's what they tell us. It turns out that you can't automate that recovery when you lose power to a disk and it hard crashes. And the interesting thing about that is we never actually have access to the clusters because there are customers, and because we have a converged and automated installation mechanism, in general, the end user, the operator shouldn't be getting on the host at all because then you can't trust the automation, and we have customers in places like Cambodia where the guy on the ground doesn't know what's running on the hardware, and the guy who has access to the operating system and the cluster as a whole, as an admin, he's five, 10, 12,000 miles away, and he has no idea what's going on. This happened in our dev lab, so it was a little bit less terrifying, but getting that call when you're away was kind of terrifying. Subsequently, SEF has changed their recommendation from using ButterFS to using XFS, and the performance is slower, but it is safer. And it's actually gotten faster and faster. When we were doing intensive measurement on ButterFS versus XFS, what we also found was, in the beginning, ButterFS was so fast, it was amazing, and then over time, as more and more operations happened, it would get slower and slower and slower and slower until it was so mind-numbingly slow that it was not worth using. But the real issue is just ButterFS isn't quite there. It's an amazing thing. We are excited for what ButterFS can offer us. It's just, use XFS for now. Yeah, it's probably better for now. I would like to point out that even if the SEF bug off by one error didn't exist, we lost three drives on three different servers, and they were all unrecoverable, so we lost data. So in our case, we actually didn't lose data. So the SEF Pocalypse happened to us in our Dream Compute cluster, oh gosh, three, four months ago now. And so we had lots of instances running. We don't have any ephemeral storage at all. Everybody runs on the SEF cluster, and we were making a change to the SEF cluster. And when you do that, when you add in new OSDs or remove OSDs, you effectively changed the topology of your SEF cluster, and then it will attempt to rebalance the data to ensure that you've got a good even distribution, which is brilliant and wonderful. The problem was we had a bug where we had, and I'm reading this from my other engineer so I can make sure I get it right, we had our crush map had all the OSDs weighted to zero, which you shouldn't do, and all the machines and racks at one. And the result of this was that all of the data in our entire thousands of disks cluster, all of the data was stored in the first disk of each rack. So it was in like eight disks. And then when we changed the topology of the cluster and fixed the crush map, it attempted to then rebalance all of the data out of those eight disks all at once to hundreds and hundreds and hundreds of other disks. This made everything very angry. And the funny thing is we were actually able to make it work. We just had to slow things down and SEF offers you all sorts of tuning mechanisms to say, okay, don't freak out and move all this data like all right now. This is all coming out of one or two disks. Slow it down, relax. And the really amazing thing for me is I think it took like two weeks to rebalance the data. But once we had it all rebalanced, all of the instances that I had running with applications running just came back up again at the same point. Like my applications were all there, it was shockingly awesome. So yeah, this is why we created those Nagios plugins that I recommend that you go download from our GitHub. If your Crush map is managed properly and monitored properly, Calamari makes this a lot easier as well which is why we're excited about it. You know, you won't have these kinds of problems but we wanted to share what not to do.  I kind of feel like we're the old, old guys telling all these young kids that they're gonna be in our day, we had to walk up and down hills to get to school. My chef clustered red on butter FS. Darn kids, get off my lawn. We had other questions. We have 15 minutes left. So if you guys have questions, beer, go. So about XFS, are either company brave enough to run it with barriers turned off? Is that bad news? Was there no barrier mount option for XFS? I have no idea, do you know? I have no idea if we're doing that. That's a good question. We're not. We, because of that week, the week from hell, we have a instinctive reaction against losing customer data. That's been my gut reaction as well. I like to sleep. Awesome. We tend to choose the safe option for data safety. Do you mind passing off your beer to the guy behind you? He doesn't seem to have one. Thank you. It's empty. Good. Okay, my question is, our customer need a disaster recovery. So that means we need to have two installation of SAFT. So when you transfer the image between, or the volume between different set of installation, you are going to lost a couple of things. For example, the theme provisioning, right? And also probably you lose those benefit that for Kapiak write. So how you deal with that problem? Kapiak write actually works between different volume services, between different pools. So if you're using two pools in the same SAFT cluster, Kapiak write actually works between both of them. But if there is like disaster recovery, one data center, one another. If it's in a totally different SAFT cluster, then yeah, I think you lose Kapiak write. Well, you lose Kapiak write, you lose Kapiak write unless you have the image in both places. And also you lose the theme provisioning also. Right. So you can do it. And actually SAFT has grown some really, we aren't using this right now. So it's hard for me to speak to it too much, but you can talk to some of the ink tank guys. I'll tell you all about it. I know they have some capabilities now for those specific DR use cases where it'll do like replication and stuff. I can't speak much for them now. But that's only for the, oh, sorry, go ahead. They were talking with that at the SAFT community session on Tuesday morning. Okay. The remote replication stuff is currently only for the object store. So it'll work for like, sorry, Glance images, but it won't work so much for volumes that are backed on Cinder. Right. Currently, that's expected to be fixed. But we have, do you have another question? No, thank you. Okay. Does the guy behind you have a beer? Ah, that, all right. Welcome. Have you had any problems with the physical hardware failures, disks, servers, that kind of thing? Sorry, I didn't hear the question. Do we have problems with physical hardware failures, disks going out, et cetera? Oh yes. Discs are not reliable, turns out. They fail. The great thing is that SAFT deals with it very well. It's designed for failure. That's what it's designed to do. So yes, we have that problem. Discs fail, servers fail, racks fail.  Oh God, please. The last time that happened, it resulted in a catastrophic temporal cascade failure. And the entire cluster disappears into the same temporal rift that Tashiyar came out of in season three of Star Trek on the Enterprise C. Just don't do it. Really, that's interesting. It did come back eventually, to be fair. So did Tashiyar in season four and five. Yes. Is this thing on? Yeah, it's on. Apologize for not holding a beer in my hand. I still have too much beer on my system from the latest live party yesterday. I hope you accept that as an apology. We'll accept, yes.  I can't have any more beer for the next week. Anyway, wanted to follow up on the discussion of DR recovery in a remote data center and replication for different self clusters. What you currently have is only Rados Gateway, which means you can't even use that stuff for Glance. All you can use that is for Swift API. But in the next self-release in Giant, which is planned to come out this summer, there will be same kind of replication for RBD, which will mean Glance, Ascender, Nova ephemeral volumes, the whole thing. We're looking very much forward to that. That will be fantastic. So what he said is there's RBD replication coming for the Giant release, so. Are you working on Ceph personally? I work on Ceph support in Mirantis OpenStack. Are you, damn, I forgot what I was gonna ask. Are you working on getting the, or do you know off the top of your head if the Erasure Coding or the Erasure Coded RBDs will have the same sort of disaster recovery? I don't know personally, so this was curiosity on my part. I don't think this was brought up in the Ceph community session yesterday, but I don't see any reason why not.  Right. We actually run that on top of SSDs within the cluster. Yeah. And if anybody does wanna ask any questions about the object storage stuff, we know, we both know about that as well, so we're happy to talk about the Redis Gateway too. Thanks. Thank you. Are there any other questions, or should we start looking at our list again? We have 10 minutes left. The guys in the back are giving us evil looks to make sure that we stay on time. Apparently the last guy didn't, so. Two questions. Two questions. Him first. I do have it. My question is about BitRot and Silent Data Corruption. Since you guys had to switch away from ButterFast to XFS, I'm wondering, do we, do you care about it? Or, it's just one of those. Well, you know, we got copies all over the place, so not really. I mean. Well, but it's an inconsistency, that's the potential. So, I was at the Library of Congress last year in September talking to their data archivalists, and they very much care about this. The OpenStack community on the Swift side and the Ceph community don't actually care about this right now, and we should be. There are some middleware we should be building as a community. I built some of it for the Swift middleware to determine, or to continually upgrade, or check the hash of the data, but neither Swift nor Ceph do that currently. And it's something we really ought to be doing. Especially if you're gonna be running Cinder, if you're running all your block storage through it. Right, I agree. Is anyone brave enough to run ZFS Fuse? Not yet. Nope. If you look up here, we're cowards. Terrified, terrified cowards. We look out into the. Hashtag OpsLife, that's what we all are. Yeah. Do either party or anyone else in the room have any experience offloading journals for performance onto low latency media? We put journals on SSDs is a really good idea. Can you quantify it really good? I'll let Mandel talk about it. We've experienced the whole gamut of where journal placement is. Yeah, so we originally put journals on the disk. Turns out that's not optimal, especially on spinning disk. Even putting it on a separate partition on a spinning disk, it just gets really, really painful. Especially when it's on the same disk, it turns out, because you're seeking constantly. We put it on SSDs now. We recommend our customers do that. We don't force them to do that, but we highly recommend it and usually they can see the difference if they try it. Actually, with the SSD or journaling on SSDs, we've had some really interesting benchmarking cases where if you build a volume on our slow pool, which is running on rotational media like 700 or 7200 RPM, state of drives. The IO benchmarks are small enough that they fit in the journal, so you can't tell the difference between a fast pool volume and a slow pool volume because all of your data's actually, or all of your benchmarking's happening on the journal itself. For the rights, anyways. For the rights, anyways.   I can't remember offhand, but I think they're four gigabytes on the SSDs. I'm not 100% sure. And for our customers, that's configurable. They can change that if they want. We don't make any recommendations one way or the other. What about you guys? What's your journal size at? Or do you just consume an entire SSD? I have no idea. Yeah, I'm not sure. Where do you, I think Jeremy would probably have the answer to that. So my recommendation, if you wanted the answer to that question, is go to the DreamHose booth and camp out until Jeremy Hanmer comes out. He's going to look like he's- He's mostly dead right now, but I understand that he will be mostly living sometime this afternoon. Cool. Thank you. Do you have a beer or are you a beer? I actually have a breathalyzer hooked up to Soodoo, so I'm not allowed to drink and root at this time. Oh, fantastic. Cool. My question is, are you guys using, and I came in late, so my apologies if you covered this, but are you using RAID underneath as you're backing for SOF or- Absolutely not. Okay, so you just have individual disks managed by SOF itself? Right, SOF operates against OSDs, which really should map one OSD per disk. And doing RAID underneath, you actually lose all of the performance benefits that SOF or SWIFT, object storage in general is designed to maximize throughput of files through the system, and doing that by throwing multiple replicas onto individual drives is better than trying to stripe them across multiple ones. Okay. Yeah, we actually do go through a RAID controller, but we're not actually doing any RAID on it at all, so we run an OSD per drive. Are you taking advantage of the caching that the RAID controller provides? No, no, we don't use any of that. Okay.   Like, from the perspective of the end user who's actually deploying it, it's wasted money on the part of the vendors that are selling them. Yes, I agree completely. This is probably one of our biggest performance gripes is just around the hardware. Like, as the software gets smarter, I want the hardware to get dumber.  Give me as many disks as I can wedge into a rack as possible. That's what I want, right? But there's all this other stuff that they do, and it's like, for this particular use case, and the good thing is, I think Ink Tank has done a good job now of working with the hardware vendors to make sure that they know what kind of device makes good sense for this use case, and now that they're with Red Hat, I think that'll accelerate a lot, so. And we've had some success dealing with these guys. It turns out the value add that the name brand vendors think they're providing is... There's value cache, but it's actually negative value. You had a question? Yes, you're talking about some interesting thing, for example, put in the journal on the SSD, but isn't that frequent write going to kill the SSD pretty fast? Could you repeat the question, please? What I mean is that you put the journal on the SSD, right, to make it perform better, but isn't that frequent write going to kill the SSD pretty fast, right, because you have limited lifecycle on the write? It can be limited lifecycle, but I don't know about how Dreamhost sees this, but the volumes, Cinder volumes, have frequent writes because they're really network file systems, they're block devices that you're going to be writing to, so if you're doing anything on them, you're continually writing to those disks. And they're 64K blocks, 64 meg blocks? I don't know. Man, you should know. I should know, but they're, it's small. Yeah, so they're really small blocks. Can you go to the, yeah, do you want to answer? Yeah, you're going to burn through SSDs really quickly, is the question. My apologies. Okay, I think the answer to that question is it's worth it. That kind of depends on the size of your Cef cluster too, but as with any SSD usage, you need to monitor and make sure that you're replacing them before they're failing. Yeah, get good SSDs, meaning Intel, and, you know, yeah. We've had issues with cheap SSDs. They fail way earlier than you would expect. Friends don't let friends use anything but Intel SSDs. Never mind. Have I had too much beer? Maybe, so we have three minutes left. Any other questions? This worked really, really well. Thank you guys for, for drinking with us. Yeah, I appreciate it. Okay, go ahead. I believe Cef, the default number of copies of each object is two. Do you guys run with a higher number? Three is the magic number. Monitors, we run, oh yes, we run with three copies. I think that actually is the default three, but I could be wrong. It didn't used to be. Okay, recently, okay. But we've always, you use three on our object storage and our block storage, always have. And if, monitors five, five is the magic number. That shall not count to four, that shall not count to six. Unless you're using three. Unless you're using three. Or seven. But don't use three. Yeah. Go to five as quickly as you can. And we go to seven at 120-ish nodes, but that's just a thing. You had a question? Do you have a beer? Are you a beer? I am a beer. Okay. How many OSDs do you run per journal? We have one journal for one OSD. Yeah. Anyone have an abacus? So I heard the ratio is normally five OSDs to one SSD. Oh, we, I mean, like I say, our customers have the ability to do more, but we strongly recommend no more than four. No more than four. Okay. Great, thanks. Upgrades. That's, so. Upgrade ability. One of the things that we were going to cover was upgrade ability. We didn't ask about it. So yeah. Well, we have one minute. So I'll discuss very briefly. We know a lot about upgrades because we were the first production deployment of SEF at scale. We have a multi-petabyte object storage service called DreamObjects. And it's essentially just a deployment of SEF in the Rados Gateway. And we were live in production that that in pre-Argonaut releases of SEF. And so we've upgraded through basically everything ever. And all of the QA that has happened on that upgrade process has been us. So you're welcome, everyone in the room. And the answer is we're getting better and better at that. And each upgrade is getting better and better. And Ink Tank is doing a better job on each release. The good news is on when you're starting from a newer release, the upgrades are so much smoother now. So the upgrades that we've done on Dream Compute's block storage cluster have been easier than the ones we've done on DreamObjects which is coming from the dawn of time, if that makes sense. So it's just, there's still vestiges. Every single time we upgrade, we see something new and different that we're the only people ever to experience. So you guys won't have any of these problems as long as you don't install that extremely old version of SEF. Start new and it upgrades are easy. Yeah. Ur. Our only issue with upgrades is that we've got a converged architecture and so the recommended upgrade path doesn't work really well for us. Ah. The OSDs and the monitors in the SEF upgrade path, you can upgrade live as long as they are as far away from each other as possible.  And I think my biggest recommendation to close out, like the one thing, if there was one piece of advice I had for you if you're gonna run SEF is follow Ink Tank's advice. There's a reason that they come up with the defaults and they aren't always right, but for the most part they are now, right? So they make mistakes like anyone does, but the recommendations are really good starting point. Your use case may vary a little bit and you can tweak from there, but it's the best place to start is the recommendations from Ink Tank. They fixed the eight pools per, or eight PG's per pool, right? They did, I think so. All right, we're out of time. If there's any other questions we can come up and talk to us. The beer rule still exists until we get out of the room, but thank you guys, thank you very much. Thanks.


The state government has continued with the replacement and extension of pipelines in areas without water across the metropolis and its outskirts. Governor Peter Rimbaud was on inspection tour of some ongoing water supply extension projects in some parts of the capital city yesterday charged residents to take ownership of the water facilities and protect them against sabotage and vandalism, while in those deliberately vandalising state assets, leading to frequent leakage of water and a huge cost to the government to desist from the illegal act of face severe consequences. People have been going around doing all sorts of things, including cutting pipelines, vandalising water collection points, just stealing the help and in some instances breaking the pipes completely. The two things that we have thought about doing as a government one is to provide adequate security in some of these locations. But more important is for communities to take ownership of the infrastructure being brought to their communities. These are people that have not had water over a decade. So we brought water to them, put those points as first line intervention, depending when the reticulation gets into their homes. So we also expect in working with the chairman, the councillors, some of the community leaders, the PG leaders to be able to now say to them, government is doing what they are promised now and to ensure the sustainability of this infrastructure. You have to be committed in that process. And that's the conversation we had with them earlier and there was commitment. But you do know in this sense they are going to be outliers, they're going to be instances where people are going to try to push an agenda that's contrary. So that's what you're seeing. But when you look at the quantum of vandalism going on, it's not the way it would have been if it hadn't been for the interventions we have put in place at the start of this strategy. So this is something that we're working very actively. His Excellency is very, very intentional about the whole process of providing security, but it has to be a collective effort. It can be an effort of something that is being done out of the lion building. Everybody has to be involved in safeguarding this and changing the narrative around Inugu.