All right, welcome to the final session of the seftey track here at open source days. Go ahead and take your seat when you get started. Just a reminder, if you do have questions, either in the middle or at the end, to make sure you use the microphones so that the recording that we have going for posterity will pick you up as well as the rest of us in the room. Our next and final speaker is the creator and current project lead for SAF, Sagewile. He's going to talk about some of the more recent work that's been going into the upcoming release of Luminous and a few other things that we have going on. So, Sage? Thanks. Can everyone hear me okay? Yes. All right. My name is Sagewile. I'm the SAF project lead. I work at Red Hat in the Office of Technology and Overseas SAF development and so on. So, I'm going to talk a bit about the release that's about to come out, Luminous, what's in it, what's cool, and then I'm going to talk about what we're working on after that and then a bit about contributors, stats, and so on. So, just a level set. SAF does a regular release cadence. We do, normally we do releases every six months. Every other one is an LTS, which means we do backports for bug fixes and so on. Luminous is about to come out. It's supposed to be spring. It's going to sort of be another month or two before it's out. So, it'll be, I guess, early summer. We're a little bit behind. The next release after that is going to be Mimic, which will be a non-LTS in the fall or maybe winter if we continue our sort of laggard pace and then the one after that will be the end release, probably Nautilus, but the name isn't really finalized yet. What's the deal with the names? They're all names of cephalopods and they go by letters increasing. So, Luminous is Luminous Squid, which is beautiful. If you go, look at Google images and wonderful, so on. So, lots of good stuff coming in Luminous. It's going to be a really good release. I'm very excited about it. The biggest one, the biggest piece that I'm most excited about because I worked on it primarily was Bluestore. Bluestore is going to be stable in Luminous and it's going to be the default back end for the OSDs. So, that's a big milestone. Bluestore consumes a raw block device in contrast to our sort of legacy file store, which consumes XFS. We use ROX to be internally for metadata, but it's all sort of packaged up one big thing that we control. It's very fast on hard disks, roughly twice as fast, both for large IOs and small IOs. For regular SSDs, it's also faster than file store, maybe more like one and a half times, but it varies on your workload. For NVMe, it isn't that different than file store because the NVMe isn't actually the slow part. We have other issues to deal with the optimizing stuff itself, so it uses less CPU. But Bluestore is the future and that's sort of where we're trying to get to. More importantly, it sort of gets rid of all this legacy stuff that we had with file store.  So, Bluestore has full data checksums on everything, so every time you read any data from the disk, it gets checksum verified, so you won't get sort of bad errors. It also does inline compression with Zlub or Snappy, which is nice and it's going to be a stable thing. Lots of people contributed to it as a group effort and we're very excited to finally have it done and out there. I'm going to show a few quick performance plots. These are showing large and small write IOs, random writes, on trying throughput and latency. The whole bunch of development branches, sort of the top one is the one that all got merged. These are a little bit old, but it's roughly twice as fast for large and small IOs, sort of the takeaway. Same thing, similar picture when you mix reads and writes. The reads aren't necessarily twice as fast, so it's a bit of a blend there. But more importantly, if you look at sort of the aggregate workload, not just sort of a micro benchmark, things like Rados Gateway that are doing index updates on the bucket indices that are stored in Rados. There were all these weird annoying things that Filestore had to do in order to make that work properly and be consistent and safe. BlueStore does it much better and so the performance improvement is more like 3x or 4x, depending on what your workload is because it just isn't as done as it used to be. So we're very excited about that. As a consequence of all that, this also enabled us to do another big feature which is erasure code support for Rados block device, finally. So the key missing piece before was erasure coded pools didn't support overwrites of existing data, now they do, so you can put a block device on top of those objects. It requires BlueStore in order to perform because we have to do a two-phase commit to be able to roll back and that it's implemented on Filestore but it's horrifically slow. And the other thing is that we rely on the check sums in BlueStore in order to do the deep scrubbing and so if you're using Filestore with the easy overwrites you can't deep scrub, it doesn't actually verify anything, let's just go read the data. So BlueStore and easy overwrites sort of go together in luminous so it's there, it's good. It's a significant improvement in efficiency over 3x replication. Even if you use a very narrow code like a 2 plus 2 or a 4 plus 2, it's like a factor of 2x or 50 cost improvement, much less storage that you have to buy. It's not perfect though, within erasure code small writes are slower because you have to update more devices when you update a full stripe with the erasure code. Hopefully we hope to mitigate that with BlueStore although we haven't done sort of the final testing that pits Filestore 3x against BlueStore erasure code so that's sort of in progress so we'll sort of know with the final how we how we net out. But on the flip side large writes are actually faster than replication because you're actually doing less IO to your devices which is also good. The implementation is still doing sort of the simple thing. When you do a small write it's updating a full stripe so we want to do things that are more clever but it's going to take a little bit more time before we're able to make those optimizations. But it works in Luminus. It's there, it's ready to go. The other big piece of work that went into Luminus was the new stuff manager daemon. It actually appeared in Kraken but didn't actually do anything useful yet. In Luminus it does lots of useful things. The main thing is that it offloads a whole bunch of work that the monitor used to do into a new daemon. So the old monitor previously dealt with all the PG stats. It ended up with a whole bunch of data that was just turning through Paxos and slowing down the monitor and limiting our overall scalability. That's all gone. Manager is doing it instead. It doesn't have any durable state and so it's much faster and more efficient. And monitor can focus just on the things that are important for keeping your cluster up and consistent and storing data. So it's going to make Sethmon scale again. And coincidentally this morning I just got word that CERN has like a 10,000 OSD cluster that we're going to be able to use for two weeks in a couple of weeks to do another Seth scale test. And so it's perfectly time to test all this new Luminus stuff to actually see how it does with these big because we don't usually get to buy that much hardware at Red Hat, unfortunately. So that's going to happen in the next couple of weeks. Very excited about that. Sethmanager also has a new REST API. We sort of took the Calamar API, adapted it. It uses the pecan framework now. It's written in Python. Manager has this nice Python plug-in framework that you can use. So that's going to be there. And there's also going to be a built-in dashboard. It's super simple. It's basically like a Seth-S on the web. But it works. Here's a screenshot. It's pretty simplistic right now. This is a one OSD cluster on my laptop, so it doesn't really show much industrying. But it shows all your demons, all your health warnings, whether things health is good or bad. It shows the log. Just really basic stuff like that. So not much in Luminus yet, but this is going to be a building block moving forward. Eventually we're going to add all the metrics in there that the manager already has. Actually, it just doesn't present them through the GUI. And that sort of thing. So moving forward there. There's a new network messenger implementation in Luminus. It actually was new in Kraken, but it's Luminus or jewel. It's new since jewel. New implementation, it doesn't use up lots of threads. It's much more efficient. It's event-driven. It's a fresh code base. It's great. It's so much better. It also has a pluggable back-in. So there's an RDMA back-end for the messenger that is built by default. It isn't tested very heavily, but I don't have any RDMA gear in our community lab. But it's being used in a few places in production with good results. So it seems stable, but it's not sort of officially supported and tested yet. So your mileage may vary. There's also an experimental DBTK back-end that uses Intel's user space acceleration library stuff. That also looks very promising. It's definitely in a prototype stage. It's not ready to go, but the code's there, and you can build it and play with it if you want. So very excited there. Mellanox has been an ex-guide. They've been the main people working on the RDMA stuff. And the other sort of nice thing that's coming in jewel is that we're finally going to have perfectly balanced OSTs. So everybody who operates a large cluster is dealing with the variation between the least utilized OST and the most utilized OST, and dealing with three weights and capacity planning and just a headache.  So the two tools are something called shoezarchs for crush, which basically is sort of a way to feed in all the specific parameters for a particular pool that tweak the weights in the ID so you can sort of get it to do exactly what you want. It's sort of a generic capability, but what it allows us to do is run a numeric optimization that just does a gradient descent and fiddles with all the weights so you actually get the actual output is exactly what the weights are that you intended when you put it in. So it installs that in balance problem, but also addresses something that we've been calling the multi-pick anomaly, although it's a very imprecise mathematical term. But it basically is an issue with the way that the underlying mathematics of probability that crush is based on where if you have a device that has a very low weight and a bunch of things that have larger weights, for example, if you have a bunch of racks and you start a new rack with one server in it, crush tends to put too much data on those devices and overflow them and this it's a it's annoying math but we didn't even notice it for a long time, but we can we can correct that as well by using adjusted probabilities for the second and third replica choices in crush. So the good news is that the imbalance portion of that optimizing with that is actually going to be backwards compatible with older clients. If you want to correct for the multi-pick part then you have to wait until all your clients are running luminous and understand the new stuff. The other tool is something called pgupmap which is just like it's just the ability to put an explicit exception mapping in the osd map that says this pg is stored on these osd's period so it just overrides whatever crush says and says put it here and so there's a really simple optimizer that looks at your distribution and says this osd has one pg too many and this one has one too few so i'm just going to move it there and does that. So both those tools are there but that pgupmap also requires luminous clients so you won't really be able to use it in a production cluster until everybody's upgraded on the client side. And a few other odds and ends for the rate of side. Crush has something new called device classes where you can just tag the osd's in your system as being a particular class or type so you can say these are ssds these are hard disks these are NVMEs and then you can write a crush rule that's just really simple that says map to osd's that are ssds and map to hard disks. Previously if you want to do this you had to create you had to manually edit your crush map and create two parallel hierarchies and fuzz with all the names and all the automatic crush manipulation stuff just kind of broke super tedious now it it works out of the box it's really simple so that's nice. There's a streamlined disk replacement process that's well documented so ssds you can replace osd's for using the same id's and it's going to be simple and it's actually going to work so that'll be nice. There's a new configure option that lets you as an administrator just declare what the oldest client version is that you want to support and then the monitor will sort of gate all of the other things that manipulate crush tendibles and so forth so you don't screw up so you can just say I want to be able to be compatible with hammer clients and you tell the cluster that and it'll just prevent you from doing anything that would break break that constraint just to make operators lives a little bit easier. We're annotating and documenting all the config options in the code so you can just do a dump and see all the config options what they mean and what whether you should touch them or not they'll be marked as like experimental developer only do not touch versus something that you should adjust expert only that sort of thing. There's a mechanism so that if a pg or object is stuck then there's a back off mechanism so the clients will stop sending requests which in certain recovery situations can bite people in the whatever so that they can't actually talk to the resty so there's some things like that that are fixed for having better eio handling peering and recovery speed ups new and cracking actually but nuisance jewel stuff is now in most cases if an OSD fails it immediately notices you don't have to wait for a heartbeat timeout so it's much faster failure detection and and cluster moves on so lots of good stuff that's there's an ongoing list of just sort of random little robustness stuff that's improving so good things there I'm sort of moving out of rados into the rados gateway we sort of have this high level view that in the future most data is going to be story and object stores so while block is obviously very important particularly for cloud workloads and hosting bms that's not actually where most of the data is going to be most of it's going to end up in objects like s3 and swift apis and so there's a whole raft of features that we're looking at here things like erasure coding tearing multi site federation and so on so new and luminous sort of the biggest new thing that's most exciting is rados gateway metadata search so we already have this mechanism I might build kind of screwed up here on this mechanism where you can take seph clusters and multiple data centers or the same data center but you can have multiple zones sort of causally independent rgw installations that are federated with each other so they share a bucket namespace and user and you can put a bucket in a particular zone you can replicate across zones you can do all the stuff with the new federation active active all kinds of stuff but the mechanism that allows that that does actually the syncing is pretty pluggable and so we have a plug-in that does syncing of just metadata and it dumps it all into elastic search and then there's a new set of apis in the rados gateway that are search apis that go query elastic search so if you set up one of these zones to index your object gateway content you can have an index either the default stuff or whatever headers you care about then you can go do search queries to find out what you're storing you know what file types what headers they're setting whatever you want to do so that's exciting and totally new a bunch of other stuff with the rados gateway there's a new nfs gateway it's actually been present in some versions of of dual it got back ported to some of the downstream stuff I can't remember if actually if it was upstream before but this is for the rados gateway there's a very simple nfs gateway that lets you mount nfs v4 v3 to copy in data or copy out data which is great for migrating existing workloads from sort of file based storage systems to object as you make that transition it's not meant to be a full POSIX file system it doesn't do small writes and renames and truncates and all that random crap but for just copying data in and out it works great so that's big for a lot of users so the biggest management operations headache that we are resolving is dynamic bucket index starting so for rgw users the bucket indexes if you put too many objects in a bucket the index would get big and there was a tool that you could do offline that would restart it or if you created a bucket you could declare decide what the sharding was at front but it was kind of a headache and you had to plan ahead and it wasn't very friendly finally a luminous that's just going to be automatic so as the bucket gets big it will reshard on its own and you won't have to do anything it'll happen online it'll just not be something you have to worry about just sort of if you might there's a bit of a theme of not having to worry about annoying things that we're trying to chip away these things so so that's good there are also a couple other sort of headline features that came into the gateway through a team of Marantis did a bunch of great work there's inline compression so RGW will compress the data as it comes into the cluster and write that compressed data to RADOS so that's good and sort of happens transparently there's also a bunch of encryption APIs that were implemented these are following the S3 encryption APIs I can't remember what the name of that the whole category is but you can set keys on the buckets on the users if the whole bunch of it's a big complicated API that Amazon made up I'm basically implemented it so it's there yep and then there's just a whole bunch of stuff with the S3 and Swift APIs that's been improved and added and updated and there's always a constant flow of issues there that get resolved those are sort of the big exciting things on the RADOS gateway side on the RADOS block device also lots of stuff going on the biggest thing obviously is erasure coding which I already mentioned but I'm going to mention it again because it's a big deal you can run RADOS block devices on an erasure coded pool and buy less hard disks and SSDs and everything else so it's pretty simple you just specify the data pool when you're creating the RADOS block device and it puts just the data blocks in that pool there's also a lot of work went into the RADOS mirroring mechanism so the RADOS mirroring demons are now there are multiple ones of them and they're sharing the load and they're HA and all that stuff whereas in the dual it existed but it was just one demon now it's a bunch of them and they scale out and all that stuff so lots of stuff there mostly just around robustness and not so much around new feature capabilities improved sender integration always is ongoing stuff in open stack a lot of work going into iSCSI this has sort of been a multi-year journey of various false starts and attempts to use different kernel interfaces that get tanked by upstream kernel whatever but the final the new the latest iSCSI approach is based on LAO's TCMU runner which is basically user space pass-through so the iSCSI kernel target in the kernel is going to pass through to user space two of RBD which is nice because you get the full LRBD feature set the latest and the greatest there and the performance penalty of doing that pass-through is very modest actually so that's good it's going to be a full HA solution that does fail over and SCSI reservations and all that stuff so that's coming and on the kernel side there have been lots of RBD improvements keeping up with the crush and OST and cluster protocol changes that have happened that's all there RBD specifically the exclusive locking stuff is in the kernel now upstream and support for the object map stuff which are both kind of old features but they're now in the kernel so if you're using the native kernel block device you can get that stuff and finally CephFS so if you've seen in my talks last few years or last year I guess you've seen this before but we used to talk about CephFS and Ceph saying that all these parts of Ceph were awesome but CephFS was nearly awesome because it wasn't ready yet and yada yada and finally now CephFS is production ready it's stable it is now fully awesome yay so I said I said exactly the same thing at the open-sac talk six months ago except it's at 2016 instead of 2017 the new the latest fully awesome part that is now fully awesome is that Luminous will have support for multiple MDSs active-active which also has been a long time coming and just finally there so yada yada yeah I think we lost it oh hopefully not long ago all right let me see if I can catch up I blame the door yeah there we go okay there we go yes so multiple active MDSs finally and there's a bunch of stuff to go along with that so the multiple MDSs have this load balancing framework that's all heuristic based and it's tries to understand your workload to move things around but it's hard to understand what client workloads are doing so there's also this manual mechanism is you can just go in and say this subtree this directory I'm just going to pin it to that MDS so if you want to you can just manually just enforce whatever the subtree partition is that you want if you don't want to rely on the automatic thing to do its thing which it might do right it might do wrong it's it's sort of ongoing work so there's that directory fragmentation is finally on by default this is stuff of us dealing with very large directories it'll break them up into little pieces and put them in separate objects and multiple MDSs and all that stuff it was off by default for a long time just because we didn't have the test coverage it's finally on so that's good there's been so many tests written and so many bugs fixed it's just that the stuff of us team has just been kicking their butts really getting confidence in the stability and so forth so that's all there and a lot of work also going in on the on the kernel client side also keeping the kernel client up to date with all the changes in user space and improve fixing bugs and so on so group effort here mostly from Red Hat and Souser developers but it's been it's been good so we're really excited about CFFS and if you saw the user survey also you might have seen that the Manila CFFS driver is like hugely popular number one yeah which that's pretty cool so yay yay open source so that's that's mostly it for all the stuff that's coming in luminous we're sort of wrapping up the development cycle as we sort of there are a couple of last features that were we're finalizing and getting merged into the tree and in the meantime we're also focusing on what sort of the low hanging fruit for just usability and making things easier to manage and deal with and less confusing that we can squeeze into the release because that's sort of you might have noticed there's a bit of a theme of trying to make CF less hard and so we're trying to get as much of that into luminous as we can but then after luminous there's more stuff so what's what's coming next the next release is going to be Mimic it's named after the Mimic Octopus I strongly encourage all of you to Google Mimic Octopus on YouTube but look at them they're like super super super amazing it's definitely the coolest cephalopod I've googled they're awesome anyway so Mimic lots of stuff again the sort of the sort of the highest the main motivation I say the main priorities are like you know makes us faster performance but mostly as far as features feel like we're in pretty good position the main challenge I think that a lot of OpenStack users are facing or not OpenStack CF users are facing is around usability it's just hard to manage it's complicated it's hard to set up and so we're just trying to make it easier unless confusing and a lot of it yeah there's a lot of stuff that we can do that just is sort of needlessly obtuse and opaque so we're trying to trying to improve that but there's also performance is a big thing too so on the rate of side the biggest thing that's going to happen post luminous is we have sort of a big refactor cleanup optimization exercise plan so there's some peering stuff to want to fix up but the main thing is that the main IO path where messages come in and they get processed with the thread pool and then they get handed off to the objects whatever it needs to be refactored cleaned up restructured to use more asynchronous state driven model for programming for probably use futures and all these fancy language futures and whatever but it's going to be painful but it's really important because the current structure of the code is hard to maintain because it's gotten so complicated and it doesn't perform as well as it needs to so as our storage devices get faster and faster we really need to address this sort of elephant in the room in order to to make progress so that's going to happen ongoing work in blue fs and rocks v2 there are some sort of tactical items that we're dealing with there but really we're limited by that OST piece we've done a lot of optimization on the messenger side you saw the talks earlier with rdma and so on that's getting much much faster getting stuff into and out of stuff and blue store is much much better and getting stuff on disk and off disk we sort of eliminated the main issues there but it's really everything in between that needs to be fixed up so so that's the big thing that's going to keep at least some piece of our team pretty busy but there's some other exciting rate of stuff that's coming too so one of the efforts that's been going on for quite a while now but sort of has been sort of a secondary priority has been working on quality of service this ongoing background development around the dmclock algorithm that was published several years ago in an academic conference is distributed quality of service that gives you two things you can give minimum reservations to clients or classes of clients and you can do priority based waiting for everything above that so you can guarantee so many IOs to certain clients and then whatever's leftover will be proportionally shared with waits among the other clients and the idea is to have a range of policy so we can use this just to prioritize types of traffic like client IO versus recovery IO we can prioritize pools so that certain pools will get more be faster than other pools on the same OST or just do it based on client so that this client gets a minimum reservation and this one gets whatever's left over but the problem is it's just a complicated problem especially when you talk about distributed systems and things that replicate so you're actually signing up to do IO and other people's nodes it's complicated but despite that our initial testing is actually shown pretty good results so we're encouraged that despite not having sort of a complete solution necessarily it's actually actually seems to be working pretty well the main thing missing right now is really having any kind of management framework so we have a lot of the underlying queuing stuff done but we don't know how it's going to be configured and how what the user experience is going to look at that's sort of all for TBD but the initial results are promising so this is this is an example of a test run a month or two got back where you have a couple clients that have minimum IOPS reservations of 100 or no 50 and 100 IOPS and then the third one had a very high priority so everything that was left over beyond that ended up given to the third client and not the first two and you can see that it actually you know does what it's supposed to do so so that's exciting we're sort of getting pieces that merged and it's sort of coming together but it'll probably be a couple releases before it's actually a complete usable thing the other thing that's going on is there's more work in the tiering department so once upon a time we did this thing called cash tiering and it worked okay but not great and so we sort of stopped talking about it and doing much with it this the new tiering stuff is coming and it's it's based on some pretty simple primitives so the basic idea is the a concept of a redirect so you can have a Rados object that's basically a simling to another lob object but from the client's perspective you don't know you just talk to the OSD and it proxies it through to wherever it is so instead of you'd be able to move from a cash tiering type model where you have sort of a sparse set of objects that may or may not be there in the cash tier and if you miss you go through the base tier to the new model where you go straight to the base tier which is essentially an index and it knows what all the objects are and either they're either there or there's a pointer to where they are and then they can be either in you know one slow pool or different slow pool it enables us to do other things so deduplication is a project that the the folks at SK have been working on for a while and we've been helping out a little bit with and it's sort of built on this basic concept of a redirect so the idea is to generalize a pointer to somewhere else with a manifest that says you know this part of the object is over in that piece and this part of the object is over in that piece so you can have fragments that are stored in other pools so we break objects into chunks we can store those chunks in content addressable pools where you you hash the content and so you're dedubing based on the content and you reference count those chunks and then you can have these manifest that point to a bunch of different chunks so it's how the basic idea how all the you know dedubing storage systems work with you know chunking things and so on but we're it's scale out in the sense that that's sort of that base rate us to here is acting as the index that says you know if this is the name of the object you look up the name in that pool and it tells you what the chunks are and where they're stored so that's that's that's the basic idea and that's the direction we're going in and there's a lot that's sort of still to be determined is this going to be in line chunking and storing or is it going to be post processing is that going to happen inside the osd or by an external agent is it going to be what are the policies that are going to control when you chunk things when you don't these are all sort of tbd and we're just sort of getting the core underlying functionality in place lots more work going into the stuff manager it's sort of been built as a new place to do new stuff in stuff and so there's a lot of things that we want to make it do one of the first things is metrics aggregation it's already slurping up all the performance counters and all the demons into the manager but they're not really going anywhere yet so in the short term we want to have sort of the manager provide time series data just in memory out of the box the short history the last few minutes or whatever so you can get a little Iops graph without any additional work but eventually we'll want to have that stream off to external platforms so if you have like Prometheus or Zavix or whatever your big thing is you can also just turn on the fire hose and send it all there so that's that's coming it might even it's even possible maybe that we'll get the Prometheus stuff in for the witness but maybe I don't know we'll see depends on how fast they are but there's also the intent of manager is that it's a good host for other management functions so it has this whole Python runtime environment so you can just add all these Python plugins so you can add new stuff to it pretty easily so it's going to be where we do the automatic crush optimization where we're automatically balancing your crush your crush weights and stuff it'll happen out of the box but we can also do other things like automatically identify which devices in your cluster are slow and migrate workload away from them steer IO away from those devices or even do device failure prediction so that if we're pulling smart data off the disks we can run a prediction model that decides which ones are about to fail and preemptively copy data off of them so lots to do there and it's kind of exciting because it's that there's because you can write things in Python that are more policy-based than it brings in a whole new pool of contributors that can that can write that kind of code so that's good there's also stuff on the architecture front so there are arm 64 builds I've mentioned this before but we're still trying to get enough hardware in the lab where we can actually do these on a regular basis and get them into the CICD pipeline so we have some of the hardware we're still waiting on a few more boxes but the intent is that going forward all of the new releases are going to have arm 64 packages for both CentOS and for Ubuntu there are also a bunch of patches coming in recently on PowerPC adding support for that as well and we're talking about getting PowerPC hardware in the community lab to do those builds so yeah power a while back we did some work with arm 32 builds because we built this 500 node cluster for petabytes out of these little micro servers it's essentially a hard disk with an arm server on the hard disk speaking ethernet so you're running those these on the hard disk literally no boxes hosting them that was pretty fun that was with WD labs and they're actually doing an update to their platform they're doing a that was a gen 2 drive they're doing a gen 3 drive that has a 64 bit arm yeah and more RAM and better networking and all kinds of stuff so we're working with them it's exciting if you're interested in that project or these things seem interesting to you you should contact Jim Walsher at WDC they're looking for POC people to work with so exciting good stuff and then finally client caching sort of across the board so there's on the Rados Gateway side there's a project with Boston University and Intel I think worked on it where they added a persistent cash for Rados Gateway to support their big data workloads over RGW and they it worked great they're putting stuff on NVMe they're like saturating the NVMe they're getting really good performance I mean it didn't sacrifice consistency because of the way that it was architected with they're doing immutable optics only so that was great and the students couple of the students who worked on that are now interns for the summer at Red Hat so we're planning to get all that code cleaned up and merged into the tree so that's exciting and on the RBD front we're also very interested in doing client side caching if you saw the talk earlier with Jason and Tishar we're looking both at immutable caching so that you have if you have snapshots that are the basis for clones then that that sort of immutable parent can be cached and then also write back cash so you get sort of a little latency rights that then get streamed back to the cluster CFFS actually already has a persistent client side cache if you're using the kernel client it's been there for a while there's a generic kernel infrastructure called FSCash that plugs into CFFS so if you're on the CFFS side you already have client caches at least to read all my cache it's not doesn't do written data but yes so client caches are good and that's that's sort of it for my whirlwind tour of all the new development stuff I'm going to talk a little bit about all the people who are helping us do it so these graphs are a little bit old I didn't get updated once unfortunately but lots of people are contributing to CFFS the number of contributors is increasing or it's great we love it it's it's a challenge for us to keep up with all the pull requests and reviews so apologies if you've submitted a pull request and you feel ignored just keep pinging us and we're busy but we we want you to keep doing it and the community is broad and expanding so these are the most these are the top contributors from it's updated a bit since my last talk but you'll see that there's sort of a broad set of people here so you can see all of the open stack vendors on this list with you know the Linux people and the easy stack United Stack you see a whole bunch of cloud operators a bunch of them and an APAC and also in Europe public cloud private clouds all across the board not all of them are using open stack although I think most of them are you also see hardware and solution vendors that are selling software products based on self sef or in some cases hardware products based on sef which is very exciting people like quantum twice in fact that you don't usually see on these lists and a couple of the people that actually I don't really know what they do I guess I could have googled it but but it's exciting it's exciting to see the the breadth I guess of a contribution so there are lots of ways to get involved there's mailing list we do a set developer monthly so every month we have a developer video call we alternate APAC friendly and EMEA friendly times and we just talk about whatever development issues are pending it's all virtual IRC whatever so you can join and then if you want more events like this of course there are sef days this one's awfully convenient because it's at open sac summit but we do them all across the world about once a month and you can go see the schedule there as the next ones are all in in Asia the next few months there are meetups that you can search for in various locales we also do sef tech talks I think this is like the last Thursday of every month or something there's always an on it's like a YouTube blue jeans thing where somebody just does some technical presentation of some subject on some subject related to sef those tend to be to pretty interesting all of this stuff is recorded and ends up in a sef YouTube channel so if you go there there are like hundreds of talks recorded talks over the past many years that you can go watch and learn all kinds of good stuff and of course you can follow us on Twitter yay and that's it and don't forget to google the mimic octopus because it is awesome and I'm happy to take any questions on on luminous what's coming after on old stuff cephalopods yeah to use the microphones though because they're trying to record hi yes I have one quick question about you know we're really happy to see the work on QOS coming does that also cover like per client metrics in the manager yes that is the intention so the way that DM clock works the M clock piece is the the actual Q prioritize Q that does waiting that does both minimum reservations and waiting and the D part is the distributed part so there's metadata that's shared between IOs by the clients across those Ds so you get a a global reservation and not just a local one so the intention there is that you can tag this client is going to get this reservation and even though it's talking to lots of devices it reaches that that goal but what's again what's missing is how you're going to configure it and how you store what the policies are and how that that's we have to figure that out we're interested in both sides of both being able to limit people and also being able to find the people who are abusing yes so just measuring usage is one of the things I should have mentioned that that we want to do in manager so have the have all the OSDs in the system sample the request streams and send that information back to the manager so they can build like a top top view basically of who's doing all the IO in the system okay thanks yes do you have any plan on CPU optimization because now self itself consumes occasionally more CPU than disks yes yes we have many plans to optimize the CPU utilization so that's largely what the the OSD refactor one of the main things that's looking to address but also just across the board we're doing a lot of profiling we're trying to figure out where we're wasting CPU and data structures and whatever else that just shouldn't be done the way it does so if you're a if you like optimizing and profiling whatever then like we'd love to have you be involved but yes there's lots of work going on there that's that's a known issue especially as the devices get faster and faster we have to reduce the amount of CPU that we're using yeah is there any improved plan about the cross region RGW improvements for the RGW cross region across region so in dual the the multi-site federation for RGW was almost completely rewritten and so there's a whole new way to configure zones and zone groups and replication bidirectional stuff and there are ongoing improvements to that bug fixes but there's no new feature per se in Luminus except for the the metadata indexing so but it's it's changed since hammer and firefly so if that was the last time you looked at it it's it's newer and better and more robust and all that yeah two brief questions I was wondering if there's a way that we're going to be able to detect the list or the version of all the clients that are connected to a cluster because it's a bit hard when you're doing upgrades and if you forget one of your clients yep I mentioned usability a couple times we're building a trouble board with all the annoying things and that was one of the cars I just added the other day I think it's a simple enough thing that's going to make it into into Luminus so that we can also gate changing that minimum required client on whether those clients are connected so it'll prevent you from saying require Luminus if you have hammer clients that are talking to the cluster or something yeah and then the the other thing is is there kind of work around figuring out if you have some OSDs that are example have high latency because usually those will cause a lot of issues in a cluster and are hard to find yes you're not going to have hard to appreciate so the the OSDs already report just their sort of average latency metric to the monitor and there's already a command called seph OSD perf that you can pipe to sort dash K2 and whatever and you can you can see it but it's it's annoying you have to go do it so one of the ideas is that the manager will be able to now that it it has all those metrics it can do it's do that automatically and we can you can write you know easier to understand code in python that just looks for the slowest OSDs and then writes some policy around that like set the primary affinity on those devices so that they're not primaries and so the reads go to other devices and you mitigate that or preemptively fail them or whatever it is you want to do yes thank you yeah any other questions all right thank you very much


నాతితా ఆమరింా కారీ నంత్ వింప్నతా fingernaత్నత్నిన్ర్. షిమాపిఎందినంయర. నాంప్నినంత్నట్పది నవివిబితంత్న. వింినేయానింొా ఇదినిబర్న్యిలి కోార్. ఇదింాని వ్నినాన్యకలండికోపెవాంసండ ప్దక్యసంచాంచి. and communicate to the formers through SMS. In the second case, a prototype was discussed variant as a case study from Tamil Nadu. The weather windows have been identified earlier through permutation and combination process and by that for Tamil Nadu they have identified 54 weather windows which covers the entire మింి నానినాంచాస్కి. మోత్ప్కి అనిసకికిక్ మారిక్తోనికికిసి ఎఅత్గకిసి. ఇదిసిమి మిటిటక్టరి మరిత్టక్టికటికిని బంఘిమిరంచౕి� in the constant plot. So this computer after selection the server after developing the weather forecast goes to the weather window constant weather window is selected and any crop is needed the crop constant crop is being considered and agro advisory is automatically communicated to the farmers through SMS everything automated at one end our scientist and the other end the farmer in between all interface with all machines and servers and everything. Now it is a third type it is being done at form level especially at Australia this response forming see a type of forming system I can say form planning is being practiced at Australia till now it has been developed by steward et al there is a good concept since the individual land holding is more than 5000 hectares in Australia this system is going on very well with that country but this was also just at about 20 years back in India but somehow it was not taken by our farmers due to many reasons but it is a wonderful technology it can be practiced at the form level if the farmer is very much knowing the information about response forming let us discuss today this type of forming this is very good here you are combining the seasonal climate forecast with the individual form decisions no intervention from any agro advisory institution or anything else the farmer himself prepares the agro advisory based on the weather information he obtains from the constant meteorological organization of that country in respect of India this is India meteorological department if you go to Australia the beru of meteorology like that different countries have their own meteorological organization they provide seasonal climate forecast here only they consider the seasonal climate forecast of a season or in terms of India it is long term or long range weather forecast being considered so here in the people many people tried to give definition now I like to go to one example you know drought means no rainfall and dryness everything for this situation more than 100 definitions are available around the world similarly for this response forming also we could be able to spell out too many definitions but I have screened and like to provide about two or three so that you can understand what is response forming one is framing flexible adaptation strategies adaptation very very important for the forthcoming rainy season so you are preparing yourself you are meeting the challenge from the forthcoming seasons that is a definition the second one is it is a meteorology that identifies very very important critical value and quantify seasonal rainfall related risk so in advance you are able to identify the forthcoming seasonal rainfall risk and you can develop the strategies for addressing them at form level this is very very important form level means individual farmers level then the third definition says that coupling seasonal rainfall forecast with appropriate agronomic responses how the terminology gets deferred so when they define the response forming means you can able to understand what is response forming by different sources and another definition is given by somebody it is a flexible system of forming in which key form decisions key important priority decisions affecting crop water utilization see we were discussing about crop water requirement when we were discussing about rainfall I was telling that rice requires about 1400 mm pulses is 200 so based on the forthcoming rainfall season we estimate what would be the available water accordingly you can select your crop for that season this is the apt definition for this one more definition is it focus on water and its management form level similar to A and B then these are the some definition given but to overall response you are responding to my questions similar that response forming is not met in response to these forthcoming seasonal rainfall rainfall what you do at your form we call it as response forming now there must be a hypothesis this was tested experimentally at Australia they put a hypothesis that solution to forming problems may be found it is possible that is solution to forming problems may be found for improved forecast of expected seasonal rainfall behavior in the cropping season very nice you can able to select or you can able to provide a solution for the risk to be received from the forthcoming rainfall season this is a wonderful hypothesis where it is possible or not it was tested and this hypothesis was agreed that solutions are already available so you can pick up the solutions based on your knowledge based on the scientific knowledge to meet the challenge from the forthcoming rainfall season then there must be a aim also very wonderfully they have put it exploiting the high rainfall potential if rain comes more how to use it can you store it can you conserve it without losing to any sea or ocean so exploiting the high seasonal rainfall you change your land use pattern you change your cropping pattern from pulses to rise to exploit the high rainfall and minimizing the risk of crop failures during the poor rainfall season so you are meeting both the situations I was telling about weather code drought code and flood code so response forming covers both the course also so this is how this is the response forming is like that very very important to be considered then which forecast is suitable we have studied now cost and short range forecast then media range forecast of a different in nature that is a regional climate model numerical weather protection and also long range forecast seasonal climate forecast so we have land a little bit on these forecast types what are the forecast would be highly suitable for these response forming is your long range weather forecast and seasonal forecast because they have lead time 45 to 15 days earlier to the season you could be able to smell this is the quantity of rainfall you are going to get it so by that time you can prepare yourself for meeting the challenge so you can take a land use patterns see normally the farmer in Australia they cultivate wheat if a farmer has a 5000 hectares he may cultivate a 2000 hectares wheat suppose there is a drought then he will reduce the area 2000 hectares or 500 hectares even they call out the animals when the drought is anticipated they may have 50,000 blocks or 5000 kettles they mercilessly they reduce the cattle number to 100 or 50 so that forage can be fed in the case of the drought so they prepare themselves for the forthcoming seasonal rainfall you can select technology if drought comes means you can select the drought management technology like that many things are available in scientific knowledge so those can be rightly utilized for this purpose then where response forming useful can we practice in humid climate I was saying humid climate in humid climate rainfall and PIT potency of transportation are equal same if it is 2000 mm your rainfall the operation also 2000 mm there is no imbalance to equality so it cannot be practiced in humid environment it can be practiced in year 8 where the rainfall is lesser than 50% of the potential evapotranspiration annual basis semi-year 8 it is 0.5 to 0.8 I told already you kindly refer your earlier class notes then regions with the climate variability this is very very important variability means change in the climate now I can explain between the climate change and the climate variability earlier we have pendulum clocks so the pendulum oscillates like this so pendulum is here it goes to positive direction it comes to negative direction then comes to middle this is the climate variability this may be 5 years come to normal then negative side 5 years come to normal so oscillating normal code your drought code and the flood code this we call it as climate variability in the case of the climate change the pendulum weather goes to minus stage there permanently the mean stage there does not come to the normality that is the climate change so this technology is highly useful in the case of the climate variability I think you must have understand the difference between these two things so response forming must be highly suitable for this situation and also certain villages by having the indigenous knowledge they have developed the strategies against the flood and drought see we were discussing yesterday I hope climate managers such wonderful people are are still available in certain villages and they have tailored certain technologies local technologies to meet the weather operation situations now additional tool what other tools you want in addition to seasonal climate rainfall forecast you would you must have a document on answer time when the rainfall is going to set in a particular village or particular domain then these are some statistical conditional probability of rainfall conditional probability of rainfall means if some rainfall you fix a certain quantity of rainfall whether it comes or not that is a conditional probability in the probability of your fixed rainfall is 30% means you may not expect that amount of rainfall the conditional probability probability is more than 90% then you get definitely that rainfall you can decide your form decision making process then probability of initial probability of seasonal rainfall then rainfall return period and length of growing period form size information land use pattern these are all optional if you do not have do not worry you get your seasonal climate forecast and do your form decision making at your form level that seems to be very excellent now I like to produce a case study from Coimbatore whether the response forming was very good or bad so the normal crop of a particular village I am talking about a village not individual the normal crop of the particular villages rise during October November they get in addition to rainfall water support from your ground water the normal rainfall of the village is 420 during the season so 420 millimeter rise needs about 1400 millimeter so excess is being obtained from the ground water the seasonal climate forecast is says for the village that the village is going to get only 250 mm rainfall against 420 so deficit only 50% of the normal rainfall the farmers are going to get so farmers response to Asia what the farmers have decided 80% of the farmers did not believe our forecast they went for rise and 20% listened to our forecast they went for blocker this was 80% farmers went for conventional rise cultivation 20% having understood in the importance of weather forecast they had gone to blocker what happened the rainfall was around 250 millimeter as forecasted and rise whoever raised they met with failures and the fellow who raised the blocker came with success so here cost of cultivation farmer with blocker this is per acre or 4000 meter it is around 5000 rupees the fellow rise farmer along with the nursery development everything he went up to 12000 rupees revenue this fellow has got 7500 while this because of the crop earlier he did not get anything so gains to the farmer of the blocker is this 7400 and this cost of cultivation so 19500 minus your cost of cultivation so net gain is 14500 by listening to the forecast is it not wonderful of doing the response farming in the case of considering the seasonal rainfall so this is a best opportunity to utilize this mode so you use the agro advices whether it is coming from institution or through automated mode or through the response farming you do weather wise farming system and get reduce the risk and get your crop success that is all thank you very much.


Okay. So we said, okay, we need to discuss. We need to hear each other. We know that that's a difficult step in order to resolve conflict, but it's a necessary step. Right. So let me just share this. Okay. So it's something that needs to be done. So lovingly discuss.  It's easy for us to discuss happy things. But when we hear someone say, okay, you know, here are some things that happened and because of which, you know, you fought and when somebody points out, speaking the truth in love and saying, okay, this is where you went wrong. It's not easy for us to take. It's not easy for us to receive it. Right. So, but it needs to be done. Okay. One of the things that really help help in this discussion. Okay, one of the things that really help us is this is the kind of statements that we can use to share what went wrong. Okay. Now, suppose, like, let's say the husband and wife wanted to go to church on time. Okay, now this is what normally happens. So they wanted to reach church on time. The church service was at, let's say, nine o'clock nine a.m. And something happened, and they went late. So they went, they reached by 930. And yeah, so so that is that cost a conflict. So what happened was they started late from the house. So as they were going, both of them were hurling in cells at each other. This is what you did. This is why we are late. And the other person was saying, you are always late. You always do this. Remember last time also you said that so all the while they are fighting, and then they go to church and then they are not even looking at each other. And on top of that, that this sermon was about forgiveness and the whole thing of unforgiveness. And so, you know, they are very condemned, very convicted. So all this happened, right? So, let's say they are trying to solve, they're trying to resolve this and sitting down and talking. So what would be the best way to communicate what happened? Okay, maybe it was the fault of the husband who woke up late. Okay, and maybe the clothes were not ironed. The wife's clothes were not ironed. And they started ironing it only in the morning. Okay, so that also contributed to it. And, well, you know, so many other things, right? All these small details. So what would be the best way to communicate that to each other? You know, you're speaking the truth in love and you need to share this. Because this needs to be sorted, this needs to be spoken. So the best way to share this is to use these I statements. Okay, so to say that I felt hurt or I got upset. I got upset when I saw you ironing your clothes in the morning. Because I've seen I've told you some 10 times earlier, you know, that's not the correct way to do it. I've, you know, we discussed it in the past that that should not be the case that we need to, you know, get everything ready the previous But I got upset.  You know, the thing is that the same truth. Okay, what is the fact one person got upset. And that person who got upset is telling the other person that I got upset. Right, is communicating that and what caused that person to be upset. It was because that person did not get ready and was ironing the clothes at in the morning. Okay, but when you say it this way, you know, I got upset. I was angry. I was upset. I was disappointed when I saw you ironing your clothes in the morning, instead of getting ready the previous evening itself, getting them ready the previous evening itself. And the fact that it delayed us. And because of which we went late, I was upset. I was angry. And I'm sorry. Right, the other way to say it is, you make me angry. Every time you iron your clothes in the morning, instead of getting it done the previous evening, you made me angry. You made me upset. You delayed us. Right. The difference is the same thing. The facts are the same. But when you say when you make these I statements, you are deflecting, you're changing the direction of focus. Right. Instead of blame, it is like you're objectively stating how you were feeling, how you felt because of the situation. Okay, so same thing is communicated. But here, you're just communicating how you felt, and you're not making a judgment on the other person. Okay, so then it helps the other person to again, you know, reevaluate what they were saying, what they did, and come to a place saying, yeah, I know. I'm sorry I did that. Well, now it's the turn of the other person. I felt hurt when you constantly criticized me and when you constantly blame me. I felt hurt.  You know, the neighbors could hear you as we get into the car or as we got on the bike, the neighbors could hear. In fact, the children were watching from the balcony and you didn't, you know, when I felt hurt, I felt ashamed by what you said in public. Right. So, in this way, we begin to share and we begin to, okay, come to a place of saying, okay, now what do I do? How can we avoid this? How can we, how can we resolve this? Okay, so, so you come up with some solutions. Okay, let's, okay, I'll, it's something that works. Okay, let's get ready. I'll, okay, I'll choose to get ready. On the previous day, you come up with something and follow through with it. Right. So that's something that, okay, it's not just an idea, but it's something that you decide to implement, carry out that idea, implement. Right. So, so all this is done in an environment of goodwill and peace, which is something for us to learn. Okay. Because all through, whenever we've had conflicts, we've probably just let loose. Right.  It's okay. Just let loose. Just forget about it. We'll deal with it later. Now, right now, the other person needs to know how I'm feeling. And I'm just going to know, I'm not going to hold back. I'm just going to give it to them. Right. So maybe that's how we have learned. Okay, conflict, trigger, anger. This is it. This is the learned behavior. This is how I saw my parents do it. My parents were yelling at each other, arguing with each other. So the kid learns, the child learns and we learn. Okay, this is how I should respond. The slightest trigger as the slightest thing that people are not getting ready or people are not doing what I want them to do. I'm going to, you know, just yell out at them. I'm going to talk down at them. I'm going to blame them. I'm going to make fun of them. You know, maybe that's how we've learned it all this while. We need to unlearn those things. We need to learn not to put down the person. We need to learn, you know, not to be sarcastic.  The other person is not laughing. It's not funny anymore, right? When the person is not laughing with you, but you are actually laughing at them. It's not funny anymore, right? One person is feeling bad and the other person is just going at it. So we need to, you know, discuss it, and this needs to be done in an environment of Shalom or peace, you know, resolving it in peace. Now, like we said, when we choose to resolve in peace, the Lord Jesus is backing up that person. Blessed are the peacemakers. You know, making an attempt, you may not be perfect, but you're making an attempt and your intent is very clear that you want to make peace, right? You want to come to a place of agreement. You know, sometimes we would need to let go. We would need to, you know, just like what the Lord would have done, you need to forgive and let go, right? And there are those times when we need to do that. And this pride comes in the way, right? And there are times when you're thinking, you know, how many times do I need to let go? I'm the one, even the last time, you know, I was the one who was doing this. I was the one who backed off, how many times, right? Well, when you know that the Lord is applauding, you know that the Lord is backing you up in this, right? You can just go ahead and do that. Okay. Well, the Lord will deal with the other person. You can be sure of that.  And maybe, you know, it's not as mature as you are. It's not as patient as you are. But you can be assured that the Lord will deal. You can be assured that the Lord is reaching out. The Lord is changing, pulling on their hearts and making their hearts soft and tender. So you can be assured of that. But it's very important that we do the right thing. We want to choose to do the right thing in that, you know, in that whole conflict or resolving the conflict that you choose irrespective of what the other person's response is. Now, that's a difficult thing, right? So irrespective of it. You want, you need to want to do that. You desire to do the right thing. So we looked at five steps so far, right? The next thing is to forgive. To receive forgiveness and to give forgiveness. Both are important. Okay, to forgive. To forgive, when we forgive, we are not saying that many times we find it difficult to forgive because we think that it is, you know, how should I put it? Because we think that, okay, we are actually not, we are condoning what the other person did. Okay, we're just brushing out whatever that person caused, that hurt that they caused. You know, when we say forgiving, I forgive you.  Well, it's not that. Forgiveness is not that. Forgiveness is acknowledging, yes, this is what, this is not denial. So many times we find it difficult to forgive. How can I forgive? How can I say that all this didn't happen? It's not denial. We're saying all this happened. Well, the person is guilty. The person is guilty of saying this person is guilty of doing all this. And yet, I'm choosing not to be, not to take revenge. I'm choosing to forgive. So what is the standard of forgiveness? We see in Ephesians 4 and verse 32, standard of forgiveness. Okay, be kind to one another, tender hearted, forgiving one another, even as God in Christ forgave you. Okay, so that's Ephesians 4, verse 32. We see that even as God in Christ forgave, so that is a standard for forgiveness. So if Scripture gives us a standard, not just forgiveness, but any aspect of life, okay, this is how you ought to live. We can be sure that the Spirit of God will equip and empower us to do it. Because there's no point, you know, if this is not possible, if it's not possible to live in this manner, then the Bible has no right to dictate to us or instruct us and say, okay, you live in this manner. The Lord would not do it. The very fact that the Lord gives an instruction, it's because it is possible. It is possible as He empowers us by His Spirit. It is possible as we cooperate with Him. It is possible. So this is the standard for forgiveness, saying, how God in Christ forgave. So when you look at it, it seems like, wow, how did He forgive? He forgave unconditionally, He forgave. You know, not looking at my response or not looking at my reaction, He just forgave. So that's how, that is what is expected. And He forgave repeatedly. So that is what is again expected. So if I have that posture, then I can forgive, you know, God, you forgive me. So I'm willing to forgive. You know, sometimes the person's act has got so much trauma. It could be violence. It could be domestic violence. It could be, you know, maybe violence, verbal violence, like verbal abuse. It could be physical abuse. It could be sexual abuse. Now, in such cases when there's, when the conflict is very, very intense and it's so difficult, it's good to get someone to intermediate, you know, maybe a counselor, maybe a counselor, maybe a pastor, maybe a believer who's unbiased to help navigate this whole process. Because the hurt is too deep. And also maybe it has been something of a repeated thing that has happened over years. This total breakdown of trust, right? So in order to come to that place of forgiveness, you need someone to help, right, individually and also to facilitate that forgiveness, maybe, you know, as a couple as well. Right. So it's okay to get help. It's okay. It's not a stigma. Maybe in some cultures we feel that, oh, it's a stigma. How can I talk about, you know, these things that have happened to me? It's a shame. You know, I'm ashamed to even speak of that. You know, how can I even do that? But the fact is that as long as we keep it, you know, stuffed in and we are not really receiving healing, we need to be able to talk about it, right? There's no stigma. There's no shame if we want to receive help. There's no stigma in reaching out to get help. In fact, it is a brave thing to do. It is a courageous thing to do to reach out and get help. So here we're saying, okay, to receive or to extend forgiveness. What are we doing? We are accepting. We are being specific. We are accepting what we could have done wrong. We acknowledge our wrongdoing and we apologize specifically for what we did wrong. Sometimes, and it really helps. Rather than saying, I'm sorry for all that I did. You know, in a blanket statement, I'm sorry for all that I did. I'm sorry for all that I said. Well, it's good, but it's good to specify. You know, I know I should not have said that. I'm sorry. I know I should not have done that. I'm really, you know, please forgive me. It's good to, it's good to be specific. And it really helped in the healing process. It will help the resolving of the conflict to be specific and to say that. And to actually, you know, it's a difficult thing because you're owning up. So it's good to, you know, look at the person and face the person and say that. And maybe, you know, maybe not at that place yet to face the person and say it. Maybe you're writing a letter or a note would help to start, sorry, to start off. Okay. I'm sorry I did and I'm willing to change. I, you know, I will not give an assurance I will not do this again. So can you please forgive me? Can you please forgive? So the thing is this that, you know, one thing is to ask for forgiveness. The other thing is if someone asks us for forgiveness, right? So are you willing to forgive, extend forgiveness? You know, both are important. One is to ask for forgiveness and receive forgiveness. And if the other person asks for forgiveness from you, are you willing to extend that forgiveness? The other person is saying, I'm sorry. You know, sometimes we, well, it needs to be, it's need to be said in all sincerity. It should not be, you know, done in a very superficial manner, trivial manner. But we are not quick to forgive, right? We want to take our time.  I want to see some tears. They're too happy. How can I forgive? Right? But the thing is forgiveness is a decision, right? Extending forgiveness is a decision. And this is what we see. Sometimes, you know, it's because it's so deep, we are not unable to forgive. But this is the promise that if we are willing, okay, we can actually come to that place and say, Lord, I'm willing. And as we come to that place of willingness, the Lord gives us the ability. Sometimes we don't even feel those emotions. You don't emotionally feel like forgiving because it's been so traumatic. What we've been through is very, very traumatic, very hurtful. So if we come to a place of saying, Lord, I'm willing, right? You make that decision and saying, okay, God, I'm willing to forgive. I'm not able to, but I'm willing. Two different things, right? One is the will, the decision. The other one is the ability. And you're saying, God, I'm willing, God. Yeah, I can, I'm willing, but I'm not able. I don't feel it, God. I'm not able to. The Lord understands. So if we are willing, the Lord gives us the ability to do it. You know, very interesting verse, right? Philippians 2 and verse 13. Philippians 2, maybe you can just read verses 12 and 13. Philippians 2. Therefore, my beloved, as you have always obeyed, not as in my presence only, but now much more in my absence, work out your own salvation with fear and trembling. Verse 13. For it is God who works in you both to will and to do for his good pleasure. Right? God who works in you both to will and to do. So both to come to the place of willingness. Well, God works in us and we saw how we can prepare our hearts, spend time in his presence, you know, receive his love, receive his enabling. So it's God who works in us both to will and to do the implementation, the action part of it. Right? The decision and the action. God actually works in us. The Holy Spirit works in us. Right? So you see that, you know, in marriage, you see the part that we want God, we need God. We can't just do it by ourselves. He designed marriage. He designed it. So we need the designer. We need the creator to be part of our lives. Right? Okay. So both to will and to do for his good pleasure. Right? So we release forgiveness, we extend forgiveness. And the last step is to release a blessing. Now this is even more. Again, it's going to draw out a lot more. You're saying good things about them now. Lord, bless them. Lord, prosper them. Right? God, you know, it's not this, I forgive. It is to extend forgiveness. Yes. But it's also to release a blessing. Right?  Pray for those who persecute you. Bless those who curse. So it's a blessing. You're releasing. You're releasing the, just think about it. You've come to that place of extending the same forgiveness. Now you're blessing each other. And I remember we had this in one of the workshops. And we just spent some time, of course, it was not about resolving conflicts or anything, but we just did this exercise where people would come and speak blessing. Okay.  Right? Speaking words of affirmation from the scriptures and just blessing. It is very powerful because it's the word of God.  And then, you know, from the word we just, just declaring scripture, declaring blessing. And it's very powerful. Very powerful, right? Ephesians 4 talks about how when we speak edifying words, there's an impartation of grace. There's an impartation of grace, which means there is an enabling, there's a enabling power because of the grace of God. So when you release, when you speak a blessing, there is that impartation of grace in their lives. So it really seals the entire process. Okay. So we looked at seven steps.  You are, well, we are, we are choosing to, you know, even though we might be reminded during that whole process, we might be reminded of certain things which causes us to become angry, maybe you're looking at that person, you're seeing the expression of the person and you're thinking, okay, I know I need to, you know, you're reminded of maybe taking revenge or not doing things from your heart and all that, but you're choosing to stick to, you know, the process of resolving and knowing that, you know, this will result in both of you coming to a place of agreement. Now, again, just want to remind us about that verse. You know, we saw in, was it 1 Peter, chapter 3, it is to come to a place of understanding. Husbands likewise, dwell with them with understanding, giving honor to the wife as to the weaker vessel. What is the result? That your prayers may not be hindered. Okay. So you're coming to that place of agreement. And the Lord Jesus also said, you know, if two of you agree towards symphony, meaning coming to that place of agreement and praying and asking, touching anything that you ask, it will be done. So this conflicts, these conflicts prevent us from coming to a place of agreement, right? So our prayers are hindered. We're not able to pray effectively and we're not able to do much as a couple. And this whole thing is interfering with the communication, interfering with the peace in the house and all that. So God really wants us to come to the place of agreement, dwell with understanding, right? So we are refusing to be divided. We're refusing to let the enemy divide, but come to a place of solving conflict and releasing a blessing. Okay. So, well, okay, that's what we looked at. All the matter in peace, give the last three, you know, give and receive forgiveness and release blessing. So we pray and prepare. We receive God's empowering to love and forgive and receive God's wisdom to address the situation. So the first three are what we do on our own personally. Then we address the matter. We discuss it with the other person. We are speaking the truth in love. We are making those I statements. We're saying, okay, I felt this way or I experienced this when you said it or when you did this. It was upset when this happened, right? And we are choosing to resolve in an atmosphere of peace and goodwill. So we're thinking of options. We're thinking of ideas. Maybe we are brainstorming things and saying, how can we solve this problem? And how can we move forward? What should we avoid and all that? And all the way you're saying, okay, I'll do this. I will do this. And then you're receiving forgiveness. You're giving forgiveness, extending forgiveness, keeping in mind that, yes, the standard for forgiveness is how God and Christ forgave me. That is the standard. And then what will we receive forgiveness? But also we extend forgiveness to others, to our spouse, and we don't stop there. We choose to release a blessing upon their lives. So we do that. And then we resolve, we resolve, okay. So any questions here, based on maybe your own experience of resolving a conflict, maybe there could be some challenges. Maybe you're saying, okay, this is, sounds good in theory, but practically does it really work? So any questions? Where do you think the challenges are? Maybe some things that you need to work on? Or maybe you felt that, okay, this is not working. Maybe we could talk about that. Any questions? Anything from your own lives personally? Maybe if you're a single person, you could talk about maybe how you tried resolving a conflict generally with another person, and what worked, what didn't work. And if you're a married person, you could just talk from experience, share from experience, or based on your experience, maybe you have some questions, what worked, what didn't work, how can things be better? So yeah. Any questions, anything that you want to share? Go through all these seven steps. Anybody who tried it before? And maybe all these seven steps, maybe you did it in some order, in some measure, and you felt that, okay, somewhere something didn't happen, or maybe it was a great success, you could share about that as well. Anyone? There was a conflict between better, and for the first few weeks, we didn't just talk about it, and I thought it'll just go away, if we just don't talk about it, it'll just go away, and it'll all be alright. I think it was before I came to Balakalach. So I was actually packing for Bangalore, and I just made him busy working so that we just didn't talk about it. I was just telling him, we packed this, get this, but there was a little disturbance over the conversations, even when we were talking. Something was not right, something was not good. So I think before I came here, we should speak at night, after every month. We just got together, and I told him, whatever I said, I'm sorry, whatever I did, I'm sorry. I'm also sorry, it was good, because I was about to move, so I don't want to leave with such conflict, from home.  So that was good. After that, I think it's good. Right, wonderful. So the thing is, now you took the initiative, so that's a big thing. In a sense, it's a courageous thing to do. You felt trouble in your spirit, and you took the initiative. You said, okay, I can't leave this undone, I need to address it, I need to do this. Now that's a big step. Many of us, we feel that something needs to be done, but okay, I'm going to wait for him. I'm going to wait for her to actually take the step. So that's where we sometimes just miss those opportunities, and then it just gets over. Time just goes by, without reconciling, things could actually become worse, worse off. So the thing is, to take that big step, I think that's a great, that's a good takeaway from this whole thing. You took the initiative, and so did you feel a little uncomfortable, in a sense, a little weird, a little uncomfortable, just to talk about those things, or was it okay for you? Yeah, I think I took time, actually, one week. I wanted to solve the whole thing, but then I'll go to the room, and I think I'll speak about it, and then I'll come back, not speaking about it, I'll just speak about something, and then I'll come back. He was also not showing any expressions, he was also acting so alright, and I was like, okay, maybe he's alright, maybe I'm not like, he's alright, but then I spoke, I came to know that, you were also not alright, he was also having that feeling inside, and I spoke, it was not easy, it was not at all easy, because I think the conflict hurted both of us, we also want to be healed on the inside first, before speaking about it, I can't just go and shout at him, he was also hurted, he was also hurted, so I can't just go, why, how can you do this to me, I can't ask in that first week, maybe I would have just shouted, I think, because how can you ask these questions to me or something, but I took some time to be healed, to understand what is his perspective, why he said all of those things to me, and then I finally came, okay, this is his perspective, so he did it for my good, he thought about all of these things, and I told him what I thought at that time also, I also told him why I was hurted, why those things were hurting to me, why he didn't understand my perspective, so I also told that this is my perspective, this is the whole thing, and you talked on this side, but somehow it's all okay. Yeah, so that's the thing, see it doesn't, so that's another thing to keep in mind, that it's okay to feel a little uncomfortable, it's okay to feel a little weird, we might know the person well, we might not know the person well, but then it's going to feel a little uncomfortable to be able to do this, so as we take the initiative, as we want to do this, to expect that, so you know that okay, this is how it's going to be, but then it's okay, I need to do it, right? So the thing is that, when we consider God's perspective, God is for, the Lord is for peace, the Lord is for reconciliation, and we see it in so many ways, in terms of forgiveness, in terms of the Lord blessing the peacemaker, and calling them sons of God, so he's for reconciliation, he's for the resolving of conflict, okay? Now, so one beautiful thing happens when it comes to resolving of conflict is that the bond becomes even stronger, okay? So we're talking about husband and wife, and the relationship becomes even stronger than before, right? Because simply because it's in God's perspective and God's eyes, that's his will, when you work out his will, then you're in the will of God, you're in the plan and purpose of God, then there is God releasing his blessing upon it, so it becomes even stronger than before, and it's amazing how things can turn out, how the marriage can improve, how the marriage relationship can improve, how the communication can improve, and so on. But here's the thing, you know, one last thing before we close this chapter is that we need to resolve to keep strife out of our life, okay? So what is strife? You know, strife is intense quarreling, strife is intense, like fighting maybe. So we need to make a decision, okay, to the best of my ability, right? I need to keep strife out. You know it.  Oh, this is a possibility. Now, the dam is going to break if I, you know, if we are going to pursue in this manner, we're going to keep talking in this manner. Well, that's it. The dam's going to break, there's going to be a flood of waters and it's going to be, we are going to be overwhelmed. You know, it's going to sweep us, all that has been built, you know, over these days and over these months, it's going to break down, right? You know that. And in fact, God actually wants, as he gives you a check on the inside, you know that. Okay, I should not pursue in this. So this thing is to put a pause, right? And say, okay, let's not go any further with this. Let's just be quiet and say, okay. So that is what we see, you know, even in the scriptures that we read today from Proverbs 25, we see that, well, the word fitly spoken, or by long for parents, a gentle tongue, well, this is going to change the whole scenario. It's going to change the equation. It's going to change the atmosphere, right? It's going to do something in the relationship. So yeah, so the thing is to pause, to take a step back and not proceed further in the same manner, right? So quarreling, contention, strife, it's going to affect everything in the home. Just understand that. It doesn't affect children. We think, okay, when we have quarrels, I don't remember, like when my parents used to quarrel, well, it affected me. For some reason, you know, there was a lot of unpleasantness and I just wished that, okay, things would just get better. I just wish the things were that they would not quarrel, that they would not be angry. And sometimes I would feel guilty saying, though it was not a fault of mine, maybe I should do something to stop. I felt powerless to stop. I was not able to stop. And sometimes, you know, I used to feel guilty. I think maybe it's my fault. Because as a child, you know, you're not mature enough. And you're processing things differently and you're seeing and these things happen and it's kind of affecting you as well emotionally. So you think about that. Think about all that. You actually do not stand to gain when there's constant strife and quarrel and arguments in the house. It changes the whole atmosphere. Scripture says that in the tense of the righteous, there is a voice of rejoicing. Well, that's not there anymore. It talks about, you know, how the Kingdom of God is righteousness, peace and joy in the Holy Spirit. Now, that is not an automatic thing. It means that we need to cooperate with the Holy Spirit in releasing, in cooperating with Him, to have His Kingdom, to have His rule and reign in our homes. So strife is the opposite of that. Strife breaks down everything and strife is the weapon of the enemy. And the enemy creeps in very subtly to create a strife, to create disruption, to create disagreements. And we know the plan of the enemy, John 10 and verse 10, very clear, it is to steal and kill and destroy.  Destroy the very marriage itself. So that is Saviour's plan. And the Lord has come that we might have life and life in its fullness. So when we choose to keep strife out, now do we need to do it in the right manner? Just because we need to keep strife out, you cannot live in denial. Like if somebody is just abusing, if somebody is doing something, just because you don't want to, just set that right, you're going to live in denial. If you're going to just brush things off, that's not going to help. We need to address, we need to confront, but even it's done in a loving manner. Okay, so we'll stop here. Now there could be several questions in the sense I know, what if one person is all, one person is all in and wants to solve and the other person is unwilling, totally unwilling? What do I do? I'm all for resolving, but the other person is not. So obviously it needs a lot more from us, a lot more from you. One who understands that, well, God wants this. And so for both the husband and the wife to have the same, I mean the same goal or the same vision and saying this is what we want, it's very important. It's very important for both the husband and the wife to understand the goal of marriage. It's very important for both husband and the wife to understand, okay? This is what God wants. This is what God has designed marriage to be. So that's why it's important at the outset to have a proper understanding of, so what if it is not there? Well, in some way we can facilitate, right? Well, maybe the church can facilitate, maybe the ministry can facilitate. You have a lot of resources out there, maybe a Bible study on marriage, a series on marriage. There are ministries which have marriage workshops, talking about marriage, and they do it in a very non-threatening, in a fun way at the same time, exposing the couple to the Word of God, exposing the couple to God's plans and God's designs. So it can be deeply convicting, and you come to a place of saying, yeah, it is possible and I want this, right? So that is very important. So the thing is to get help, okay? The thing is to get help, the thing is to put pride away and say, and have those heart-to-heart conversations with the spouse and saying, we need to change. We need things to change. It's not helping. And to find out the root cause, why is it? Maybe one person is in some form of addiction. As believers, as carnal believers, some form of addiction, something, some emotional affair, something which is causing this, which is opening the door for them, something that needs to change. So that needs to be addressed. So the thing is to get help, get understanding, and then do it, okay? Yeah, so this we see as something that is very important and especially as a young couple about to get married. It is for everyone, people who are married, especially for people who are desiring to be married. So the thing is during courtship days, everything is fine, everything is happy, and they're all very starry-eyed and refusing to look at the negative side of the other person. We're also presenting our best self to the other person. I don't want this person to not like me. So you're just presenting your best self, and you're not really, the person doesn't see the negative side of you, or at least just glimpses of it. So it is very important for such people who are single and who are desiring to get married to actually have this capability or to have this skill if we have a conflict. How are we going to resolve it? To have this understanding, right? Okay, so we shall stop here. If there are questions, we can always ask in the next class as well. So we'll stop here. Thank you. God bless. We'll meet again soon. God bless.