So thank you, everyone, for coming. So I'm going to be talking about event sourcing, and I know it's not a drupal topic, like a typical drupal topic. It's actually a more type of high-story data, but I think it's an interesting topic nonetheless, and I want to talk about it, so I hope you guys will enjoy. So my talk is named The Future is a Thing in the Past, and, yes, it is a very indescriptive title, but that's, I think, a fun part about it. So my name is Mitchell, and I'm an event sourcer. A friend of mine made this.  So let's not waste any time, because I have a lot of ground to cover, so let's just get it right into the core, right? So into our domain. So in this talk, we'll be talking about having a board game, like running an actual board game online shop, and we'll be covering a couple of processes throughout this web shop. We're going to sell a board game, or actually start selling a board game, and then we're also going to restock that board game, because if it's so popular, it gets out of stock, and then, you know, we have to restock it. So these processes are very domain-oriented, and which means that they're actually very, very much tied to the business, and not some crud operation. They're not some SQL query we're actually going to execute on the database or whatever. They're like, we use a language of the business, right? Because our manager might say, well, we have to restock the board game, but developers might say, well, we have to, I don't know, increase the stock integer on the database, right? That equals the two actually restocking the board game. So this has a lot to do with domain-driven design. Who here has actually heard of domain-driven design? Please raise your hands. That's a couple people, that's nice. Domain-driven design is also known as DDD, just for the younger crowd, D to the double E. I'll quickly continue. So domain-driven design is all about understanding the business, like having conversations with the business, trying to understand how all the processes work between departments and everything, like how HR and how sales work together, and how all those things just basically work. Like you have to understand how the business works before you can actually make a useful application for it or a website for it, right? So, and domain-driven design in that aspect fits very well with, or sorry, that fits very well into domain events. So what are domain events? Well, they are the concept behind event sourcing. So the thing is that without event source, without domain events, event sourcing would not work. And the thing, it will get away around though, because you can have domain events in your system and I'll explain a little bit what domain events actually mean, or what they are. But you can have domain events in your system without using event sourcing, without event sourcing them, right? So domain events, they describe a change in your system. They are kind of like facts in your system, like how board game stock was replenished is a fact in your system. We executed that action and then the stock was replenished of the system, of the board game in our system, right? And these are a couple of names of just domain event examples. Like we have member was registered and somewhere like we have cargo has shipped or order was fulfilled, or even if we run an application that detects earthquakes, for instance, we can have earthquake has occurred or some arbitrary thing, right? So what you notice is all of these things have something to do with the language. They're not some technical, they don't have technical details, they don't have crud operations, they don't have all that. Basically everyone can read this and understand what was happened, right? There's no technical aspect to this. So we might have some normal class, so we might have something like this, but this is how we would normally do when we have crud. But, oh, did it just drop? Oh, sorry, I didn't, I stopped here myself. So this is how we would normally have it, but we would actually want to look at this because this is understandable for basically everyone. That's what we might want to get to, right? So the words that you use in these names, in these domain events are extremely important, because if one of those words actually means something else in a different department, or if you use product, for instance, and product means something different to someone else, then you're going to have conflicts. So if you say like product was added to shopping cart, for instance, and someone thinks a product means something else, then you can have conflict in your business. You can have conflict in this logic. And domain events also capture a change in time, and it tells you actually what has happened in the past. So again, in vague words, like what are domain events or what are events? Well, events are a log, history facts, and sending stone. I thought I had more than that. So what does a domain event look like? What's the anatomy of a domain event? Well, it has a name, it has a payload, or it has data. It's one of the two. I use payload, some other people use data. It's really, you can just use them interchangeably, right? So using the board game stock was a plenty of example again. What we want is we want to have all these small little things. So we have the board game stock with the planters, we have the ID of the board game, and I'm using a UUIDs, not incremental, auto-incremental IDs because there's a whole lot of pain when you're using auto-incremental IDs when you start to talk about distributed systems and just systems in general. And I do not personally like auto-incremental IDs. That's a personal thing. And then we also have the amount that actually was restocked, right? So these domain events are supposed to be very small and not something like member was updated with 78 or whatever, how many fields in there because this actually doesn't tell us anything. If we look at this, well, member was updated, but we have no clue what was actually updated. We would have to dig into the data and then try to figure out, try to understand these patterns and that is all going to just cost us time and why not just sort of correct events, right? So what are some conventions of these domain events? Well, first of all, I would like to say that please do not take these conventions as absolute. Just look at them briefly, take them with a grain of salt and then, you know, create your own basically. So first the name, use past tense because these domain events, because events have happened in the past, these changes in our system have happened in the past, it's extremely logical to actually use past tense to use past tense in these names, right? And also try to make sentences because it's easier to read sentences than something that, I don't know, it's just basically unreadable for you. So you try to make it as humanly readable as possible and then actually leave out all the technical stuff, like cash keys or whatever in like, or other technical words in the name, that all doesn't really belong there because these domain events are something that is important to the business and the business doesn't really, I don't want to say business, I mean like basically everyone, like the business is kind of like a weird concept, but the business is basically everyone that is not making the application, that is not actually writing the code, right? Like you can have a business in whatever, anyhow, leave out the technical stuff. Who here actually plays Overwatch? No one? Damn, that's the first. Anyhow, this is one of the games that I play and they have a payload in there because I thought it was funny to add them there. So the payload, right? Again, leave out the technical stuff. If you have like a request ID or a cash ID or like anything technical, it doesn't really belong there because all of these changes like, we can talk to our manager or like to our sales person or to someone and say like, well we have like a request ID in there and he's like, well what does it do? Well, then you go into this huge as monologue with him and like he will not understand what's going on or something, so it doesn't really matter. Like that doesn't belong in these domain events because again, business. Keep an eye out for the amount of properties. I tend to go with no more than six or eight but this is really like a soft limit. If your domain event needs more than that, like 10 or 12 or how many, that doesn't really matter, just try to keep an eye out because what's possible is that you can have multiple domain events actually like housed inside this one event. So originally you thought that this one domain event was actually, this is one thing but later on you might come to an understanding this one domain event has like two other domain events and whatever, so you might want to take those out of there. And try to use simple objects. What this means is instead of in the data do not throw like he's higher object in there actually try to just throw in identifiers in there like these small little concepts, these small little objects, right? So some actual code finally. Here we have a domain event. It's extremely simple as you all can see. What it does it implements an interface that has, sorry, sparkling water, that has just some getters basically in it. It's not necessary to go into detail on that but right now this just implements an interface and it accepts all of the necessary data for this domain event. This domain event doesn't need anything else. It doesn't need the name of the board game or whatever because we already have all of this data somewhere stored. So what we need for this specific thing, for this specific change in our system we need the board game ID because we need to know what board game was replenished and we also need the actual amount. So what event sourcing is in a nutshell is instead of changing values you just append events. That's literally it. So thank you for coming and see you in a minute. So in other words, instead of modifying the columns in your database you just create a new column and have all of the changes in there, all of the data in there. So with event sourcing we literally only store events and events only. So changing your system which means that changing your state because what you have is you have all of your member information in there and that's all your state, all of your persisted application state. We cannot have a state change without having events stored. That concept is synced in. So question is why will we go through all of this like why would we go through all of this all this effort of storing events and then instead of just changing columns in your database, right? Well the thing is you won't actually lose information because when you look at this I mean this is a very simple query. So we might be able to pull some information out of there because we wrote this maybe a couple of weeks ago or maybe a couple months ago or something and we might even be using like an SQL builder or some other library to create the actual SQL statement, right? But when we look at this, this doesn't really necessarily say well we updated the board or like we replenished the stock of a board game. This doesn't mean anything to the business as well. So you also lose implicit information because what you can do is when you have a bunch of events you can start to figure out patterns between those events and that is implicit information very valuable for your business maybe because what you could see is you could see you can actually kind of like figure out the behavior of your users and then maybe improve your system according to that. So all of that information you would normally lose because I mean if you have an auto lock then you might be able to get that information as well but the thing is when you have an auto lock then why aren't you just event sourcing because the thing is that if you're event sourcing and having an auto lock in there then they will actually diverge or like I say that wrong if you have like a normal database just with regular tables and just updating those tables right and all those columns and you have an auto lock then those things they're not actually like tied together so they'll eventually just diverge because if you forget to update the code that changes the auto lock then I mean the auto lock is just like all of a sudden automatically incorrect so that's not very helpful so if you have an auto lock you need an auto lock then event sourcing are one of the most perfect things you can have so with event sourcing and with domain events specifically as well is then when we know what we actually know what and when it happened so I'll actually continue on that thing a little bit later because first we have to talk about how things are stored kind of so we have a bunch of events stored somewhere somewhere in some database I'll get to that later on that we have these events stored in some way and they're all like all sped out and what we need to do is we need to try to order them because we need to make sense of all of these events so these are a couple of events and these are tied to to specific board games well this thing one of those columns is actually called an event stream and an event stream is literally an append only array that has an identifier that has an ID and then I get most of the time with event sourcing specifically a UUID and why UUID, well the thing is you can literally generate 100 million UUIDs per second for like 100 years or something and then you might only have like one collision probably so these things are extremely unique which is very helpful if you want to have literally millions and millions and millions and millions of events or millions and millions whatever right so these domain events are stored in sequence of time and they also represent the life cycle of your domain concept of like for instance in our example we have a board game the concept of a domain concept analysis system is a board game so the question is how do we then record these events like how do we actually get this process started like because of creating one of those domain events is as simple as instantiation the class and throwing in the values in there but how do we then store these events well we do it through the object itself so who here knows actually what name constructors are awesome well there's a name constructor and what it basically is is it's actually a factory method so it's the equivalent of this literally but instead of creating the board game through the new keyword what you do is you actually use the language of the business because we're going to start telling the board game right so this is useful to actually have in there we can now look at the code and we immediately know what it's actually happening we don't have to figure out what's happening if we have new people coming as well and inside the company or whatever we have new developers then they can look at this code and then they say immediately well the board game started selling here so this is all useful information to having your code and this is all the information that's necessary for our system to have a valid board game like we just need a name and a price that's it and in this very simplistic example we're storing the price as an integer or sorry as a well yeah as an integer I wouldn't recommend this always store your money in sense obviously because you know rounding errors but this is what the actual start started selling or start selling method looks like the name and a price and then inside inside of that method we create the utility instantiate the board game because we need to somehow instantiate the board game right and then we call this record method on there and then we instantiate the actual start selling board game event class and this accepts all of those parameters to make that domain event valid to actually have like a valid consistent domain right and then we immediately apply that what you might notice here is that we're not actually changing the values we're not actually like adding the name somewhere or like adding the name or sorry we're not actually changing the name property inside this class what happens is that the apply method takes care of that right so and I'll get back a little bit later when we actually call it a prime method so we have all of these steps in here and I should have done this when I was talking through it and then we record and then we return the actual board game because we need the instance because we have to get the events out somehow so all of this means that when this when this has happened that the board game and I know this is kind of like right the board game will then hold this hold the actual instantiation of the domain event class right well then what well we get back to pending events because at that moment this specific entity this object has pending events and we need to store those events somewhere but where do we store them is your question I all hear you're asking it well we store we use an event store and an event store is literally just one table that looks like this it's the most simple thing ever we have some identifier so we have the stream ID and the stream ID is for instance in this game the board game ID because we need to we need to attach all of these domain events to just one point right then we have each event also get their own you ID because I mean maybe we want to get this one event back from the event store later instead of just the entire row and then has the version what the version is I'll explain in a second as well and then we have the payload metadata is additional information that you might want to store in there and here so if you want to actually store the IP address for instance with the request ID you have like cash or other technical information this is where you want to store it in the metadata not in the actual payload because the payload is just for the domain it's just for the business but you want to store it in the metadata name and then when I was actually recorded right so we have the instance of the client to our event store and we just get back the ID from the board game and then we write the events back to or we write the events to the event store well um what happened now is we started telling the board game it was extremely popular because I don't know if you any of you actually play board games raise hands who knows cold express the board game also well if you all play board game you should play it cold express one of the most fun games actually I've ever played it's also the best game of 2015 I think the best board game so it's insanely fun you should actually play it so it was insanely popular people bought it and heaps and now it's out of stock well we have to replenish the stock obviously but how do we then get back the actual state of that object because well we just persisted those events we just stored those events and now I mean like how do we actually get back to the state of this of this board game right well if you use doctrine for instance or if you use eloquent or some other or what you could do is you could say you could just get it back very easily and the thing is with event sourcing it's basically the same but what you do is you first get back all of the events so you have the actual ID somewhere stored and then you get back all the events and then you rebuild the actual board game because you have all of these events so from the start you get all the events then you throw them in the rebuild from this also again a name constructor so at this moment we have actually we have two ways to actually instantiate this board game and have it in a valid state so like we cannot create the board game and actually change some data out of nowhere like change some data wrong because we can only start telling the board game or we can rebuild it from previously stored events so everything like all this code in here is very simple and this is actually like the production code I use as well it's very easy honestly so we can start to hit the board game and then we apply every event and this this apply method looks like this it's the most simple thing ever again so it just checks if the instance of a domain event matches this one and then it applies and then in the apply method it literally does this just assigns the values to those properties and then we can start changing them again or do whatever so we want to restart the board game and we need a lot of them so we restart the board game 60066 then we get back all these events again and we write them back to the store well the thing is that we need to actually show these domain events not these domain events we need to actually show these board games because when we have a store and start selling a board game well if you have all these domain events that's not very user experience friendly because if you just throw all these domain events on the screen then the user will have no idea what they're actually looking at so we need a listing of these domain events but the thing is that replaying events is actually a second this is old anyhow so for getting all of that what I just said because that's old code replaying these events is cheap actually getting back all of these events through the client and then replaying all of those back that's very easy because PHP is actually kind of fast to some extent but have like thousands and thousands of events like if you my maximum that I actually use was I believe like 3,000 events or something at some point and that was just literally to have a test because normally one of those objects will never have 3,000 events unless you're working in like banking or I don't know some other gambling whatever so but at some point it'll become expensive but for most of us it won't but at some point it actually will well this is kind of where read models will come into play because if you want to if you try to replay every object that you need on every request what you'll do is you'll then for like a bunch of different objects for like the member object and the board game object and all these other objects if you try to actually get all those back every request you have to rebuild them every request so that means that you might you might actually be rebuilding a couple objects and maybe like a couple hundred events or something and if you're going to do that then that's not very optimized right so read models they are just objects that read data state from some place from from from somewhere in your in your architecture right and in this case it's a projection and this is a projection the projection is something you might actually be used to it should literally just a normal relational database table for instance or it's you could use a text file or you could use a graph queue the Neo4j for instance however you can use anything basically you want but a projection is the state of the events put somewhere and this is the latest state of our of our board game so a projection can be several things like a MySQL table an elastic source document and all those other things right so how do we create this then this projection well we have a projector and a projector is very simple it is just like an event listener that listens to these events and then gets the data from those events and just stores it somewhere so what we could do what we could have is we could have a listener that listens to those events and then we just execute a very simple SQL query that updates our table right and that'll look something like this because the projection on the right side is the equivalent of all the plate back events on the left that's how we got to that actual state drinks and water so the key thing to remember is that with projections you modify it with each event and the cool thing is that the projections are actually disposable because what you could do is you could just remove it destroy the entire thing and because you have these events in a separate storage you can just replay them again and all will be good right so this is actually very similar to how banking works and a lot of other things but banking is the most simple example I could find really is that your bank balance is literally just a sum of all of your transactions that you ever made or ever had so you can have like credit, debit and then you spend some money, got some money spend some more like the new iPhone or whatever that thing is insanely expensive I think it's even more expensive than this anyhow at some point you end up with some arbitrary number and that is then your that is then your bank balance so that is the state of your bank balance right so this is very cool if you think about it because bank balance like this one this one number is obviously a very simple example right but you can imagine that when you have all these events for like all these different objects and then at some point if you decide to actually change your system drastically like how it looks for instance and you have all these different pages well what you could do is you could literally just throw them away create a new schema all these new objects that actually read from these schemas and tables and then you can just re-project all these events grab all the events that are not necessarily and create the state that you need and then everything you know is good so with events since you're recording everything you can actually react to everything but there's one little disclaimer and it's actually like right now like a hot debate or a hot topic that some people say that you should not actually event-sourced the entire your entire application but some say that you should event-sourced the entire application anyhow I think right now still like we're kind of like in a weird face with event-sourcing I think we shouldn't personally event-sourced the entire thing because it is quite expensive to maintain and that is just because when those events are actually stored in the system and you want to actually change those events right what you need to do is you need to go through all of those events and this is all in production at some point and you would have to modify all that state and that is actually against the like the core concept and principles of event-sourcing because event-sourcing like those main events are set in stone like the history and like right now we as humans cannot also not change history right so why should we then be able to change all this anyhow the sorry did not actually store your event-sourced your entire application because of the insane high cost not of maintaining it so we actually replenish the stock of our board game now but our customers need to know about it because what are they going to do like refresh the page every five minutes and hopefully there's new stock in there right now that's not how we work because for the first century and we have cool technology like phones to get emails on or whatever or notifications anyhow so we just notify them and the thing the cool thing about event-sourcing is everything can be at some point like all of a sudden event driven because well you have these events and they are like you can actually then dispatch them through your event dispatch and then you can have all of these things listening to it and do all of these cool amazing things that you would normally have to do in the same request for instance or whatever right so we listen to the board game stock of the replenished event and then it goes through a message bus which is just literally an event dispatcher well this is the event apparently it says hi and then at some point we'll hit the actual listener because the listener is listening to this specific event and then we execute that listener and this is a very long name but it is very descriptive and I don't actually have to look in this class because by reading the actual name I immediately know what's going on and this is not the longest class I've ever had the longest class I've ever had like 33 words I believe it was and it might not have been the best name but you know I knew what was going on so anyhow and then it continues into nothing anyhow and at the end all is good like always so these listeners we were talking about those things can actually half state right so we call them process managers and so if listeners have state process managers and a very good example of this is for instance an administration application where the trial ends when 10 invoices have been sent right and these invoices the send invoices are obviously stored also in domain events so we have all these domain events fired our business is booming like we just started but at some point we already in the first month we already have 10 invoices sent bunch of money anyhow not actually money because I should continue this so 10 invoices are sent and then when the 10th actually is sent right a side effect is triggered we rebuild the entire state of the account and then we call the end trial and this will then also store another event and that will then be able to pick up a listener which will then send the email to the actual user right so this is like how you can have like these chain events of actions in your system so one last thing who here has stupid question who here is your database before raise your hands please alright cool basically everyone so you know that it's just mostly like your database oh I had another question who here has multiple systems accessing that same database raise hands alright cool that's very dangerous by the way just saying I do I have a two but you know it's it's it's a it's a tricky thing and why is it a tricky thing well because you can have shared state and what it is is actually shared mutable state so to illustrate this is we have a database and application and this inserts something into this database right and then it gets something back and then we have some other thing and then we have some more things and more things and blah blah blah and all these things are accessing the same database well this seems simple enough at first because you know we had to like we did we didn't have to change anything we just created all these new applications and we had a data we had the data there somewhere so why not just use it right well everything is access which is difficult because then the consistency will be tough to illustrate this again to illustrate consistency is that well we have two services and they both get the same data at the same time right but then service one changes that data internally like in memory to ABCD and service two changes to ABX right and service two is kind of faster it's on a new server or whatever and then it changes the it changes the it persists the latest change of that of that board game which is a very simple representation of the board game right but it changes the state in the table certainly right now you see the table that's ABX and then surface one comes along and then also persists the data but it's ABCD then at some point and then service two is all like what the hell I just changed this data because it's not there anymore so then you throw your workstation and you just go home and whatever crying a corner and the thing is that event sourcing can help with this consistency problem because with event sourcing not actually changing this data you just appending new things right so there's no real conflict in there and that's awesome in my opinion it's one of the benefits of using event sourcing that you actually not really think about but it's a very big problem if you then have inconsistent state in this one application and like you're actually trying to figure out why it's happening but you probably do not like have a well you can actually look at the logs but it's not abundantly clear what's happening because you know whatever so what other things are possible with event sourcing for amazing things are possible well with event sourcing you basically have a time machine at your disposal which is actually pretty awesome and I hear you all say and like think about this movie right back to the future and yeah when when these new events are actually stored and you're passing them on the on the event dispatcher what you do is you can have listeners generate HTML pages so what what you might have is let's say you're organizing a conference right let's say Drupal and you have a CRM and in that CRM you have all the data for your speakers but then you want to generate the actual page which is a different application when you want to generate the website which is just like a stand-alone thing but you don't want to pull all the information from the database because you want to have it as fast as possible as very snappy and you also do not want to like invest maybe in caching or whatever which is very simple of course but you want to have just generated HTML pages with the content in there because that's still the fastest thing ever and then you can also cache it to make it even faster anyhow you can have that you can have all the information in the CRM in a separate system and if that system then throws an event and you can actually listen to those events to listen to those changes you could then generate your pages from that right so that's also very cool to listen to and you can project to multiple things so let's say you have MySQL for actually showing data on the screen and then you have Elasticsearch for searching you can have one event that is thrown on the event dispatch and then you have multiple listeners listening to this same event and changing data in multiple places same thing goes for like the new version of your board game let's say that you have like a dashboard of the board game or something and you want to have 5% of the users then have the new version but you still want to have the old version running well you can just create another listener and that listener will then persist the data to this new table and if you have you obviously need some other system to actually manage that load that 5% of the people actually get the new version but the data is all there and if your application supports it then always good again and we have also the best auto-lock ever because all of the events we cannot actually change state without storing events so everything that has happened to the system is actually in events and fixing schema bugs or schema changes are very simple as well because you just throw away like I already said throw away the schema and then replay all those events and if you want let's say you have an application and at some point you want to tweet when you have like a million people or like a million users or whatever well you can have a listener that listens to that actually counts up the amount of subscribed members and this is just like a little gimmick but I think it's fun and all of these things are very possible with when you actually use events not necessarily event sourcing but more domain events but event sourcing and domain events work very well together so you can also do much more obviously but that's all for my talk I know it was kind of rambly and kind of like here and there but I don't know that's kind of like the way I present so it's whatever but I want to say one last thing a friend and I we have created like this awesome education platform it's an online workshop it's a course basically it's called event sorcery and it's an introduction to domain driven design seeker as an event sourcing so we go into insane depth into every of these topics right now I believe we like two hours ago we released another episode of seeker as literally everything on seeker as like the most descriptive the most descriptive explanation of seeker as ever in our opinion obviously but yeah we have like two and a half three hours of video right now and we're constantly adding videos so because it's kind of like a living product what we did is we were also getting back feedback from people and then if they want to see like changing the video for instance then we make those changes and re-upload the videos again so if other people like have feedback or like have questions or whatever then we can incorporate those into videos and help other people as well that's the entire thing it's like in early access right now because we don't have all of the videos in there it's like an insanely reduced price anyhow that's event source free again thanks a lot for listening to me Rambo Nick Spell is a friend of mine who made the image so he's an awesome designer whatever exactly he does a lot of things but I wanted to give him credit for that awesome image in the beginning and are there any questions if not then that's easy for me obviously but uh oh yeah there's a microphone didn't even see it I did a project where I have an application which monitors solar power plants and I did it event based by accident I didn't know after that that's actually how event sourcing gets discovered because it's like it's a natural thing of how humans tend to think because humans tend to think in events so then that's how event sourcing basically gets discovered but I still have a question about scaling the data gets in every five minutes from the power inverters okay and if I store every value after a month or so I get into scaling programs when I replay all the events to get the current state of a plant I save the states into projections per day but I don't know if it's working say five years or so okay so how long does it actually take to build these projections it takes about ten seconds ten seconds okay so I know systems that literally take an entire weekend to regenerate the extra projection okay because like there's actually how all these like highly consistent and highly available systems work like banking for instance and all these other systems have like millions of events and like if you need to generate some report or whatever it doesn't really matter if the data isn't really if the actual projection isn't immediately consistent so the thing is what you have is this concept called eventual consistency is that when you write information to the actual event store it's possible that the projection doesn't immediately have the latest data so the actual person who is viewing the website or viewing the application might not see the actual latest information it really depends on your business and on your application and like the requirements of that if you actually want to have it like immediately consistent that is like the question so is it important for you to have the data like immediately consistent or it's like one time a day one time a day an update is that perfectly fine for you that's basically a question I would ask if you can get it down to like near real time that's awesome which is like 10 seconds is insanely insanely close to real time in my book because I have projections that take like a couple of hours when I generate a daily report and the projection takes longer than a day to generate I mean there's not really a way around that honestly it's one of the things with events so yeah it's one of the problems I have some ideas on this but we might not we might want to talk about it afterwards because that could take a while but yeah so I mean the question is that doesn't really matter to your business like that's the thing and if so what you could do is you can have one of the solutions is it creates snapshots for instance that instead of so those are actually like mini projections so every 500 events for instance what you do is create a snapshot of those of that state yeah exactly they hold snapshots whatever it's different in each category anyhow that's one of the possibilities obviously other than that there's not really much you can do and maybe I don't know I'll give away as well that's also very possible because I started learning event sourcing a couple years ago and the concept is simple but all these other problems that are caused because of event sourcing it does a lot of awesome things but it also increases the complexity of your system like by a couple hundred times so yeah so yeah we could talk about it afterwards any other questions yeah you were talking about different systems contributing to the same we're writing to the same database I haven't seen how you would potentially resolve conflicts or make sure that the events are replayed in the right order when they come from different systems to identify well do you remember the version thing in the schema that is actually like the number in line so the first event appended to an event stream has a version number one the version is basically just the order number because the thing is if you have all of these very fast systems or if you have all of these systems that are like appending events every millisecond or whatever you cannot really trust time especially with distributed systems you cannot trust time because time is different in each machine and time zones or whatever right so what you want to have one of these solutions is you could have when you get back out of the events then you actually know the latest version and then you just increase that number and then you try to append that and if you try to like let's say you already have 44 events in there and you try to append another event that is number 44 then it might throw a conflict as well that's one of the things that actually like prevents you and makes it that event sourcing that your storage your state is highly consistent but the thing is that that actually makes it less available because consistency and like being both consistent and available is extremely tough especially when you have all the like have 10,000 events for instance per second or whatever that's extremely tough so you kind of have to choose one or the other like it's highly consistent or highly available and what you could do is you could like if it's highly available let's say it throws an error, it throws an exception like we already have this version or what you could do is you could create like a version 2 of that event stream that copies the entire thing and an asset on there as well and then later you can look at that and then you can manually merge those things or you can have some automated system do all of that so that's one of the options so you kind of have to choose highly consistent available it's tough I know thank you no worries I have a small question what do you do if you have a distributed like systems but then you have side effects that happen for example you have an account balance like let's say 10 euros and then you have one one place you deduct 20 and the other one adds 30 but there is an event that happens when the bank gets below zero for example but then it depends on which order they get back so this is actually fun because I started reading more about banking for the future exactly so the thing is with banking if you go to an ATM for instance it could be that the power is out like the connection is just gone it's just dropped right so what happens then is you can actually withdraw as much money as you want because banks want high availability over consistency so what happens at some point let's say you have only 10 euros in the bank and then you withdraw 30 euros then you actually have like a you're in the negative so that's how banks use it or do it because they just want to be highly available and then obviously you're not going to escape a bank I was just wondering what the side effects like your account gets cancelled when you're below zero so it depends on whether you first got the transfer from someone else and or you know what I mean yeah yeah yeah so in the kit there will be a conflict well that actually depends because again do you want to be highly consistent because if you're highly consistent then you won't have any conflict at all because it will actually lock the same thing as the event store will actually lock the thing right the actual transaction so that is one of the possibilities or you just let it happen and you figure out anyway because what's the worst thing that can happen like some accounts owes you like 30 euros well if you have like a couple hundred or a couple thousand or a couple millions users it doesn't really matter and that's like one of the things that do you really want to be programming every solution ever or do you just want to have like a good error system that actually catches that error and then notifies the correct people and they'll actually handle the situation because if this only happens like one time in a thousand or one time in two thousand then you might want to not so essentially your solution is to let it happen and try to notify the people and to shift the problem to the business side that's actually like one of the things that I always do is I don't actually make it my problem I make it their problem and then they actually give me solution and I'll program it right so that's like the best thing to do because I'm a developer and I'm a tell them developer but I do not know how your business runs I do not know how you want to run your business and I'm not actually going to make those decisions as well so every time I have a question even if it's the most stupid question ever I just ask them I mean like what's the worst thing they can have that can happen they actually think you're a complete idiot well who cares right so I mean it just depends like sometimes you might want to have an automated system that take care of that but if it's like a one in a million type case or whatever then I would just let it happen and report the error any other questions that's it then if you start playing Overwatch at me that's an awesome game and thank you for listening


Good morning. It's the last day of the Boston Summit. I know that it's early. I'm from the left coast, so I got a little bit of coffee in me. Hopefully I can get my mental rev on this morning. But looking forward to the rest of the day, and I hope you have a good summit so far. So it's want to introduce who we are. I'm Joseph Sandoval. I'm the manager, engineering manager, of the Cloud Platform team at Adobe AdCloud. Mikaela Mogalenko, he's on my team. He's one of the lead SREs for the Cloud Platform team. And he's been one of the drivers behind our adoption of the topic that we're going to talk about today, which is chat opsing your production open stack. And so just to kind of get that business side of it out so that I don't get in trouble when I come back to the office, is to let you know a little bit about us. You may have seen me speaking before last year when I was in Barcelona. And we were to Mogal. We were like a leading advertising programmatic advertising company. And at that time, we had got acquired by Adobe, which was a great opportunity for us to really lead in the AdCloud space. We're really the first end-to-end platform that allows you for managing your advertising across whether it's traditional television, it could be your computer, it could be mobile. But what it does is it really makes it simple to be able to really deliver video, display, search ads. And it's an independent cross-channel platform, so you can identify, engage with your audiences, and be able to reach them no matter what. And especially now, as you see what's happening with ad space is shifting. It's been from the traditional world to where it's in this dynamic programmatic world where there's so much interaction behind the scenes. But at Adobe AdCloud, the one thing that we love to talk about is we really want to make it an experience for our customers. And that's what we try to deliver. And as far as our team as well, that's what we try to do as well. We really want to make sure that as we serve up our platform that it's a good experience. We don't want to make things harder, introduce impedance. Or when someone has a request, we want to try to make it so that they're easily enabled to get the answers that they need. So I like to think of the platform that we run is like the bacon of the platform. I know my boss loves bacon, and that's kind of what we are. If you're a vegan, I apologize. Just think of it as the faking bacon. But it's really kind of like what is core to run our business. And this codename is really representative of our infrastructure platform. The one thing that we've been very pragmatic about is that even though that we are running primarily open stack, we have workloads that also need to be considered because of the high volume of traffic, the billions of requests per second. So bare metal is in our infrastructure. It's something that we use on our edges. And we really just try to take an infrastructure as code approach. And as well as that, we know with our business that we have to scale. And so public cloud is something that we consume in our platform. We want to make sure we're in a position that we can serve up to our business on demand what they need to be able to really make our customers happy, make them have a really good experience. And so at this point in time, we are getting to the point where we're going to have six clouds online. Currently, we have four running. They're running Ice House at this time. We have two new ones by end of summer, which will be rolling us into Mataka. And when I had a chance to really reflect on what we had, what I realized is that we actually have about a little bit over a 1,700 or 15 compute. We're trending getting closer to that 100,000 cores. It's going to be an interesting milestone when we get there. But the one thing that even though as we continue to grow, our business continues to grow is that we have a core philosophy of being lean in our approach. And so what that means is we have to really be judicious about the things that we choose in our platform. And so when it comes to OpenStack, I mean, we've took in that same approach. I mean, there's a lot of great rich set of services. But our business really, it's core. It's compute storage network. I think storage is a little bit kind of like I would say secondary. But we really need to really, as a small team, be able to support a growing platform. And so we always have to keep thinking of ways of like, how do we make things easier for us? How do we extend things out to our customers so that they can get information that they need at any point in time? And that's where the chat ops has really come in at and it's really helped us.  I know of you, you've ever had a chance to kind of look at Jesse Newlin's talk, but I think he framed it in such a concise way when he said that, you know, what chat up was. And he said, it's like putting tools right in the middle of a conversation. And for us, it could be a conversation about, you know, what we're developing. It could be a conversation about operational things that, you know, we need to get done. And if you just think about kind of like that, that normal dialogue of what happens, it's not like it's a true disruption in regards to how we go about our workflow. I mean, you think about what happens anyways when you're in a channel. A dev may ask you like, hey, I think I'm seeing something with this cluster in this region, what do you see? And then we'll probably reply back and come back with like some graph, some data and say, well, this is what we're seeing on our side. And there's that dialogue that kind of goes back and forth. So, you know, within that chat framework, you know, all we're really doing is just kind of inserting a bot that then interfaces or wraps around a lot of the tools that we're using in our infrastructure. And so one of the drivers of why we would do this is that, you know, in our world, it's really easy because, you know, we live and breathe open stack day to day. We know our tools. We know how to get the information we need.  But it's not always that way. And so for us, it's kind of a way for us to extend knowledge. We want to make it so that they can actually be power users with using chat. And so, you know, for us, there's a lot of benefits.  People can roll back. And, you know, we have teams that are in Kiev. We have teams in India. We have our team here in the US. We have some that are just spread across. You know, this is a great way for us to kind of capture like what happens. Like if something happens in the evening, we can easily roll back and like see, all right, hey, something did happen. What did they do? And so we got a nice system of record for us to be able to kind of go back and to roll back. I think when Mika and I were talking about a little bit of this before, one thing we liked, and I think I learned this when I was at Lithium Technologies, is it's nice when you can kind of enable your devs to kind of do the right thing.  And so what I mean by that is with automation and some of these toolings around it, we want to make it so that they can easily go and kind of like take care of tasks, respond to events. And so give them ability to provision as they need to provision. And it really kind of like avoids someone like that interrupt driven type stuff that can happen and take us away from what's most important since we are such a lean team. There's some times as well where, if you think about you're out and about, and this is something that happened to us recently where we're out at dinner and something happens, don't have my laptop right there, but if we have an easy way to off in, we can go and observe in a chat channel without having our Nova CLI or what other tools that we have around. So it really gives us kind of that opportunity to kind of get in there really quickly and no matter where we're at, we're able to kind of be respond, understand what's happening in our environment. So it just kind of makes people's life easier. And I had a ticket the other day because we've had some changes in regards to since being acquired, we've moved from using hip chat over to using Slack. But I like this ticket because one of my key engineers, he's basically saying like, you wanted access to Cloud Ninja, that's kind of like our pre-release name that we had for this new bot. And he really said, I just don't want to interrupt you guys. You guys are busy. He just wants to get to the information he needs. And so it really kind of allows us to be able to kind of give him that power. Even though he's not bothering us, but he knows that it interrupts us. So I really appreciate what that kind of gives us. And so just to kind of give you some background, I mean, Mikaela, he started on this journey before I got here. As I mentioned at Lithium, we kind of used chat ops as a way to kind of wrap around Kubernetes. We didn't want to expose Kube CTL. But here at TubeMobile, which is now Adobe, these guys were doing this for like a couple of years.  Like where were the initial things that you guys were tackling and how did you get started? Yeah, so basically we started two years ago. And at that point we had only hip chat. At the initial goal was to simplify day-to-day operations, but to bring like simplicity, but not complexity. So, and we tried to provide like the easy way to for duty engineers to query some information without having any access to CLI. Right now we're in progress of migration to Slack. So, what we did in the beginning, we just build all functionality like a separate library. So, thus we don't need to rewrite everything. It's easy, pluggable to any solution. So, I mean, I know definitely that was kind of like when some of the motivation and kind of getting started. I think the one error that I started kind of like realizing when we started kind of like walking through some of our, you know, what we had currently had done is I really kind of realized that it was like an indirect way to me to teach kind of like some of the junior members of the team as well.  And we wanted to kind of give them a way to kind of like get going, get, you know, hit the ground running.  And so by using this, we're able to kind of like, you know, wrap commands that they can go back, learn and see what they're doing.  So as Miquela mentioned, it kind of really simplified our day-to-day, but you know, let's get to the meat of it.  And I'll just level set, as I mentioned at the beginning, you know, because of CloudMogul, we're really across, you know, multiple types of compute resources, storage resources. But let's really kick it off Miquela about like the open stack part, control plane versus data plane. Maybe you can qualify what that is and how we, I think we all can agree, but you know, why don't you clarify that as well as like, you know, some of the use cases that you've kind of like addressed using chat ops. So yeah, as since my growing every day, so we are hybrid. So we have like public clouds, private clouds and like physical service spread all over the world. So what we do, we, for example, we just provisioning like all databases, all like load balancers, all physical equipment using chat ops. So it's not a replacement for any other tool. So we used to like, we have proper to mention in place like Jenkins and like any frame frameworks. So from open stack perspective, we can provision full open stack cloud if you want it using chat ops. So and the initial goal was to simplify all those things because it's hard to remember what particular configuration you need to have like networking, VLANs, like base operating system configuration for every environment. So and like when we use it, we are fully relying on what we do. So what if we ask to do particular command, we expect it to do it. So we are not trying to add to bring more complexity. We're not trying to replace people, but we're trying to make like it's like easiest to make it an easier way. Yeah. And then, you know, so it's where we were talking about the other day, like someone like the control plane stuff that would be automated. Cause obviously when we talk about the data plane, what we're doing is like we're extending off like, to me it's like the safe spot to really start at because you can easily extend like VM operations. People can get information about, you know, the metadata about their clusters, but the control plane, like, you know, there's some things that maybe you want to want to steer away from like what, what areas did you decide like just, it just didn't make sense for us to kind of go after? Yeah. So basically it's very useful. Like when we build our private cloud, so based on workloads, we have a different purpose for every compute. So, and what we do is the automation with chat ops, we just automatically define based on like on its role where this compute will go. So it will be running like customer facing services or it will be running like some like background task, like database and stuff like that. So it helps us like a lot. Yeah. I think, you know, coming just the manager hat, you know, for me, and I know like I have my peers, you know, constantly asking me about certain information. And so we've also used it as a way to kind of really extend out so that they can easily grab resource information, like how much am I consuming? Like, you know, we don't have a real formal like billback approach to our infrastructure, but people want to see like, you know, what are the size of VMs? What are their, what are their fleets look like? They want to get, you know, like inventory reports. And so those things are like really easy ways so that it's not tasks that come to me. I can really just push that out so that they can get the reports and information they need. I mean, I always love like the, I think GitHub did a really great job of some of these things where, as well as I think I see Dmitry from Stackstorm, where they do some cool stuff with like CI CD pipelines. Those things I really love because I mean, it's so great to have that in a conversation where you can see like a build triggers, something fails, what happens? I mean, immediately everybody's kind of in the know of like what's happening.   You really, what you're doing is like taking some of these workflows of like, you know, of like an action you take, there's an event and you're responding. I mean, it's already there. All you're doing is just kind of like wrapping chat around it. The only thing I would definitely, you know, encourage and we, you know, is to think about the security aspect. You know, with us, I wanna make sure that we're not making it a backdoor into taking destructive actions or that someone who, you know, could get access to things where they can destroy things. We wanna make sure that we really think about that as well. So I've always been like making sure, hey guys, you know, what permissions or what actions does this individual have? I think you had some comments about our back or what you would like to see. Yeah, so basically we had like amazing talks like yesterday when Verizon was like complaining that it's not probably easy to implement like better granular, to have better granularity in access. So what we do is chat ops, we just doing like providing a role-based access control. So if some people need just to get like simple information, they don't need to see what's going on overall in the cloud. They don't want to see like any specific or any other information. So they just need to get what they need to get. So, and what it helps us, what we do, so we just providing information for everyone, for all our customers, like developers, SE, 3D teams, and for our team also. So we manage like access based on ACL, so which is like much more better than like default access rules in the horizon. Yeah, so you talked about kind of like getting started, like what would be the simplest way if you're like just wanted to like kind of kick the tires and like, you know, get going? What would be your approach to taking that? How don't, you know, you gave me some ideas about that earlier. Yeah, so the best idea is to start to start like with your local environment. You can actually just start with DevStack and like to create a separate account for any like chat. As soon as you feel comfortable, you can move to UA or pre-prod. So, and don't try to make a complex thing, just try to like do like a basic stuff. And if you see if it's working for you, what it takes to implement, you can move on. Sure. All right, well, you know, we have a few things. I mean, it's, there's quite a bit of other stuff that we have, but Mikaela, we were going to like walk through just a couple of like some of the things that you were kind of like extending out to the team. Yeah, just a small demo. Okay.  Yeah, so for example, like, let me show you how we work with NOAA API. So, but by typing this command, so I'm trying to get the information about instance running in one of our public clouds. It's basically what is going under the hood. It's just no, it's a wrapper for NOAA show command. So we can see any information about instance. It's metadata, which is very helpful. And next command is like a NOAA list. It feels like a command that you wrote for me basically, as I forget everything. So this, like another one on NOAA, like it's NOAA, NOAA list and with host. So we can get like the list of things running in particular hypervisor. So it's very useful when developers trying to ask, do we run in the same cluster on the same hypervisor or not? So it's the simplest way to check if you have a proper server group stuff in place in your cloud. Basically, if they know they're using affinity or anti-affinity basically, I got it. And then you were mentioning that, you know, like some of the deploys also, we talked about some of the bare metal aspects as well. Yeah, so for us, it's very important to have like, all like information in our inventory database. So based on say, on that we can make any decisions like whether I deploy it properly, which version of OpenStack is running on it. So we basically have inventory database where do we store all information about physical server so we can easily query and check like if we have proper bias config settings which OpenStack version we have. So this particular command shows like every information about like particular compute node. So, but what was really important for our team, it was to be able to test like OpenStack releases pretty fast. So we did some work in Jenkins and we also integrated with, so when you have multiple integration in your chat, so we can easily, you can pretty fast see what's going on. So you're querying like asking about to provision OpenStack cluster and then you get information from Jenkins. For example, if I ask to deploy compute node, it will take just 10 minutes to do it. Yeah, so what basically happened under the hood, we just querying inventory database, getting only the information about networking in top, then we're triggering Jenkins job which is consists of three steps. The first one, it does just basic operating system provision. The second one is deploying OpenStack itself and of course the last one validation. Sure. So it helps a lot in our daily job. So then basically what happened is and you just outputted like the Jenkins job that's running behind the scenes then. Yes, so right now we're provisioning in our lab compute node. All right then, good. So you can see that a lot of these are pretty simple use cases, like just thinking about kind of like where we go from next. We're looking at, these are like simple easy steps but they're like building blocks and that's kind of where like thinking about like where we headed next. I mean, some of the things that we're considering is just integration, even deeper integration into monitoring systems. I was in a conversation yesterday and we were talking about some network challenges and one team is looking at it one way, another team is looking at it another way and it was like we were looking within our preferred monitoring system and I'm like we should just integrate that into it so that in the conversation we're all looking at the same data. Like we'd grab the same graph, what's the timeframe and then we can all kind of like observe and make sure we're all in agreement that we're seeing the same thing versus like bringing all these different tools and seeing different lenses and having misaligned optics about what's actually happening. The other start we thought about is like we sometimes use it already for like retrospectives, like going back, tracking data. It's easy for us to kind of go back and get into root cause and like capture all the data and what was happening and who did what at what time. The other thing that you mentioned was like what you were looking at was anomaly detection. Yeah, so it's pretty easy like when you have like a... But yeah, it requires a lot of work because you need to basically build a dependency graph so but it will generally help you to determine what's going on because sometimes if some particular service is down you don't know what really happened without any pre-analyses. Sure.  There goes my wifi. So the areas that I always like try to caution in regards and I think this has really just to do with like automation in general is like, you know, there's a saying about, you know, you can automate anything but sometimes it always doesn't make sense to really think through like with chat in a similar way, like does it really make sense? Am I really saving time? I think we talked about the complexity challenges. You may want to like, you know, think it through first before you decide. We really try to go for like the really the low hanging fruit, like start with the easy things, the things that are just constantly interrupting you every day, like find an easy way to give that to them. And I would also encourage like, you know, extending the commands so people can see what's exactly happening. You're also like indirectly like teaching them how to like go fish for themselves. The security aspect is really important. You know, now that we are like, you know, in this new world with our business, you know, I have to constantly be thinking about maintaining that we're in line with like our two factor authentication. Like we don't want to bypass these things. So, you know, think about what you're delegating. You know, you don't really need God access to the bot. I mean, that's a really, really powerful thing. So we try to really just go least privileged with everything. And I think that's the best practice we all know. And it's not, it's not hard. It's really easy. It's just make sure you don't compromise those things. And, you know, there's a lot of resources out there. You know, like I listed just a couple that are available. And, you know, it's a very rich ecosystem of tools and things that are available. You know, like where do I see us going next? I mean, I love to start taking these things and like taking these initial building blocks and making even more like, you know, more richer, you know, kind of like orchestration that happens with like our chat ops. So with that being said, I just want to say, you know, thanks for this opportunity of letting us share and, you know, we're open for any questions, even suggestions or comments. All right, it's early. I think we all need more coffee. Thanks for your time.


Calcium carbonate is a widely used commercial product with various applications and its natural formation through microbial activities can be mimicked for biotechnology purposes such as metal remediation, carbon sequestration, enhanced oil recovery and construction restoration. This article was authored by Tingting Eju and Maria Editrick.