hey what's in uh temp upper dir well it is that same file so it is now appearing at the path where my previous uh password file appeared so now i can unmount the overlay oh but i can't because denis richey is great at god as he was doesn't have sudo on this system happily i've already logged in his root on the console like did this demo before i reinstalled my system a lot it's a lot of fun anyhow um so now everything is back to normal um point out when the advantage of doing this overlay is if i bound mount that temp upper dir directory onto slash etsy the only file ls slash etsy would be password everything else would be gone i did that in earlier version of this system and of course i had to reboot if i overlay mount it then all the files that are in password except for this one that has the name conflict appear unchanged this example is silly and stupid because it's a demo but you can see real life use cases of something like this uh i mean most obviously if you want in your container that the uh password file just has root and container user in it this is a mechanism for doing that and it's it's so if you if you have features like that you're used to for live media you can change your preferences for fonts or something when you boot from an install cd this is actually how you're doing it very very often with overlays so as i said uh bind mounts are more powerful but overlays are actually a little bit simpler so um coming into my uh conclusions here um virtual file systems go all the way back to 4.2 bsd they are part of the posits heritage of linux virtual file systems being the shim layer between system calls and specific file system implementations are what makes everything is a file possible in linux uh they are really where the rubber meets the road on virtual file systems um well the where the rubber meets the road for everything is a file um procfs and sysfests that i used as examples are two of the most important file systems of linux and two of the simplest so they're kind of a good way to understand these concepts um procfs is a place where the kernel publishes statistics that are actually generated when you ask for them sysfests is an event management interface it's a place where reference counts to objects that have a lifetime are maintained by the kernel bind it mounts and overlay fs are part of the virtual file system layer um and they make a lot of the magic um of um virtualization and containers and iot devices would read only read file systems work and so they're uh not very uh they're not very sexy but they're very important and then finally the bcc tools and ebpf are much much easier to use than ftrace unlike ftrace they don't produce megabytes of output that you have to painfully grep through they typically produce one or two lines um they have wonderful documentation and i urge you to give them a try if you have not already done so and with that i'd like to thank uh some of my colleagues who made helpful comments on this talk advertised that my colleague sarah newman is giving talk about uh blind users in linux at 6 p.m in ballroom h and here are some references where you can uh learn more about this topic thank you uh but you mean the pdf file of this talk i was just asking where the pdf files are well this talk uh is posted um the link is in the announcement for the scale program if you look at the scale program schedule and you find this talk there's a link to these slides or if you go um if you go look at the title slide here i actually have the uh here's the url well no i didn't i didn't put the url in here anyhow you also can find it open first yeah great thank you yeah yes i didn't really get to that but um i did uh yeah so there's a question about collie linux i actually have that in the supplementary slides here um so collie linux is dependent on uh overlay fs i just went and thought it'd be interesting to take actual live media that a lot of people use and and look inside it and see how um the bind and overlay are used so collie linux where people don't know is a live distro that you can use for pen testing it's a kind of a generic linux but inside uh built in are a bunch of uh fuzzing pen testing tools but what i looked at in this particular instance was not the tools that are in it but how this live distro works so if you look up um if you look in the directory okay so inside inside collie linux when you insert it there's some stuff that starts at running and then it mounts a squash fs which is yet another file system implementation i don't know if this i don't know if this text is large enough to read let me zoom in let me zoom in here oh wait is that better i thought maybe it would let me zoom but i guess not if you so if i i loop mounted which isn't yet another kind of mount the iso for collie and uh if you look inside the loop mount there's a squash fs which is just a compressed file system that collie linux uses for its uh root fs that it uncompresses and runs from memory when it starts and then if you look inside the squash fs there's the suggestively named shell script called 9 9 9 0 overlay dot f h and so and then if you look inside that shell script it it makes an overlay fs like the one i just showed you and so then now you have your root fs which is actually running in memory and you while you it is a feature i mean a positive feature of a live media distro that you don't modify it right because you don't want to go modifying the pen tools when you're actually trying to attack or probe or investigate somebody else's system what you want is just to have a copy of these things that you can use you don't want to trash the files in this distro so that you have to reburn the cd or the usb drive before use it again you'd actually want them to always be the same so this is a mechanism that allows you just as i was illustrating to um temporarily changed your heart's content the things in this in memory file system and then when when you reboot all those changes go away so um yeah this this is uh really handy so if if you want to save out files uh that you're generating um when you're running something like this you have to have another storage device to copy them to um so i don't remember if i had i guess that i guess that was it but i have a you know you can get the copy of this iso um from the coli linux website coli linux is tpl an open source and you can just do the same loop mount and see so i tried to work this up as a demo but i couldn't for i you know just booting a live cd in a inside cumul wasn't very exciting to watch so i kind of gave up on that but yes your example which um which file controls the permissions of the file that the is actually seen in the file system the one that's been overlaid or the one that's been yeah good question it's actually the file that's on top um except for the fact that well let's go back to that sign make it easier to understand so it's just as a follow up while you're thinking about it also the same question applies to the directory itself you know the directory permissions of the directory right yeah so in this case if you look that we have a file in the lower file system that appears unmodified it it will um is that true yeah because that file is really on the storage that's a good question we could actually try it and see um certainly these files that are actually in memory that are in the upper directory you can you can do whatever you want to them and of course that's part of the point of us so the if these files were owned by root or you know unratable they would still be in the after the bind mount i'm pretty sure that if the file is not overlaid it retains its permissions because there's no copy of it in the temp of us we'll get actually nice thing about all this stuff is easy to it's easy to try it so i i like very much this investigation method and then afterwards if you want to change something or you're still puzzled how it works then you go and read the source code any further questions well thanks everybody uh for your attention can you guys turn me okay in the back there okay good five more minutes just uh making sure it works good afternoon i have the pleasure of introducing mr davin edwards he's going to talk about sys emin to his sre creating capacities to make tomorrow better than today i'm a little about mr edwards he is a co-founder of run dak ink the maker of run dak the popular open source self service operation platform davin has spent over 20 years working both the technology and business and of it operations and is noted for being a leader in the porting cutting edge devos techniques to large scale enterprise organizations davin is a frequent conference speaker and a variety of focus in devos sre operations human topic and is also actively in international thank you thank you with that please give him a warm welcome oh thank you thank you good start with a nice applause it's only downhill from here right so uh how many of you are systems administrators by by nature by trade everybody good anybody uh just got the sre title yet do you folks nice all right good uh good stuff so my um uh my talk is about the long suffering work of operations often thankless uh work of of operations and how you know we're always looking for better ways of working and uh sre seems to kind of hold that out as a as a as a guiding light sort of like the devops movement did uh for uh for the rest of the organization and this talk is just talks really about um kind of the what i've noticed from companies that are moving towards the sre style of working uh what's getting in their way and those that are being successful what are they focusing on so when talk is less about the folks that um have grown up their company or or you know have kind of drawn the plane board so to speak in that sre or what is now called the sre model more about companies that have what i'd call you know legacy or the history of success and have had to pull that into the uh the next generation so a little bit about myself this is really kind of matters why i'm um i'm up here so my career i've had this interesting opportunity to see inside all kinds of companies high flyers low flyers uh startups big big household names and um you know that happened through uh my previous life uh the managing partner of dto solutions we got really known for doing kind of devops and operations uh transformations and more so the ops towards dev uh than the dev towards ops that i think a lot of people really see in the uh the devops conversations um i do a lot of work in the uh community um i'm on the the content committee for gene kim's devops enterprise summit if you work in an enterprise it's a great conference uh jason cox down here raising on jason another another helpful organizer it's a great uh it's a great event uh also wrote a book in the uh the new um oraili sre book seeking sre it's a very interesting book it's a collection of essays it's the first kind of non-google sre book from um from uh from oraili so i can see inside all these companies really kind of helps me inform why i'm up here today um and so you know a little story right you know not i'm not that far away maybe in a company just like yours let's think about life and operations like what what's what happens right i mean first of all we all know this right it's the overloaded constant firefighting right it's just whether it's tickets the alarm's going off the phone's going off there's you know the project due yesterday the project due tomorrow um you know fire is going off everywhere right you know the life and operations is kind of marked by this idea that we're overloaded we're running at 110 percent right and then you know we're always waiting ticket queues for everything right so whether it's us waiting for somebody to do something for us or you know uh the other position which is uh you know a bunch of people waiting for us to do something for them right but it's a constant waiting game that's that's going on and then things keep breaking they break again they break again right so how many times you're responding to the same problems the same known incident over and over and over again and really two things are happening here one is the someone needs something fixed and they're waiting on it and then also the constant interruptions that are being pushed onto people hey can help me with this can help me with that right and this kind of sinking feeling that everybody's busy but nothing's really getting better right that um you know it's the constant firefight it's the constant overloaded uh work we're constantly being told there's a new improvement project or a new kind of initiative we have to go but the entire organization is running on on uh on overload and then to make matters worse right uh you know here comes the business execs right and they're just grumbling to themselves like those operations folks right everything's taken too long it costs too much you know it breaks too often right that's just not fair right so all of us is just a big grumpy mess right and I swear this is the only cat picture in my uh I figured it would go over at scale right so in my enterprise crowd I talked to I don't I don't put the cat in there so now I got this sre thing oh this is interesting right you know there's like a little little whispers here and there's little books you can read that sound like fantasy lands right uh you know and then you see but but now you've got like the walt Disney company right you got standard charter bank out of Singapore right you know even ibm's talking about a ticket master right you know talk about big big legacy uh you know people are talking about this sre thing and how it's working for for them right so it generally goes like this the same executives that were griping about those operations folks right they're like hey here's this sre thing it's great google does it right so here's what we're gonna do right we've got sysadmins now so we're gonna make them sre's just like google's got and uh you know it's it's gonna be great right nothing nothing nothing's gonna happen right so they got the plan right they put in the okr it's gonna be amazing then the boss comes down is like oops oh I missed it sorry so it says hey look your new title is sre please write code and be better at operations right and you're sitting here going all right I got my you know I've got my cab meeting calendar I've got as long you know this this 13 week provisioning process and it's uh you know these exertions at quality's job one right if we just would concentrate harder we'd stop making problems right um so you know what happens is basically we have the old system in world right we're you know over uh overloaded constant firefighting waiting in queues things are breaking breaking again everyone's busy bosses are griping that things are taking too long costing too much right so now we've got this new sre world everyone's in everyone's in sre oh it's the same problems right we're in the exact same place around so and the reality is is that you know changing job titles or adding individual skills doesn't make systems administrators sre's right we got our you know our our friend here it's great we can dump all these you know new skills and say hey observability traceability these new programming skills you know think about things in code put everything in an sdlc uh let's learn about distributed architectures let's get Mike Nygaard's release it book and learn all the anti patterns let's get up on our instant responses and our blameless postmortems but the reality is if we're still wrapped in that same organizational trauma so to speak nothing's really getting better right and we're not these are not sre's they're just we're just long suffering systems administrators with new more marketable skills and they'll probably put that on linkedin and they'll be gone and you know six months for for uh richer pastures and so you know the reality is if you kind of dig into what's going on and the organizations are being successful sre's are rethinking of how operations work work gets done it's not about the tools of the technology it's about how do we approach what's the mindset and what's our organizational philosophy around this you know operations uh operations work so if you think about what are these principles what sets sre apart um the google folks were helpful and kind of define these three core principles uh what it means to them i think they're pretty much spot on so the first one is sre needs service level objectives with consequences right with consequences being the key part so let's let's talk about that right so in the old-fashioned world uh we have this notion of a sla which was something we agreed to operations agreed to and if enough operations blew it the penalty would be on on them right but in this slo world um it's just one letter different but it's got some some key uh key differences uh we start by saying art there's some service level indicator and that's got to be a business measurable metric right could be response time up time you know uh transaction time saying a lot of time things over and over again but there's other other things as well and we're going to say this service level indicator uh it gets together with the business and say what are we actually what's acceptable right um f uh you know if partners only submit their their requests for something in we know once a day and we expect uh you know a the next day turnaround we don't need five nines on that on that service right what is the acceptable level of error or accepted acceptable level of of slowness or whatever it might be that will that will that will take and what's interesting about that is instead of kind of the traditional sla world which is ops must be perfect right uh if there's errors at all you know what's wrong with you you know your sla says you're gonna be penalized if you fall below this level but if you're not hitting a hundred percent or something must be something something wrong in this world it's saying well first what's kind of odd is saying well this level between the service level objective and perfection that's a budget right that should be used to say you know we can go faster we can run things cheaper uh you know we can do whatever we want as long as we're within that that budget and we haven't we haven't we haven't blown it um but reality is you know this is about shared responsibility and unlike the sla world which is oh operations you're signing up for this sla generally under duress and you know political pressure um and you know thou shalt keep this thing at this standard or else the consequences are on you in this slo world um it's pretty dramatically different everybody development operations people will build the people who run and the people who pay the bills are all agreeing to that the most important thing in the business is the running service and meeting that slo so something goes wrong it's no longer operations responsibility it's everyone's responsibility meaning that that development has to swarm to help figure out what's wrong with that problem and that means that new business features are not getting done and that means the business has to be cool with that and uh and okay with that and there's no you know oh we've got to get on to the next set of customer features while the site is on is on uh it's on fire so it's radically different departure from kind of the mindset of traditional operations and slas it's really all about the shared responsibilities right so that's number one number two uh of the three is sre's have to have time uh to make tomorrow better than today you'll probably notice that from my my title this is a biggie to me and uh this idea of toil right uh you know people starting to hear toil kind of he's throwing around and people heard that term places yeah so uh the vex row uh google wrote the chapter on toil in the google book and i should say about the google books uh you know people kind of get a little mixed feelings about them because they're very googly right but the reality is you'll see there's a lot of trends kind of in our industry that's been going on that are all sort of filtering sort that same idea so much like dev ops suddenly sprang up when patrick debaugh had the first dev ops days you can look to all kinds of people in the industry who are arriving at the same principle i think if you look at what's going on the google books you'll see that there's a lot of things going on at different companies what they're arriving at the same idea just happens to be the google flavor but they did a very good job of defining some of these key ideas and terms so um it's a toil is the kind of work tied to running a production service that tends to be manual right stuff you do by hand repetitive doing it over and over again automatable you know like you you could have automated it uh it's tactical and devoid of enduring value that devoid of enduring value i think is the key the key message there that's not adding to the business right we're not making tomorrow better than it is today we're just you know fighting a fire or doing whatever is necessary to uh to uh keep the lights on and last one is a very interesting litmus test it scales linearly as the service grows right if you have five people taking care of a service right and say you have you know a thousand users right suddenly something happens and you have a hundred thousand users can you run it with the same five people or do you need do you need 20 30 40 you know people running that service that's a good sign of how much toil is in the system how uh how much that uh the care and feeding of it grows with the scaling of the service ideally uh the care and feeding is the same whether you're you know zero or a billion right so you know what if you want to kind of break down well what's the opposite of toil right the opposite of toil what we want to have is engineering work right and engineering work is is the stuff that takes the human creativity right so if you want to kind of put it side by side toil lacks enduring value right it's stuff that's ephemeral we have to get it done maybe it's necessary right um but it's not uh it's not going to move the business forward right so repetitive cleanup work or responding to an incident right it's necessary to keep the business going but it's not adding enduring value versus engineering work we're building enduring value we're doing something that's going to make tomorrow better than today for ourselves and for the business right um toil tends to be wrote in repetitive where sometimes it's nice it's nice to kind of zone out and do the same things over and over again but we're not really adding value because what we should be doing is things are creative and iterative we're the most expensive assets that our companies have we should be doing things that move the ball move the ball forward and quite require human ingenuity uh you know toil tends to be tactical engineering work tends to be strategic uh that scale factor i was talking about toil increases with scale um engineering work enables scaling and then toil can be automated probably should have been automated whereas engineering work by definition generally can't be automated so stuff that requires our brain to you know to to do something we want to be doing that right it's better for the business it's better for us it's better for humanity right so what's what's the what's the deal here right the deal here is we've got to be careful about the balance of the toil versus the engineering work right because what does engineer what can you do with engineering work one is what we talked about which is improving the business but the other is um uh we use the engineering work to reduce the toil to fight back against that that toil because otherwise what happens is um what I call engineering bankruptcy is when the toil is at this high level or crowds out our engineering work we not only aren't improving the business but we have no capacity to reduce that toil right so at that point we're in a death spiral right we can't help ourselves um we're just going to be shoveling as fast as we can uh for as long as we can until we fall over because you don't have the engineering capacity to fix whatever it is that's causing that that toil and toil importantly note it's a naturally occurring force right um it's kind of very interesting that you know google and they're borg and you know sort of they are the poster children for the uh the new cloud native world have an entire book about operations and a whole section is about about these ideas of toil because you know you never have unlimited budget and unlimited time so you always sort of start with this kind of um Niall Murphy who's now at runs sre for uh azure talks about this says that you know because of budget constraints and just no one has perfect foresight you always start with a whole set of things in the beginning of the new service you don't have automation for you've got to log in and hack away at something or redeploy something uh kind of you know level two is uh i've got my script that i hacked together right that's in my directory uh that's my externally maintained system specific specific system specific automation then you know there's okay well let's let's share that that that script right or maybe start to put it into uh like a chef or a puppeter or an ansible or something like that that's like my externally maintained um generic automation then i'm starting to ship that stuff with my releases right so it kind of comes like the helper scripts that come in that helper script directory that i can get at with my with my service and then ultimately where i get to i've built it in the system you know i've taught i've written the application to maintain its own database right or i've created an admin panel in the application where people can add their own you know users or partner interfaces or whatever it might be so you're going to go through this cycle you know over and over again uh for each each application each time you're doing this the toil is going to get less and less and less but you're going to be launching new things and it's really this kind of naturally occurring occurring uh cycle that kind of goes over and over and over again so toil is not some thing that we're going to solve once and be uh and be gone with so let's get to the third principle then right so the third principle of uh of sre is sre's have the ability to regulate their workload right this is where it gets it gets crazy for people right so it kind of goes the idea that what if handing off responsibility to sre or ops or whatever you want to call it wasn't a right right that it wasn't guaranteed that you couldn't just say hey i'm done marketing's ready to go uh here you go ops you know it's got to go to production you you uh you take over it and so really it's this idea of that you know we're going to separate the running in production from it being run or owned by sre or operations right and that's usually the point that these people come in or they're like you're nuts right what are you talking about right it's that that can't possibly happen anything that goes into an into production must be owned by by you know our production operation operations team and it's really a pretty dramatic idea but if you look at it you have the right guard rails in place and you think about your organization as dividing those two things up um you'll see more and more organizations saying hey development you run your stuff in production you take care of it right and if you look at whether it's the google model they actually have a safety valve at some point to say hey look you know once it hits this certain threshold of stability and has this certain criteria is met you can hand this off to this other team to this sre team and they will help run it and maintain it for you but before that point uh you know it's actually the development team's responsibility to to care and feed for that service in the production environment now it takes a lot of you know tooling and sort of an enablement and kind of change in mindset but it is a it is a pretty dramatic uh departure forward and one of those core principles of you know kind of what's uh sets an sre uh way of working apart from an operations way of working and there's a great talk steven thorn did at the devops enterprise summit um there's a link there help my slides are already on my my twitter that he goes through kind of these principles in depth um but the question always comes up okay well you know where should i start right what's the practical way to get started with this stuff now this is where i kind of depart from the sort of canonical um sre uh lecture when people say well you know the place to start is number one right that if you have service level objectives you have those consequences you have that shared responsibility between dev and ops and and the business then you have an sre organization right well the problem with that is that the company wide culture changed right and that's really hard takes a long time maybe just come in talking about that they might think you're a little bit you're a little bit crazy um so i'm not discounting that that's where you eventually have to get to but probably not the best place to start likewise with this idea that hey we should separate the going to production from you know turning over the care and feeding our application to to uh to to production to the job operations that's another company wide culture change that's real painful um you know those conversations need need to happen uh but i find time and time again the companies that get started and make progress that actually can be under their own control it's really focusing on this middle one here right reducing toil so one of the things where um you know everybody wins um you know when you when you do it you know who's really against the idea of oh we're going to be more effective and we're going to get more done and we're going to save more time and we're going to feel less out of control right no one really says says no to that so i you know strongly recommend that that is where you want to want to focus is reducing that toil so you have time to make tomorrow better than today so the rest of my talk is going to be about about about just that so why focus on reducing toil number one uh you know it's got lots of value independent of sre you can never say the word sre and people you still you sell the idea of toil and reducing toil and what you're going to get out of it as an organization and what individually individuals are going to get out of it everybody you know there's not many you're not going to find many takers on the other side of that uh of that argument right it's a pretty uh it's a pretty mom and apple pie kind of thing and it's obvious right your people are most expensive assets right so you know stay out of their way right so whether we're in toil kind of comes in two areas right one of it is in you know delivering plan work right so do we have a lot of ticket systems we have to bounce through do we have a lot of repetitive things we have to do over and over again you have to constantly talk the same firewall team to get the next thing you know changed or whatever might happen or from the firewall team is are people day in day out asking me for the next sequence in in ports to sign up another partner or something like that uh so the the plan works side a lot of toil comes into and also in responding to incidents right are we giving people you know the right instrument the instrumentation the right uh you know collaboration or checklist investigatory tools are we empowering them to make decisions pushing control closer to the to the edge and are we empowering them to take uh to take to take action you know in both plan works and respond incidents is all kinds of places we can we can remove toil from the uh from the process but there's a lot that gets in the way um I did this talk at the uh a conference last year the links in the slides there kind of spent like half hour going through all the things that get in the way of traditional operations organizations even when the dev side of the house is you know a cloud native and dev ops friendly you can watch that I think it's a fun talk but the the long and the short of it is silos and queues are the major causes of these dysfunction just dysfunction right so what am I talking about so silos um so people think about silos as oh it's just a team right it's a team comes from the do with teams and it generally doesn't have a lot to do with the organizational structure it's generally more about how a team is is working right so we think about it if everyone's working in kind of one room you know I think we're like a five person startup or whatever it might be uh you know we got the same uh the same backlog the same information or context we're working from the same tools we're using using them the same way which is important uh it's kind of same set of priorities probably the same boss or the same the same backlog but you know that's great and we can work fast we can work furious we can you know get things done very little friction between us but the problem is you know nothing ever lives in isolation in the enterprise you're always going to need something from somebody else right and this is where these silo effects start to take place because you know I'm focused on my work I'm in silo a someone in silo b is focused on their work and I'm pushing work on them right or requests on them or waiting for something from them and all that does is kind of disrupt my flow of work and so I'm going to turn more and more inward I'm going to focus on optimizing what I do I'm the best firewall rule changer west of the Mississippi that is all I'm going to worry about right and if someone comes and tells me that my process slows them down well you know forget them I'm looking at my card it says firewall engineer right I'm I'm focused on on that so you know these silo effects really start to cause these these these disconnects right these in these in these mix mix matches between you know the context people are working in the processes the tooling I mean I'm trying to see people that they're both using a common tool but they're using totally different different ways and a mismatch of capacity you know that my best firewall rule changer in the missus west of Mississippi there's only three of them and there's you know hundred operations people right you know there's a mismatch and maybe you know 500 developers so you know to watch off that so this is a big source of of toil these disease mismatches causes these disconnects and these and these problems and so how we how we cope with this right is we got this this beautiful thing called the ticket queue right or the ticket system and you know the team is we're just going to put this work in there and that's going to batch it up and it's going to send it on to the next person but the reality is we see that you know we know what happens right you you put something in there it sits and it's sit you got to go get a project manager to get somebody to poke that person to tell them to do something then they do it they look at it they don't understand what you're talking about because you're not the firewall rule expert they are right so they send it back to you and say this is no good and you send it back to them they tell you got to wait till next Tuesday right and they do it and they didn't realize it when they did it that they disconnected something else that you also needed and that caused another you know rounding around and around right and sometimes you just say forget it I'll fix it myself on my end other times you got to scrap it and throw it and throw it back but you know these queues are really working through these queues cause all these interruptions and waiting and toil right this is a major source of this toil all that extra work that we have to do is getting in the is getting in the way and you know these ticket queues are also expert we call a snowflake makers right so you know the term snowflake it's yeah it means it's great because I didn't invent it up to someone else did but this idea that you know it's technically acceptable right but it's it's this brittle unreproducible thing right and manually done things a lot of one off when you have these ticket queues style of working the queues don't really learn it's just basically become a bunch of functional silos of experts and you know they're probably good at what they do they get a ticket they parachute in they do something they get the next ticket parachute in do the next thing and you know whatever they're leaving behind it might be technically perfect but it's just a little bit off or a little bit different or the next person's not sure what it is and not not the mixed metaphors here but all those little snowflakes become time bombs that you know that can either interfere with somebody doing something manually making another snowflake or trying to do automation right because only worse than than than automation that doesn't work is automation it's just a little bit a little bit wrong right so you know these these ticket queues become these snowflake blowers right that you know is that's another another major source of of toil so you know how do we how do we get rid of this stuff right you know a few things right so number one we have to actually track the toil levels for each team and I don't mean like pure time tracking type stuff usually seem folks do it they'll use kind of their internal sort of scrum master project management functionality just to interview people or get a sense on a given a week or you know days or whatever might be you know how much time are you spending with this toil and how much time is being spent on engineering work and importantly setting these toil limits for for each team right so being specific about it and saying look you know sort of the canonical out there is to say hey it's 50 percent right that we don't want any of our operations teams spending more than 50 percent of their time on toil this interrupt driven work we want to spend more than 50 percent of their time working on engineering work and by setting those limits and be hard about them we can actually start to fund those efforts where the you know those efforts to reduce the toil right so it seems very you know obvious but surprised how many few organizations are trying to go down this path without actually starting with let's actually track the toil level set those toil limits and then fund these efforts now you know the first obvious one is well let's just get rid of it right so let's get rid of the excessive dependence on the ticket queues let's get to solve these problems in our applications let's just you know stop bouncing things around you know it comes through our processes a refactoring our apps our tools our processes that's a that's a it's one way to go it's a good way to go and you can kind of kind of catch it as you as you as you can but often you're depending on other people in that scenario it's a little hard to get started there so it's good to focus on that but the one I want to talk more about is when kind of near and dear to my heart is the idea of applying a self-service design pattern right so get out of this idea that you're waiting for somebody to do something or someone's waiting for you to do something or you're constantly being interrupted by those things when people are waiting for you and let's get into you know pushing self-service right so and you know people often approach the self-service question and say well where do I start right and the most powerful way I've seen and guys Scott Peru at a company called CSGI it's one of the biggest outsourced building companies in the world I should have a link to him was talking about this recently that they kind of made a 20% time project they just said let's empower the teams to find and fix all these little things that they can and turn on self-service for other people in the organization so just look at it look at your ticket queues that you're constantly interacting with and find out what are all the little things that we can start to start to chip chip away at you do that at scale you know they've kind of kind of massive efficiency improvements so you know some examples like you know they do this do it again and then do it again right those the obvious ones the repetitive requests you're incurring waiting on the requesting side you're pushing interruptions on to people on the other side the whole time they're they're you know kind of filling those repetitive requests they're ignoring the other work they should be doing a big source of toil in most organizations and often it's a desk by a thousand cuts it's never like one particular thing it's just all these little repetitive things right so you know looking to put self-service in place you have the experts set up those those procedures you know and then what's effectively happening is is those people are happy you know kind of the Tom Sawyer idea they're happy painting your fence for you because they get that fast feedback right and they they get to do what they need to be done and you can focus on the things that are moving the business business forward like getting rid of a toil you know the classic one is always will access control the big problem I could do that but I can't get to it that's got customer data there so I can't I can't restart that service even though I know it craps out all the time right so I now have to push work on to this other you know part of the organization to do it for me more toil one of the kind of cool things I've seen in big companies with this self-service idea is that they can actually hand out access to privileged environments to people traditionally didn't have access to those privileged environments right letting developers to restart in production letting a low test team you know refresh and reconfigure environments that have sensitive customer data into in it right letting people do database maintenance or you know schema updates that traditionally weren't allowed to allowed to do that you know making firewall rule changes for repetitive things the like partner onboarding or spinning up new environments not to set up new new firewall routes you know instead of having say I have one trusted team that could do that I can now I can now hand that off to other folks and distribute control and that actually passes muster with the audit crowd they actually prefer it because they say oh this people have limited access it's not like the traditional way of saying well you know here's a shell script and your SSH key and your pseudo privileges and you know good luck right it's more like saying here we've set up this self-service route for you you can these guardrails are in place for you great way to cut out that toil a fun one we've seen this time and time again see I'm an expert I don't read that wiki right so you know the service has changed right because I mentioned people are changing things right and they put in the dock they wrote in the wiki right they said you know use this flag or bad things will happen right or hey you know the knocks like can you pause that monitoring because every time you do something like it wakes all this up and it takes us 20 minutes to figure out that it's just you you know doing a restart right because you know what always happens is then that person rolls out of bed three in the morning I've been here for seven years I know what I'm doing I'm not going to read a wiki to do like a restart on a service and you know boom the whole thing uh the whole thing blows up right and then I think about if looking at from the self-service perspective is that you have the team collaborating on those jobs right so it's basically like the the the documentation is the code so so to speak right and so when these inevitable changes come up these little things that need to happen um by putting them into that self-service job when that person rolls out of bed at three in the morning where you roll at bed at three in the morning to say you know what was that firewall rule team told me last Thursday right you don't have to remember that it's in the self-service job we know what buttons that we know what buttons to hit and or what command line to hit and you know we're off and off and running and kind of last one here which is another kind of anti-pattern um but that's pretty prevalent which is this idea that you know dev work is more expensive than ops work right ah developers they're hot shots right ops folks uh you know they make a lot of noise they'll be fine right so um you know oh I didn't change the slide that's why so you know you see happens is this kind of time and time again we get these problems someone goes ah service problem oh I know this one you fix it you move on right sure enough you know week later month later day later whatever it is same problem happens you go you fix it you move on and eventually the operation side goes well hey can we fix this come on you know and I've seen this more times than not sometimes it's just passive aggressive dev side just closes this closes the bug closed closed sometimes they'll give an answer and say oh it's not in the budget right and besides you have a workaround so don't come to us with your problem you can you can fix it uh we actually had a um customer bars was funny their first run deck job they ever put together was because they had a it was insurance like a mid-level insurance company and they had they did this specialty underwriting where to underwrite a policy like five or six people had to log into a web app and they had to do their underwriting their underwriting stuff and there was some bug in the code that fd a person got a cup of coffee in their web session timed out or they closed their browser and they didn't hit that release policy button the policy was locked at the database record level and so in general when they found this out was when some high value you know mr big shot owns 19 buildings that's coming into the insurance broker's office to talk about their policies and the broker can't get into the policy because the thing's locked and everybody freaks out the entire customer service center is getting is getting yelled at and they were literally they had to had to raise a p1 bug against the dba team uh to actually go and fix that this has happened 30 to 40 times a month right this this would happen because there's hundreds of people involved underwriting process and they literally kept telling the operations team you know sorry there's no budget to fix that we're going to be replacing this system you know often that happens right within the next 18 months or two years or whatever and you know of course it never happened and they're like this is this is insane and it kind of went up to fight up to the up to the cio and the cio actually um you know uh sided with the development headed development at the same time there was a meeting the very next day where he was yelling out the operations team for not being able to hit all of their all of their uh all their milestones to improve the service and this is a great example of sort of you know how things ran in the organization so just by creating a self-service job to let those customer service agents unlock the record themselves a little power shell script that they that they ran to uh to do it uh you know they basically were saving 30 to 40 uh p1 dba bugs and the whole fire drill uh per month right so you know well yes it's the best way to do it is to you know to be able to to say hey let's fix these problems i think that is again like kind of like those um other longer conversations like i you know i mentioned earlier this idea that hey you know this operations work matters this is our business this is what we do and you know springing services or keep us in business and we have to prioritize that and and you know looking at the true cost of this stuff um in the meantime you can throw some self-service in there and you can kind of get rid of a lot of these repetitive uh repetitive uh problems so you know this pattern i'm talking about i mean this is something that we've seen uh from different folks in the uh in the industry i know that the uh not not any one company you know the disney folks this morning we're talking a lot about empowering self-service i mentioned scott prue um some other folks i'm gonna talk about later that that have done this sort of thing but and we're trying to capture this idea but what's kind of shaping up to be is that we have these uh consumer somebody needs an operations capability right so they have to be on the dev you know outside of operations could be within operations and we have this expertise um in the house and the idea is find a way to let the experts create the standard operating procedures that we can then let the other folks uh you know need as as needed and the key kind of ideas see key patterns to see over and over again number one uh that's pull based right that it's not um you know i'm asking somebody to inform me it's i can use it as i need on demand uh number two we see this idea that you know people try to solve this problem by saying oh we're gonna pick the next great tool right like last year well this year it's kubernetes right this brings change everything last year is ansible before that it was chef before that was puppet before that it was blade logic before that i was cf engine and now when you go into the company is they have everything right so the idea of one tool to rule them all just never really never shapes up right and frankly it's kind of an anti lean or a sort of an anti pattern to um to try to push tools on other people in other groups it's better to let them as much as you can pick the things they want to work with if that's the power shell team that's awesome if they're an ansible team you know or maybe a pure python that's great right if those people love shell scripts over there that's all they know that's fantastic too let them use the tools that that that work for them and focus on encapsulating them and creating this this uh this cell service and in doing that you know it's really because they're not focusing on sort of forcing everybody into one um language it lets you kind of define sort of a meta layer around it and we see people are really focusing on setting up these you know idea of guardrails right how do we make that these jobs safer and safer to run because that's going to the more the safer I can make this job to run whether it's making it you know item potent meaning that you know you can't start you know 35 press the button 35 times you only get one service running not you know 35 right um or uh you know allowing sort of allowed values right that other experts in a particular area they might know the right incantations and the right flags that flag but other folks don't so guide them in the way of of doing that uh another key uh principally see time and time again is that self-service isn't just the ability to push a button that's kind of like a rigid self-service but true self-service is more like the aws style of self-service right so think about like savvice or these old sort of horizon business hosting before they had a self-service portal right but you hit that button and it kind of fired off a static process to go tell people you know do something from this this catalog versus look at the aws model right which is like hey we're going to give you these guardrails but you can define your own button you define your own images you define your own you know configurations that you want to do and we're going to kind of put you in these guardrails and put you in this kind of you know framework so it's standardized but you can do what you need to do inside of that and we see the people are really successful with self-service are the ones that are really driving that idea of letting people define the procedures that they need to go and run so it's not just a one way a one way street and really critical is to focus on building security and compliance in right so access control is a must but also you know you see folks teaming up with the compliance folks to say hey this is actually a better system of record right because our old way of kind of you know our ticket-driven the theater of compliance was we had all this rowdy and all these approvals and all these you know kind of itals and templates and then at the end it came down to somebody who's going to do work and it's bob again and bob just says done right like well what happened like do we know what happened was that really bob that did it like what did bob do did bob even do anything you know did bob run bob script version one or bob scripts version seven you know maybe bob ran jane script right nobody nobody knows so the idea of being able to actually record things through this kind of this this kind of self-service capability actually is a great way to get funding and it gets compliance on to your side and we'll ask a couple things here you know it's interesting you see people who have taken this self-service idea on their own and they've kind of used as a foundation for more strategic initiative so hometown heroes here ticket master jody mulkey was their former cto now he's over at aspiration the big unicorn bank but they had this problem where a very strategic which was the average mptr for this is back in the 2014 time frame the average mptr for web-facing outages like yankees can't print playoff tickets or you can't buy your adult tickets was 47 minutes right and for a company that was already getting kind of beat up by the by the world that's rough right because that's not tech crunch news that's like you know new york times news when when something goes something goes wrong so the idea they had is well hey look you know instead of do this stuff and throw it over the wall and then they threw it over the wall they had this long escalation chain before it came back to the people who knew anything about it right so they had this knock and the knock was like a bunch of escalate they're the escalators right they looked at the screen they you know they would call people and escalation chain and go up a lot of time was being was being spent on that and then they had all the context as they're seeing things happen yet the people they're waking up and calling had none of the context and now they got to get involved is a big problem so what they did is they started working on the idea of saying well let's hand off these procedures to the knock and turn them back into operators right in fact let's test those operational procedures all throughout our process so finally when it went to when it went kind of into operations we can empower this operations team to actually take charge or we're pushing control to them they can solve these problems up front eventually I think they've got to the point where they've even gotten rid of the knock all together they're kind of pushing that control all the way back to those development teams but in this context in about 18 months or so they went from the average MCTR being 47 minutes down to 3.8 minutes right it's like in 90 something percent reduction or something like that I can't really do the math but you know dramatic all from just saying hey let's empower the people closest to the problem to solve the problem and those people that have to get involved that we have to escalate to let's get them quick self-service access to production environments as well so taking the idea of self-service and turning into a really strategic initiative you know MCTR reduced by 92 percent reduced escalations by 50 percent that made everybody super happy right they didn't have to be woken up you know half they're woken up half as many times right and the overall support cost dropped by about half as well so huge a huge win I said I think they've gone you know a lot farther since then this kind of foundational thinking has also really made their Kubernetes introduction the last few years I think go go quite quite well it's a cool story one more story that's Sean Norris a standard charter bank they're a huge bank that no one's ever heard of they're based in Singapore and London and they have branches pretty much everywhere except for the US and and the UK so all over Africa all over Asia a lot of emerging markets so they have you know 86,000 employees in 60 plus countries so that means there's 60 different regulatory regimes they have to they have to adhere to so you know a a systems administrator in Malaysia can't access someone's data in Singapore I mean it's that kind of crazy crazy stuff right and so because of this and I think like Queen Victoria signed their banking charter like 300 years ago it's nuts and so everything their bank was built for compliance right so they had this problem which is nothing got done because it's a bunch of tickets a bunch of manual work a bunch of one-offs because of all those different regulatory regimes and all those points that couldn't that couldn't connect so their whole point of rolling out the self-service was just just strategically number one to have a common way of of doing things right so just they can stop all this or the manual toil I think they had like five or six thousand systems administrators kind of doing things doing things manually but more importantly it was to really be able to distribute that workload across their organization and allow people to solve problems for themselves without having to escalate put into the ticket black hole gets routed you know all over different countries different teams before they can get somebody involved it's a really compelling story as talk it's a it's a lot more than just this but it talks about in just a year's worth of time they save 28 person years of time just with trying to roll this roll this out 13,000 operations tasks and privileged environments that didn't require review oh this is this is super fun so they had this deal because of the banking regulation that they had to prove that people who did anything in a production environment that they did what they said they did right so they couldn't figure out how to do this so what so what the regular regulators made them do is the lowest common denominator across all these different countries is if you had to do something in production and you're already in background checked and everything you had to go to this special ticket system you had to request a production access token you got that token then you had to log in to this kind of special shell that they have put your token that token in there that'll turn on the screen recording the keystroke logging screen recording software that will actually record everything you do and at the end of your session it will then email a notification to your direct boss the video is ready for review and they have to watch the entire video and certify under power of like you know their job that you didn't do anything that was untoward right so that happened 13 they avoided 13,000 video watching sessions in one year just from going down this service path yeah it's pretty miserable but that's life right and you know they had 200 less customer impacting outages right so their biggest problem is not you know random things breaking because it never does is all self-inflicted right so they were to cut down on 200 those you know 200 less gunshots to the foot uh during uh during that during that year so again little kernel of self-service that started just like the ticketmaster when i just mentioned was a guy named mark mon that actually started kind of as a semi-skunkworks project to build self-service capabilities got leveraged by the cto into great things same thing at uh shaw Norris the svp big big you know above all the uh the organization saw some a few people that kind of just dissimilar wasn't really a skunkwork so it was you know they got approval for it but went through the pain themselves to set up self-service for their own teams to do these things got picked up and spread throughout the organization and you know now they're all they're all heroes so um where could you guys help uh run that we're trying to document this pattern we're trying to talk to different people if you your company if you're um you know moving to more an sre driven self-service way of working and i'd love to talk to you about it um we're trying to kind of kind of catalog these ideas turn them into sort of resources for uh for the organization because i feel like there's a gap in the world there's a gap between the people that the organizations that grew up with an sre cloud native point of view and those that have the history of being around for you know 10 20 years and kind of came from a different dna how they're moving towards that so if you want to talk about it i'd love to talk about with you guys um rundoc.com slash self-service is where you can see this it's an open book project um but love uh feedback on it and love to talk to you about it so recap um sre is more than just a title right gotta focus on those those principles focus on you know changing how operations work it's better for us it's better for everybody around us it's better for our companies um you know things like air budgets and toil limits are really kind of what a different way of thinking sets us apart um i'd say be practical you know start focusing on the toil first it's the it it helps people's lives individually and it kind of shows the you know the the business that we're working on something new that actually has delivers immediate value find a fix those anti patterns it's the best way to do it empower people to get going and solve the little things that get in their way look in their ticket queues figure out which are their petitive things they can start to uh to tackle and then apply that uh self-service pattern wherever you can't um refactor things and you know get rid of those handoffs all together so that's my uh talk i'm damon edwards on twitter the slides are already up there uh email anytime and um i'll take any questions or comments maybe i'll take it tickets and uh you know the comments that maybe uh there are a lot of overheads that people have worked work with because uh maybe people don't there's there's no um recognition of what went underneath to actually solve the problems we're talking about yeah firewall um what we did at my organization is we implemented a system uh first of all it involved uh smart commits where we required uh to have uh ticket numbers and every kind of commit to a code version of this and we used this bucket server uh we used rundep to automate some tasks and through a combination of python shell rundep and smart ticketing uh we were able to provide automation so we actually required people to put a ticket number into a system before something like a tagging operation took place and so it became self-documenting right right we would publish the results of the operation and people could see what signs of repo got changed or altered yeah during the tagging process sure so that i i wanted to mention that my first experience is i think it's a key to effectively leveraging a ticketing system um it's actually required people to document things and it's so you can actually use the ticketing system properly right and that's probably a policy yeah and and my my beef with tickets aren't tickets my beef is with the ticket queue right it's people that that force their work through it through it through a queue you know they break up the work into little functional pieces and it sits in people's queues it's more the queue is the problem the ticket as a record i think is actually a great thing and so we see people doing that stuff all the time which is i'm still using the ticket as a system of record but i'm not actually put sitting in a queue anymore i'm empowering people with self-service and i'm just using the ticket as a way of documenting it now we might argue that well that's kind of an odd thing if you had to start from scratch why don't you use a database like why are we you know using a ticket system but if that's already a fact of life in an organization we see that as being a positive thing so ticket as a record is a good idea having records the ticket queue to drive people's work is where things start to go to go wrong right so self-service is that key thing of it's not it's getting rid of pushing work onto other people it's getting rid of of the i'm injecting waiting on my end and interruptions on their end um if i'm going to do it myself and there's a ticket involved me doing it myself then that's great if i'm going to open up a ticket to make someone else as a work order to make someone else do work for me that's when the organizational toxicity starts starts to pile up we see that a lot we see folks automating in self-service creation of tickets like oh i just know because of regulation i've got to have these 30 tickets to do it stand up a new environment so i'm just going to automate those 30 tickets right yeah so i don't want to monopolize the time just one one quick comment sure when you talk about the fr e uh goals objective i think you had something like time to improve in the middle um control of your workload yeah also i think that what developers are sending over controlling that they're actually sending over the right amount of work they can i feel they're calling that service only contract and service level agreement and in my experience is when you start talking about sometimes talking about that sometimes you'll get up for management to agree to actually go with the the middle piece which is give people time to improve the system yeah well one comment i'll say about the itel thing that's very interesting and if you look at devops plus sre it actually forms a self-regulating system their feedback mechanism to make a self-regulating system versus the itel model which is it's kind of a top-down hierarchical control there's just some body above me that's going to improve my work and it's going to be kind of driven through these different formalized processes versus you see the high flying uh or the faster moving sre plus devops organizations they're horizontally self-regulating like they put these shared responsibilities together to where they control the system themselves because of that shared responsibility they've created between development business and operations they don't need all that overhead they don't need all of that the cab to tell them what or what not to do because they've built the feedback mechanisms in the system themselves so i'm not getting kind of off in the weeds here but it's a very interesting idea that i think that way of working with fundamentally breaks in the itel world is the hierarchical command and control and and we're going to catch inspection from some external party like the cab i just don't see that that hasn't worked yet and you see the sre plus devops way they build the self-regulating system in to the into how people work it's a much more effective way to go about things that's a whole different that's a whole different talk thanks anybody uh so if your if your team has no read-up on tools basically yeah so like basically everyone can use whatever language they want whatever you know ansible puppet shop whatever they want as long as they get the job done yeah um how do you how do you encounter that you know since like um i'm i'm i'm pulling into that it's basically like how do you go from the situation you have all these different tools everybody's been using their own thing the side of the sort of natural reform yeah how do you combat that without you know taking everybody off by saying you have to use this tool sure i think you focus on the integration points and you focus on having the standard toolchain architecture so we see it so the easy example would be a deployment pipeline right you know one team can use bamboo another team can use Jenkins but if you guys get together as an organization and say this is what this is this is what a ci server is this is what a a source repository is this is how we're going to divide up our source our source code this is how we're going to um you know are we going to deploy off of trunk are we going to have long live branches right these these kind of meta decisions that once you make them as an organization you can use artifactory or you could use nexus or you can use github and you can use someone can be using you know uh spn right it's not it's going to matter a lot less if you focus on the toolchain architecture so we have the same picture and then focus on the integration points right so if i have to actually expect you to to work on my puppet manifest vice versa then yeah that's a problem we have to work we have to work that out and come to some standardization of skills but if we're able to encapsulate things in a toolchain capacity and say hey i've got this higher level self-service and it doesn't matter whether i'm calling a shell script or i'm calling some other you know homegrown you know full-fledged you know java-based uh uh management tool it doesn't matter because i'm thinking about it from a that's my modular automation and this is my standard tool for for calling that right so it's more of a loosely coupled toolchain idea that i see working working well it's not just saying anybody can do whatever they want right there still has to kind of be that that cohesive pattern i hope that answered the question sort of we talk about that afterwards they're like i can give you a rendic example but i don't want to shell sell the rendic thing so i'll tell you i'll talk about it the idea of having a framework that you can encapsulate those things right so that's that's the idea and then i'm interfacing with that framework i'm not interfacing with your python scripts right um that's kind of the idea but if we have to share that fight we have to share work on that script then yes then we need to figure out yeah they might figure that out yeah you can't just say everybody can do everything that they that they want comes more attractive the process and then you can use that as a carrot and incentivize people like you can keep managing that crappy little bash tool over there or you could just take five minutes and adopt the better tools and ultimately there's a lot of organizational change that just comes with culture change and that little bit of influence about influencing culture by just you know make your tool if not be better just look better it will get more people starting to use it and then that will help drive conversion at least that's in my experience so i hope the microphone got that no worries so uh hopefully a quick question sure no i think s re would be like a subset of the broader devops conversation i mean devops is like really an umbrella right it's a whole bunch of kind of it's a bunch of ideas aligned toward sort of a set of problem statements it's not really like a specific methodology but i think if you take traditional operations work and you apply devops thinking to it that's when you get s re which is which is a great which is a specific thing right it's a team it's a way it's a you know it's a set of principles it's a set of kind of you know skills um so i think you know they're both lean you know like lean manufacturing that sense they're both lean to start and then devops i think is a more overarching idea and i would look at s re as a instantiation of devops specific how do we do operations work um because that's not going away right operations work not going away and i think some people call it service reliability engineering which is probably a better name than site reliability engineering but i think we're kind of stuck with uh with that okay laid on me no um so i wonder if you were mentioning about integration points between the devops team the s re team would you say that that a possible good integration point would be the artifact repository meaning devops job is kind of gets your artifact to the repository and then s re's job is taking to make sure it's reliable in production i mean i think there's different organizational models so i know there's like the the google model where there is a separate development team that has s re's embedded in it the full functional unit that builds and really deploys the begin application right full stack right into their you know board system but there's a divide between ops and dev right different teams and they kind of they do work differently um then you look like the netflix model right their s re's are really more like incident commanders because there is no centralized operations operations is inherent to these cross functional teams everybody is their own kind of autonomous team and you build it you run it you own it right there is no orphans in that in that world both achieving high velocity results and the same shared responsibility model one does it by putting everybody in the same team the other does it by building things like slo's and and and air budgets and toil limits to make itself regulating so they can build that shared responsibility so i think the org model matters a lot less than the way of working and the only complaint i would have with what you just said is that um the artifact repository a complaint a yes and here make sure the artifact repository is the full stack right that it's what containers are so great right that's oh it's not just here's my war file it's here's here's the container everything's everything's running in here's the data i mean it's like it's really almost like the environment i want to hand off an environment to that team rather than i'm going to hand off artifacts that they're going to have to then kind of reassemble on the other side now maybe the tools automatically we'll build it for them and then that's fine too right give you artifacts plus some kind of build manifest that's going to go into something and produce that environment for me but the full stack should be built and run by one side and then built and then run by the other the other side now whether that side is inside your team or that side is some other organization is totally up to kind of i think the dna of how your organization works because i look at the netflix model and the google model is perfect examples of they both seem to do a pretty darn good job at least themselves and um totally different organization structure anybody else think we're out of time i'll be around if you want to talk thanks so much appreciate it it's fun to be back at scale yeah thanks for the introduction so yeah my name is mike ausendorf from barrios barrios stands stands for backup archiving recovery open source so some people who call it bros but that's not the way it's pronounced it's it's barrios and stands for backup archiving recovery and open source once again so today we want to talk about resilience and disaster recovery in times of ransomware and other threats we will elaborate why it's especially crucial to have open source software when making backups if you think about data sovereignty that's the crucial point and then we will i will give you an overview about barrios what it does and how it works so i hope nobody of you have seen a screen like this on your computer you can kick calm down it's just a screenshot so it's not actually happening here if this happens to you you have hopefully two options one option is to pay the fee to release your data and the other option is you have a backup and can just restore your data and you don't need to pay the fee so and that is what we are talking about we're not talking about that so ransomware is just one possible threat for your data of course other things might may happen like physical damage software failure user failure inclusion all kind of things and of course there's always the unknown the the unexpected that may happen and in all those cases the only thing that can solve you is to have a backup and let you restore your data um so the title says last line of defense that means um there are things our things are very important to avoid any kind of such situations so you probably have few DMs that your firewalls your virus can ask you have intrusion detection systems hopefully training to prevent from social engineering that's all important but that's scope for other talks our subject here is resilience and disaster recovery and um so I like uh Star Trek and here's one quote from Commander William T. Ryker our daily routine is the unexpected and that's what backup is all about some unexpected anyone knows what this picture is about it has to do with a backup but in another context so um that's the small world global seed world I find that in in Norway and it was designed to hold a lot of seed types in order to ensure nutrition after a big catastrophe so um this world holds more than 800,000 different seed types it was funded by the Norwegian government and it was used much sooner than the people who constructed it expected and 2015 a research center in Syria was um near the civil war region and they decided to close that um research center they brought their stuff over to Svalbard and then later on opened another location and retrieved the the lentil seeds they were working with so that means you don't have to make backup of your data but also about other assets and important things and the lesson learned is sometimes it happens much sooner than you expect that you need your backup so back to our subject um some general guidelines if you talk about backup so if you make a copy of your data to a disk somewhere on your network then it's wise to have it separated from the rest of your network so means have a dedicated backup server and make sure that nobody can access the data otherwise if you have gotten untuned they might want to try to destroy your backup as well and the second point is um make your application backup your backup that means it's always good to have more than one backup so what most people do is they start with a backup to disk and then make copy of those backup data to cloud storage or to tape media so that you have two different kind of technologies where you store your backups if you backup to cloud of course you should consider um encryption or in other words encryption is mandatory if you hand over your your data especially your backup data to someone else because cloud cloud is just a synonym for someone else's computer so make sure that it's um encrypted and then the last point is um some people start to make backups and after a while they find out oh my i run out of this space because they haven't calculated how much space they need if they start backup so if you do regular full backups then um calculate what space you need otherwise you get into problems soon and um more people than you expect get this kind of problem so that's um from the from the field these uh this guideline so if talking about your end somewhere some other things needs to be considered um i said before if you backup to disk make sure that it's uh separated from the rest of your network only accessible by the special backup protocol and even that might not be enough if an intruder can imagine can um can act in a way that he can control your backup software he might also get into a position to destroy your backup data even it's only accessible by the specified protocol so one really good advice is to have read only medium like warm tapes write once reach many tapes so that is really on a physical way protected against unwanted overwriting of your backup data and then again if you're using backup data encryption you usually need the key to restore the data again so make an extra copy of that key because if you lose that key or if someone manages to encrypt your decryption key you will be in the situation that you have backup data but you can't use it because you can't um decrypt it um some things about long-term availability if you think about having backup as a kind of archive a lot of companies need to store or be able to recover data for a period of 10 years or even more then you have to think this time span into the future and the means of technology 10 years is quite a long time so will your backup software be available and will it run on future hardware you have to think about it how you can ensure that will hardware available in the future be able to read your media so imagine you have a new pc in a couple of years and you have a tape drive and then you have to make sure that the hardware you have in the future will work with the medium you now have so think about that and then if you think about the software you're using avoid vendor lock-in and there are separate or different kinds of vendor lock-in one is for example a paper used when you need to restore so there's a popular cloud service where you can quite cheap store your data and then you might get a surprise when you need it back because retrieving the data is what costs and of course the vendor knows what he's doing because if you need your data back you usually have no choice but to pay I think it's a business model that's okay you can do that but if you use it you have to know what you have to pay when you get your data back so should be somewhere in your in your budget and then if you use proprietary software or software that has restriction usage restrictions and these are two examples from the field it's a backup software that needs a license key to work and if the license key expires the vendor will tell you you can use your software in the future until the next reboot because after the next reboot or after you restart the backup software it will try to validate the license key and if you don't pay the subscription fee then you won't be able to use it and that's of course not what you want because then you don't have the sovereignty over your data so that's bad another example from the field is after subscription ends you are obligated to delete the software the backup software and of course without backup software you can't restore so that's not not good to have and then think about what happens if the vendor of a backup software goes out of market so over 10 years that can always happen and so the answer to all these questions is open source software is crucial for backups so one thing is you really should distinguish between real open source and open core open core is in the end not different than proprietary software just it may have some parts of it being open source but to be fully operated or fully operable for you to restore your data you probably will need parts of the proprietary software so that's nothing else than proprietary and when it comes to the to the effect so with real open source you have no render lock-in at all even if companies making a project disappear the code is still available and can be adapted the worst case is that you have to pay developer to do the work for you but you are in the position to theoretically can do that without having open source software it's just not possible um future proof and adaptable to future hardware can only be done with open source so if you're running a backup software fully open source on linux as complete open source operating system you have the full stack as open source and so you will always be in the position to adapt the source to the needs you have so and especially if you use cloud services for backup it's important that the service provider uses open source because otherwise you won't be able to retrieve your data when you decide to move away from that cloud service so always have a look that it's um that the software that a cloud provider uses is compatible with an open source solution so requirement summary backup software only future proof is 100% open source if you want to be prepared against ransomware and other unexpected things keep extra copies of your encryption keys separate your backup data use backup replication backup your backup if possible use warm media and a last point it's always good if your backup data is easily accessible in case of a complete crash even of your backup server um it's it's good to start restoring your backup server but that might take a long time until you get everything running up and working and it's good that you if you can directly access the backup data to retrieve the stuff that's in there without having first to install a backup server again so any questions until until this part yeah okay sorry uh yes the question was if we're getting into a cloud we will see that later what kind of searches we we're going to support yeah okay so this part was more the general part and I would now switch to the uh barrios part so I said before stands for backup archive recovery open sourced we started as a fork of the betular dot org project one of our co-founders is microphone veering he used to work in the betular project as a voluntary developer for a long time and he started back in 2010 to collect patches that didn't make it into betular mainstream and around 2012 he decided that we would make an own fork based on that work already done um why did we decide that we wanted to implement our ideas we wanted to be a little bit faster than the the kid our project was at that time and we wanted to make sure that it stays open source so we didn't want to make things like open core or so we really wanted to be open source so and then we released the first version in 2013 and since then it's uh yeah yearly a new major version the current release is 18.2 we will see a little bit more about that on the following slides so it's all about data sovereignty no vendor lock-in barrios is open source licenses a fair new public a fair new public license agpl we did a lot of code cleanup and factoring um developed a lot of new features over time one thing or one of the first things was a python plugin interface which made it a lot of lot easier to implement plugins for example to backup databases and stuff like that we see a growing open source community around barrios um we have an open storage format so it's derived from the back killer format and it's completely documented and um what we also have is we have command line tools which makes it really easy to extract data from tapes or other backup media without having to set up a complete barrios server before so in case of a disaster recovery this helps you to quickly get up again in an operating mode so here's a little overview on the left column you'll see the client side that is from what we can take data to backup so we have a windows client a linux unix clients um max client a bsd is not on the slide but also bsd client we have a plug-in that makes the mvf snapshots and incrementally backup those snapshots using cbt change block tracking and we have plugins that can um backup gluster and cest cloud storage is on a on the block level now we have plugins to do incremental backup of microsoft sql server and do point-in-time recovery we have a similar plugin for mysql other databases like postgres we backup by making dumps so there's no specific plugin but for those two we have point-in-time recovery and ndmp is a network data management protocol used by vendors like netapp or emc and we support this protocol to directly fetch data from storages which is done pretty fast so then in the in the central column we have the barrios server that is where we control everything we have schedules that say when do we run backup jobs like for example a full backup every month and an incremental every day and there you configure what to backup from which client when and to what destination and that's the right column that is where we're going to write the backup data so the easiest way is this storage you just have enough this space on your barrios server you can directly write the backup data there that is what most people do in the first place and then of course we support tapes and tapes libraries and different kind of cloud storages so glaster and sef is named here meanwhile we also accept we also support s3 interfaces so we can write to amazon if you want or in this case sef for example has sometimes s3 interfaces so we can use it as well okay question yes we have we am there we have a dedicated plug-in hyper we not yet yeah we support s3 interface from from amazon yeah that's basically what we support those two and s3 is not on that picture yet okay are you using a barrios with s3 or i was google cloud okay all right would be interesting how you do this okay very good okay yeah and then we also see in the middle column the central column we have a web interface with a file browser so there you can select the client and then see what's in the backup mark directories or single files and then do the restore and of course we also have a text console a command line interface to do that to select files for restore and we have a scripting interface so you can automate it or include it in other frameworks you might have like a web front and so so there are some providers internet providers or better say managed service providers who use these kind of interfaces we have may have integrated solutions and run backup service platforms as managed services based on barrios so that's that's also possible as an overview we don't have enough time to to talk about all features that we support but all common sense features that you usually expect from a network backup system like multi-platform we have binaries for linux unix windows macOS scheduler supporting full differential incremental backups virtual full means that if you have really large data sets and you don't want to make a full backup every month you can do what we call full or virtual full backups and that means we look at all the data we have in our backup system and rebuild a new full backup out of the existing maybe two months or three months old full backup and all the incrementals we've made in between and we use this information to rebuild a new full backup so why is that interesting um if you're running full backup and then incrementals for a long time and then suddenly you have to do a restore that will be very long to run because then at that point the backup software would have to reconstruct the the file set of the originating full backup and taking in consideration all the incremental sets you've done over time and that could be 100 or so and that would be very slow so it's wise that at a certain point you take the existing backups and reconstruct a new virtual full which makes it much faster if it comes to a restore situation then accurate means that you keep track of deleted files so sometimes if you do a restore you don't want old files for example spooling files or log files or whatever so if it's important for you that you keep track of that you we have seen the so-called accurate feature then what we call a catalog is a database holding all the meta information like what file with which checksum which attributes from which computer from which date is on which backup medium so that is the the main information if you want to restore a particular set of data and that is usually usually my my sequel or postgres and those catalogs can get very big so if you have like a billion files in your backup then the catalog can grow up to a terabyte or so and that's the reason why we recommend postgres here because we realize that it's more performance and better tunable for for large environments yeah and then of course we support encryption in several ways we can encrypt the backup data directly on the client with individual keys per client we can use global clients global keys and we can encrypt on tape drives that support hardware encryption that is of course the fastest way and then baryos will take care of the keys you have will store the keys in the database so that when it comes to restore it knows what tape needs what which key and will automatically handle all the key stuff um yeah i said before as as medium backup media we support this tape library and cloud storages and what's particular of interest we have many scripting interfaces the python plugin interface is one an easier one is the pre and post jobs that you can easily configure with your jobs that certain scripts run on the central server of the barrier server or alternatively on the client where you do the backup so that is really flexible and i guess most of the task can be done with with those kind of scripts so some new features that we invented over time or that we developed over time um one i've already named is the hardware encryption support on lto drives um another is client quota support so that you can limit the amount of data a certain client can send into your backup and uh ventric limitation that is particularly of interest if you have a network uh kind of limitations and then you might want um your backup not to eat up all the uh ventric that you have available um yeah indian p is interesting if you have large storage error networks using indian p protocol like net app or emc um then we implemented replication to other backup sites so that you can have your backup in one data center and then make a migration or copy job to another data center and the barrier server will keep track of where you have your data available yeah the cloud server just already mentioned the python plug-in interface and we have a multilingual and multi-tenant web ui um supporting acl access control lists um that means you can design it in a way that you have a self-service portal for your users so example given if you have a dedicated computer for one person then you can grant access to backup data for this particular computer to one particular person or if you are a larger organization and have like it departments then you can grant access to a group of administrators for all computers belonging to that department for example so that's working pretty well and i think we have now like 10 languages um in the web ui most of it coming from the community so like chinese i don't speak chinese nor do my colleagues so someone committed it and that's that's growing the base so a few words about the current release um what we've done there and that was a lot more work than we expected um we implemented a way that transport encryption is preconfigured and working out of the box by default so before the 18.2 release if you wanted to have transport encryption you had to create individual certificates for each client you had to configure everything you had to roll out and you had to make sure that it really was encrypted because there was no log message or so indicating that encryption worked or not um and um if you think back to the one of the first slides with the architecture diagram uh you saw the connections from the client to the director and we do this by authentication by password by shared secrets um all over the time and at a certain point we thought well we already have this um shared secret passwords why not use it um to enable tls transcription by default and so that's one of the major new things in the 18.2 release um and it took a lot of time to make sure that it works with all the clients not supporting this kind of encryption um that's the reason why the release came a little bit later than uh than announced yeah but now this works pretty good um we use kind of header inspection that were kind of pre-header inspection that means if a connection is established we see how it's um it's encrypted or not um and then we started if it's an older client without encryption or with a newer than automatic encryption and of course the the old way with individual tls certificates is still supported uh because it's um a little bit or much more secure um to use that way uh but i personally do not know anybody who has really configured that way so uh but of course we didn't want to shut down the this way which is still a little bit more or more secure and then we also introduced uh PEM authentication so and that's in conjunction with the um tls by default is a good way um to use um existing user authentication methods like PEM unix or you can use your unix password or you can connect it to um an adapter directory for authentication which is also um interesting way for for large organizations and then we did some internal work which makes life for us a lot easier so uh who of you knows what autocons is so automate and and yeah okay some some knows that's really a nightmare if you have to make changes to it so and the the old autocons system which was only used to prepare the source code to be compiled uh alone contains 76k lines just autoconscript and then we replace this by the more modern cmake and reduce it to just 5k lines which is already a lot but uh or still a lot but um better than 76k lines and then um we have a lot of really old c code legacy c code um where the the older developers used to make uh string operations with ment copy um which of course is uh really a nightmare and we started to modernize all those old constructs by uh c++ language features like strings but so this is something that the user usually don't see but which takes a lot of time in development but yeah we think it's it's worth doing so because uh we realize that if you want to make smaller changes that a certain piece of code uh then it has impact on a lot of other places and so we started to refactoring and making the code more modern uh which makes it easier to implement new features in the future yeah that is what we're actually working on for the next release is to continue the modernization and we are doing a lot of work on the storage demon that is the part being responsible to write data to disk or tape or cloud or so and uh one thing is that we started to implement uh scasi drive reservation that's especially interesting if you have a lot of drives and maybe um separate barrio servers who want to access the drives at the same time so they can make use of scasi reservation logic and another thing is um if you have already worked with barrios you might know um if you want to backup to disk you have to configure uh what we call virtual drives so because when we write to disk we more or less simulate the tape drive there so we have the concept of um virtual drives and uh if you want to have more parallel jobs writing to your disk space you have to configure more virtual drives which of course is kind of stupid uh because on disk you don't have this limitation so we decided to make life easier for administrators uh in a way that you can now configure um a way that barrios will handle those multiple virtual drives by itself so you don't have to take care about how many virtual drives you you need and to configure this will be completely automatically done in the in the next release and then uh with the web ui we decided to switch to a new framework called view or view j s um that has a big advantage it's uh it allows persistent connections between the server and the browser um so for example you can get push notifications if a backup job has finished then you get uh something flashing here job has successfully finished or so and also has the more modern design and will hopefully look better and one thing is uh what we will also add um the possibility to make configuration changes to barrios using the web ui which is until now not possible until now the web ui is good for restoring and for monitoring so to see which jobs have finished and um to browse the log files or so yeah and then we started to um modernize the documentation the old documentation was written based on latech which is a great tool if you write mathematical papers like i did during university times but it's not really suitable if you write technical documentations especially not if they get really long so um we decided to modernize that and we are now using rst files and things and that is the same framework that for example python uses for their internal documentation so um let's work in progress but it's almost finished and you can already see the results on docsbarrios.org another thing we are working on is uh to build more unit tests um we were using regression tests all over the time like more than 100 regression tests we had we were we are also using Jenkins to do continuous integration tests that means if we build new packages for let's say a debian distribution then we will start a virtual machine running debian and we will install the packages run backup and restore jobs and verify that everything works and we do that for every distribution that we that we support and now we started to have more unit tests so i told you before that we are going to modernize the code make it more modular and this brings us into the situation that we can do more finer granularity tests using c tests so that's also a lot of work and um that will improve the quality of the code even even more okay so one word about installation packages we are using the open build server coming originally from from susan that means from just one source on github we built the packages for all the supported linux distributions and we even built the windows installer packages um with cross compile make files using the the open build server so additionally um we built on on request so here's a picture of the web UI browser so on the on the left you can select the client that you are interested in and then you can drill down and select single files or sub director trees or whatever and restore it to the originating client client or any other client that is reachable by the server by the barrio server and you can even choose if you want to restore to the original location of the file system or you can add a prefix like temp resource and then you can um compare on system what files you want to have back yeah okay so then some things from the marketing or publicity uh way people see us so i don't know if you know black duck black duck and open hub they analyze open source systems and they found us to be about about the two two percent most active teams um on on on github and one thing that is particularly nice the french government has an inter ministerial working group that analyzes open source software and gives recommendations for the public administration which open source software they could use and they uh recommend barrios and that's the only backup software on that list so it's um quite nice for us um this is a chart showing downloads weekly unique with us on download barrios.org it's over time since we started doubled almost every nine months um since 2006 we use mirrors so there's no viable numbers since then um yeah but we still see growing demand some of our customers so we have several max blank institutes we have the the variant state archives as case study um we have um companies and organizations from all sizes so you can use barrios at your home office and you can use it and and big university environments a big company environment so even at um national archives so we have one national archives in the european country with two petabyte of data and there we backup from a cluster storage and write it to a set storage because they wanted to have different technology involved in their original archives and in their backup archives now we have some finance institutes that are under observancy of the german bundesbank the federal bank so they have to follow all the usual bank regulations yeah and we are working together with partners so as said before the software is completely open source and we provide installation packages and repositories every year when we publish a new major version uh in the time between we also do bug fixes and maintenance releases on the source code of those bug fix releases is always available on github and people can compile it by themselves if they want or they can use our commercial subscription service which is similar to reddit enterprise you know so uh customers pay them to get verified binaries um and we we deliver support concising and training services together with partners um worldwide but if you'll see on the map especially in the united states on the western part uh there are no partners yet um that is um to be honest one of the reasons why we are here so we are looking forward to meet uh company service providers that are interesting and in implementing our open source solution at their client side so if you're interested just talk to me here are some some links where you can find more information if you're looking for case studies for more technical papers the last link is of particular importance it's linked to the open source backup conference um and there's a full archive of all the 10 years history of that conference with a lot of case studies videos um pts slides and so lots about a lot of topics around open source backup then here we have some picture credits uh just for the sake of completeness and of course we are here at scale booth number 315 and um well we know some time for questions but if the time should not be enough or if you want to talk more with us we are here tomorrow at that the booth so any questions yeah ah good question the question was if the schedule always takes place on the server or if you can initiate something on the client um the answer is by default the schedule takes place on the server and the server will initiate the the backup job um but as we have an an api and a scripting interface it's quite easy to initiate jobs from the client you just have to pipe some commands into the command line interface and then you can can start a backup job from the client and a question about brick-level email backup um particular microsoft exchange um we support uh vss snapshots so the windows client uh does that so you get a consistent state of fire um but the rest is up to you so if it really works with uh with single mails i am not an exchange expert but i guess you have to oh i know that you have to restore on a file level so um if i don't know how our exchange works if they have more than one mail and in a big file this psd file uh then the smallest item you can restore is a file so i hope it answers the question yeah so if you have other email servers like your syrus imap or duskat they usually have one file per mail uh there you can restore a single single mails yeah okay next question here ah the question is uh using it at home was a practical was just one machine uh the answer is with one machine it's not practical so but uh i i thought more about um geek like home users who have a little network so for example me i have my server running around the clock then uh i have my my wife's laptop my own laptop my uh children's minecraft server um and so um there i have around my own barrier server but for one server it's not now it's not practical yeah okay next question over there okay good good question that's it's about roaming devices like laptops who are not constantly connected uh the answer is um yes that's one of the new features we've implemented a few years ago it was funded by a german isp called global ways and uh it's okay i have to dig into a little bit more into the architecture so before we had this feature the way the connection was established from the barrier server to the client was always in a way that the server initiated the connection and talked to the client and said hey client you have to make a backup and that of course only works if the client is available at that time and especially with roaming users like laptops you are often not in that situation so they wanted to have the possibility to have the um connection being established by the client well that's that's what we call a client initiated connection um and the barrier server will then uh when it gets the connection send the information hey uh some pending jobs for you so yeah so the answer is it works this way yes i think here was another question yeah okay question was uh for the home user scenario what how to backup the barrier server yeah um yeah i must admit i i'm not doing that and that is sometimes of course a problem um well what i do is i have the backup files on an external uh disk so um that's better than have it inside but of course not perfect so it would be good uh or in a perfect world you could have or you would have your own tape drive and then make backups to tape and bring the tape to your neighbor or friend or so another option is to uh make replicas of your backup to some cloud storage that you trust or trust should not be enough you should encrypt the backup and then you could make copies of it to to a cloud provider yeah okay next question okay question is what happens if the full server crashes um called bare metal restore uh yes we uh have a solution for linux we work together with another open source project that's called rear relax and recover it's a belgium project and what they're doing is pretty interesting it's a collection of scripts and um you have to a rear mix emergency boot image so you run a script on your linux server and the script collects all basic information what kernel version file system layout logical volumes drbd what what other kernel modules are used and then it creates a boot an iso image containing all this information and if you boot from this iso image it will automatically recreate the basic system layout including uh file system logical volumes kernel and modules and so on and it also offers the possibility to plug in with other backup solutions so uh you're specifying a configuration file that you want to have your barrios client on that machine on that emergency rescue system including the configuration to connect to the barrier server and if it then boots up and recreate the system layout it will finally trigger a backup job restoring the data so um that is how it works with with barriers and rear and rear also works together with a lot of other backup solutions so we are just one that plugs in there but for windows we don't have a ready solution right now we are working on it but with a low priority at the moment yeah okay next question ah the question was about rear where we store the iso uh you have separate possibilities you can just include it in your backup so that is a way that most of the customers do um and then if it comes to restore your first restore the iso image to a separate server burn it onto a cd or make it available via network boot to the to the client you want to restore but of course you have all the options you can burn it directly on a cd or put it to someone else's places yeah okay yeah another question yeah sure the question was although remark was um i'm just repeating for the video um that you some people say you don't have the backup until you verify that the backup works so yeah that's true basically um we have what we call uh verify jobs so you can uh um that's kind of internal thing that's we'll we'll check that if the information that we have in our catalog in our database is consistent with the data we have on tape or on backup medium um but of course it's better to do real-time restores and and then really check what you have um some customers do it some not um of course yeah we recommend to do it from time to time um i mentioned the the osbcon before the open source backup conference and there was i think two years ago a very interesting talk about verifying database backups and what you can practically do with it so the approach was um you do a restore to verify that you have your data and then you have a working copy of your existing data and you can use it for like analysis purpose for read only purpose in some instance so um it's just a collection of ideas what you could practically do with these kind of test resource okay other questions yeah okay then that okay how the vmware snapshot backups work um yeah it's a python plugin that makes use of the vcenter api so that's that also implies that you have to have a license from the vmware site that enables you to use these api and these api offers you the possibility to trigger a snapshot that's the first thing and the second thing is um it supports technology called cbt change block tracking so um we start with a with a full backup of a snapshot and then if you come back next day to make an incremental backup we pass the information of the timestamp we got or to be more precise uh you get a unique identifier for the snapshot you recreated the day before and we pass this identifier to the vcenter api and we require the vcenter api to give us only the change blocks since that timestamp or since that id and then we only write the the change blocks into the backup okay think one yes yeah yes yes um in fact if you do a default installation of a barrio server it will directly install the client on the server and there are some jobs preconfigured one is just a kind of self-test it will be one directory so that it proves that it works and the other thing it will make a daily backup of your catalog database so yes that that that works so you can have a single server and client environment so if it makes sense don't know but you could do yeah yeah it's doable and there's one scenario where it really makes sense um if you backup to let's say an s3 or on self storage um that can handle multiple collections at the same time so it's one of the big advantages of a cluster storage like cef that it can handle like 20 or 100 parallel connections uh but if you do it uh and the legacy barrios way you would have one uh what we call storage demon that's a server part talking to the backup storage and in that scenario you would canalize all the parallel jobs through this one server and that would be of course a bottleneck and one workaround for that is that you install a storage demon on each of your clients so that you can write in parallel to the to the cloud storage it's of course not an optimal way but that's one of the scenarios that could make sense to have a server on each client yeah but the short answer is doable okay good ah yeah okay uh yes you don't even need another barrios server we have some command line tools um one is called be extract and that will just extract everything it finds on the backup medium whether it's a tape or or disk image and then you get everything that's inside there um and another useful tool is called bscan that will reconstruct the catalog so in the really worst case that you have lost your barrios server and your database backup but your your you have your backup medium left um then you can just uh piece by piece have it scanned by the bscan utility and that will fill up your catalog again with all the data it founds on the backup medium okay other questions yeah then i think yeah we are like nine minutes ahead of time that's i think a good idea okay one one question uh name of the uh bare metal restore solution yeah rea relax and recover r e r a r relax and recover yeah okay another question yeah then um i hope you have some plans for tonight i don't know if they still have beer on the other part um but i'm pretty sure there are things to do here at pesadena i'm looking forward to go with uh some friends to a brewery and have a nice nice evening so i wish you're the same and uh yeah would like to see you again and thanks for listening bye