Here we go folks Vlog 4 of the E3 trip. We're all packed up getting ready to go out to the show floor play some games like let's go Pikachu and Eevee Got a little tree hush dream going on for you guys all day.  So Eric's already been here for a couple hours. No idea what he's been doing But we're gonna do our best to get all the coverage we can Cyber sorry, we need theorists whatever Sorry, sorry All right guys, uh, what are you guys enjoying the most so far? I've only played smash brothers And now we're both off wait. Are you online for? Pikachu Smash Bros Like I tried Kingdom Hearts 3 this morning produces some complication with the booth and not paying attention Stuff happened. Anyway, the point being is that Smash Bros is really good All right, thanks guys. Oh god man. All right. Let's guys see you guys later Another Adele taco trip. I don't see me guys. No on Twitter. I'm in a lot of pain It's been a very painful day, but hey Don't say we don't work hard Lots of videos coming your way over the next 24 hours And Eric got to play Pokemon I did we did four hours for it. Yep. I'll be playing tomorrow and Eric will win the wife for a second Yes But we're at other sides. We're done with the design of booth and we just have some uh Some third party games to hit up tomorrow and then we'll Maybe play some other games. Uh, can I get a little bit of mild and a little bit of medium? Oh


And this is architectural patterns of resilient distributed systems. I do have a pug, and I really like him, so you may see some photos there of dogs that are similar to mine. So who am I? Oh, that's one that I found in the street. So I live in San Francisco, so once I found a lady with a very small one, and I went crazy for it. So my name is Ines. I'm a distributed systems engineer at this company called Fastly. I also, I go by at random mood on the Twitter. And I also help run the San Francisco chapter of papers we love. You, in Spain, there's a Madrid chapter, there's chapters on every city. And the organization aims to bring academic research closer to practitioners, so I would really encourage you to join your own chapter. And let's get started. I mentioned that I work for a company called Fastly. I don't know if you know what Fastly is, what a content delivery network, and it works this way. So you basically, you come, you bring your content to us, we spread it all over the globe, and then it becomes closer to your users, so the web experience is much better, but we're like so much more than that. Imagine a globally distributed cache layer that you can also program, and then you've got real-time login, and temperature, and it's very nice. We're very like, I help with making sure that your bits make it there. So give us a try, nice. So this is what we're gonna talk about today. So we're gonna start a little bit about motivation, what led me to start exploring this question. A little bit of what happens, what can we find in research, and also how industry maps these concepts that we find in research, and a little bit more conclusions, and then in the conclusions section, I'll get to ponder about the world, and maybe you agree with me, and maybe you don't. But okay, let's define resilience, or what do I mean by this? So resilience is the ability of a system to adapt to keep working when changes occur. So whenever situations that are planned, or actually most of the time, it's like things that are unplanned, how well our system adapts to those situations, and how much progress can we make when things are not the way that we planned them. So let's define a little bit more, what I mean. I'm gonna toss into the term resilience, or into the bucket, resilient, a lot of other things as well. I'm going to talk about fault tolerance, the ability of a system, the scalability of a system, failure isolation, and then also complexity management. So that's what I mean when I talk about resilience right now. And why do I care about it? I care about it because it's a thing that matters, right? We can have a very nice UI, very nice product, but if our product is not up, and if our product doesn't really like our system, doesn't keep making progress, it doesn't matter. It's like this is the thing that is really important. So it's all right. So this came to me because a few years ago, then I became a distributed systems engineer, and I had been exposed to research, but when the perspective changed, all of the research became new to me, and all of these questions became things that I started to ponder on a day-to-day basis. And the thing that I want to know is like, okay, how do I construct more resilient systems now? Sometimes when you go to a new job, you have pre-existing applications, and you can see some of the patterns and some of the problems that these applications may have.  And that is an interesting thing, right?  So I wanted to make sure that my applications or my systems actually did not cause a person to swear my past self and just like, I would like to be able to see this person on the street and say hello. So how do we construct more resilient systems? Let's think about it from the terms of literature. So normally when I confront it with a problem, I just try to go and see what I can read about it and how other people have solved it. And then just like I try to contrast it with what is happening right now and then try to just mesh them together. So in literature, we're going to cover three models that really shaped my thinking when it comes to what I think about when we talk about resilience. So the first model that we're going to do is an oldie but a classic. This is the Harvest and Yield model. I don't know how many of you are familiar with this paper. Can I see some hands? Oh, one? Okay, cool, cool. This is great because I thought I was going to be here and telling you something that you already knew so this is fantastic. So this paper came about in 1999 and it formalized a lot of the concepts that we use right now to build our applications. So some of the things you're going to be like, duh, but they were discussed and then the patterns that we use now were first formalized in this paper. So this came out of Berkeley. So let's see, it has two concepts that are used to just describe the behavior of a system. You have yield and yield is the fraction of the successfully answered queries. Also, I tend to speak very fast and I don't know if people are following along. So I was supposed to get a visual cue. Somebody here, tell me if I'm speaking fast. Okay, all right. All right, this is good. Do I have to catch? Do I have to be like, meh? Or no, is everybody here? Like can we follow to why I care about this? Is that clear?  So we know that there's gonna be like models in literature and then this one, a literature by means research and then the first one is going to be harvest and yield and it addresses a specific thing about system design. So we have two concepts that are important. The first one is called yield which is kind of a funny word, but yeah, it means how much information I get back from my system and you can think about it as uptime, but it's not uptime. It's much more focused on like your experience as a user of this application rather than how available is your application. Say for example, during Christmas, you're trying to do shopping because Americans do a lot of their shopping for Christmas. That's like it defines who they are as people. How well you shop for Christmas. So imagine that you have a system like the Shopify system that is unavailable for two minutes during Christmas season and you can have the same system that is unavailable for two minutes in the middle of a weekend at 3 a.m. in the morning. Like this is why it's not uptime is that there are different impact to your customer use base. So everybody understands yield now, right? All right, let's see what harvest is. Harvest is a different perspective on the same problem. So harvest is the fraction of the completed result. And then we have this example to kind of illustrate a point. Say you know that I like bugs and say that I wanted to look for cute baby animals. And then in this particular application, we have a short database and then in some side, in server A, I have things that are cute or things that have been tagged cute. In server B, I think things that are like babies and in server C, I have animals. So if I wanted to get the cute babies and the animals, then I would send a query to the three of things and then I would get information from the three of them. So say for example that the thing where babies are held, the server that babies, not the babies, the babies are not down, but the server that has the babies is down. I will still get something pretty good, right? I will get cute animals and maybe I'm gonna get some animals that happen to be babies, but I still can do some progress and my system can still be responsive even though I don't have the entirety of my dataset. So in this particular case, it makes sense. Like we have 66% of harvest, but I still respond and I still make progress. So we understand yield, we understand harvest now. And the thing that this paper gave us is like two particular models to start thinking about availability and one is probabilistic availability and they use two things. It's like how do we degrade or how we respond to failures. And this is where like the concept of graceful degradation came about. So this paper is really nice. I think you should all read it. But it's like we keep making progress and we choose either things that are probabilistic availability or the next pattern. And then in this case, they have a few mechanisms that allow you to use the first approach and the first one could be randomness. Like in my example before, I have information that makes it randomly to different servers and that is good because you reduce the chance of a particular piece of content being down. And then the second one is replication. You can make copies of it and then if one copy is down, you can still return the other copy. So this is why the author is called probabilistic availability. And then yes, and then sometimes you can even degrade the results based on the client capability. For example, if you have a bad internet connections and you're trying to watch a video, the video may come with less quality and this is like less harvest, it's like less information, it's less rich, but you can still watch it. So all right, so this is approach number one. How are we doing with the speed? Okay? Yeah, good. All right, so normally what I'm gonna is gonna happen, I'm gonna start running out of time and then I'm gonna pick it up at the end, so we're gonna go through this together. We're trying to do it very well now. So all right, the second approach is the composition and orthogonality. And this means it's just like that you don't have to solve all of the problems in the same application, you can break it apart and then your subsystems can deal with the problems in a much more isolated way or you can actually have things that are completely different. And the example that they present in this paper is for example dealing with like certs or like security. Like sometimes maybe you can use a library that does encryption for you and you don't have to build your application with encryption on it. This is what they call orthogonality. So that is nice. So we have something that allows us first to just have availability that is probabilistic and the other one is just the things that we can break them apart and have components that handle one specific thing. But I think that the true contribution or the thing that I like the most about this paper and it's strange because as I keep going along, I keep going back to the paper and I keep picking up more things that I missed the first time is that the main takeaway for me is that if your system favors one versus the other, it's a favor of it, it's a factor of its design. Ah, disappeared. So it's an outcome of its design. So like if you don't choose one way or another, it's very hard to put it later. So that is I think the contribution of this paper. All right, the second model comes from a completely different area and this is what I'm gonna call the Cook and Rasmussen model and Richard Cook is a physician and he talks a lot about system safety and I got a chance to meet him this year and at a conference and I was very weird and I was like, oh, I've seen all of your talks on your paper and he was very worded out, but I really like him. So this is also a very good, he also wrote a paper called How Complex Systems Fail that is a very, very nice one. But this is his model. So you have this universe and then in the middle of it, you have what is called the operating point and the operating point is always in motion. In one side you have economic failure boundary, the things that are economically feasible are not for you to do. In the other side, you have the unacceptable workload boundary. If something is very tedious to do, then you're likely just going to violate that boundary or things that are just the accident whenever things go really bad. So the operating point is in motion and in most and sometimes you would have different pressures that get placed on the operating point. So in here we have pressure towards efficiency, we wanna just save more money and then it moves there closer to an accident or sometimes we wanna do less work and we're just gonna cut a corner and then sometimes then when those pressures apply then the operating point goes over a boundary and then we have an incident.  So this happens a lot and what ends up creating is a boundary that is called a marginal boundary and this defines how close you are to having an error. So this sounds very nice in principle but we know that we like sometimes to cut corners, we know that sometimes we just don't do things like or we don't test things as thoroughly as we should and then there's something that happens when you're close to this boundary and it's called flirting with the margin. So say that we have this thing and then we have an estimation or an intuition of the things that we need to do not to have another problem. So we're like from there to what is it for immune? Right hand side of the boundary, of the marginal boundary and then like, well today we didn't do as much as we thought we were going to and then we're like, oh, we didn't have an incident and then maybe we're like freak out about it and it's like tomorrow I'm going to actually enable my tests and then now I have tests but the last time the test didn't pass I deleted it and we're fine so might as well just do this now and I'm getting really comfortable with this area so what ends up happening is that I end up redefining how close I am to the error boundary without even knowing. So this is normally what happens when we start doing this. We have this illusion that it's fine but we never really know how close we are to the accident boundary and we keep pushing this limit. So the insights from this model where like this thing of how close we are to an incident or how close we are to an error was very important to me because the paper defines that resilience is a factor of like how you operated a system. It's a factor of how you explain the system to people that are new, how you respond to when an incident happened, how what happens whenever you know that you have to change parts of your application and also like how you learn from the mistakes that you have, how you keep redefining how close you want to be to this error boundary and also that the safety is not about necessarily about what can happen but it's also what you do whenever bad things happen but this is from an operational perspective. The one before was from a design perspective and Cook reminds us that how we operate our applications are very important for us to build resilience. Okay, so he also tells us how we can engineer things with system resilience. We wanna like build support for continuous maintenance. Maintenance is something that is gonna happen the moment that we have a system out and the moment that a system goes into production if it's doing anything worth doing, it will become a critical system in your organization. It will become something that we need to have like some safety in order to just like in order to keep making progress. So those are the things and then I don't know how am I doing in time, 42. Okay, and then also the main important thing is that he says that we should like we should give control of our applications or our systems to the operators and the operators are gonna do crazy shenanigans with your applications and then you can't prevent that. So you might as well just expose the knobs and expose the dial and let the operators do whatever they're going to do and then you react and then you put things in place as problems go along. So this is how Cook and Rasmussen tell us that we should think about resilience in our applications and also they say that we should think about configurations and interfaces because people want to like be able to interact with our configuration and then it's much better to explain in the paper. So like I encourage you to read it as well. Okay, so design, how we operate and then this is a third model that also inform my thinking and in San Francisco there's this person called Paul Borel and he's the best synthesizer of papers.  But I'm going to call this the Borel model because that's his last name. So Paul tells us that we can split, we can think about in a system complexity in a sense of like in a continuum where we have a rank of things on the probability of failure and in the literature like it's a little bit more explained but because I don't really have that much time he splits this work into three main areas and you have things that we consider traditional engineering, things that we consider reactive things like that they're based on operations and also there's this area that is called Unk Unk and it stands for unknown unknown and what he says is that this thing keeps going on forever and at some point everything that is unknown if you fold it into the first two categories has the same like surface as the first two and cascading of catastrophic failure comes in this Unk Unk section because if we would have known that they were going to happen we would have already like used the strategies before to catch it but anything that has to do with the unknowns are things that we couldn't possibly predict and this is where the tricky parts of the system this is why building distributed systems is also very hard so what Paul also tells us is that okay they're different failure areas and we need different strategies and different like different approaches to deal with them I don't know how many of you are familiar with the work that Kyle Kingsbury has done to prove the correctness of a database but we have an area then for Paul anything that Kyle has done to prove like how distributed systems work deals with things that attack the first two areas and then we have another researcher and his name is Peter Alderal that wrote a very important paper two years ago called Molly where he attempts to reason from a correct system design to see where it could have gone wrong so say that I did a computation that gave me a good state he tries to chop the computation and analyze the execution path to see if the problem arrives to the same like if the program arrives to the same good spot it's gonna crazy that he does this but he reasons from the outcome up to see what are the things that could be going wrong so he's saying that all right we have people that are already attacking these two things these two areas and collectively we can do something that is more robust than is more resilient so that is interesting about the Borel model where we now have an awareness that first we can have design and we can have everything the traditional engineering gives us and then we can have things that we can do in operation to reduce the likelihood for us to have a problem and then we have things that we can't do anything about and at that point we're screwed okay but there's some things that you could do right like in classical engineer we have things like coding standards we have programming patterns we have like testing of the full system and if you know what I mean by the full test system is that you have to test in a distributed system the client the server code and also the provisioning code because most people don't necessarily like test the provisioning code if your system takes a configuration the thing that configures it should be tested as well as what your system does whenever the configuration is wrong because that is going to happen and also that are right and we think about these systems as they like they converge to some good state with classical engineering that's what engineer classical engineer helps us do in reactive operations we have okay like inventory of things that could be like hazardous to your application you need to know them at redundancies like you can use feature flag to deploy code to production you can do dark deploys where you push it out and then it's just like it's out in the world but no traffic has been turned on and you can have also like run books and documentation are very important and you can just like have canaries and these are the things that you can do within the reactive operations to make an application much more resilient and in Ununc you can do like formal methods fault injection and things that allow you to do some sort of system verification of stuff that you couldn't have predicted also the goal between these three areas is to reduce like domain independence and then to reduce the amount the amount of like the possibility for an error but you do different strategies so this is what Paul tells us when you think about building resilience using a single discipline is insufficient so if you think about it from one single perspective is insufficient and we need different strategies all at the same time which kind of sucks but it does follow our intuition but now we mean so we have to attack three different areas and then one of them can screw us at any time it's like we just don't know what it's going to be and it's not really great all right so I bumped you out enough I've told you that there's different approaches so let's see how what happens in industry with them obviously whenever we talk about the things and how we're done like at scale you always have trouble like you have like three papers from like few companies and Google is like yes I'm gonna be a cliche here but this paper is really really good and I really like this paper because they describe how they constructed their locking service and then it's like a seminal paper and it's really really fantastic and here are the key insights that I took from this paper like they pondered doing this thing that managed distributed locking at a company scale and they pondered two things whether they were going to be in a library for developers who use or for engineers who use or they're gonna do a centralized service and they chose to do a centralized service and provide the client libraries for people to use so they would have the control that is necessary to just to do something that allows you to have resiliency also they did limited the scope of the problem by just only offering storage of small data file with restricted operations and then they said on the paper something that is very, is a little controversial but and it hits close to home so I have very like a lot of feels whenever somebody tells me this but it's like okay in the paper they say that engineers don't plan for availability, consensus, primary elections, failures, their own bugs, operability, the future and they also don't understand distributed systems and this shit is happening at Google so like okay the monstrosity that is Google in terms of like systems applications and share knowledge and we get this insight from them so it's just like again it's like it's pretty depressing but then another thing that they tell us to it's like okay we have a centralized service or they have a centralized service to manage distributed locking and it's hard to construct so these problems are inherently difficult and they're difficult to reason with and they're difficult to put out and you can just like dedicate effort into architecting well if they're very well, if they're very well scoped and then they said that by having a service that everybody uses allow them to pull resources and make sure that that service was very, very resilient and they just like threw everything at the service to make it fall tolerant so that's pretty nice right like if we're in a company of that size some of these problems may be solved for us and also they said that the cook model says that people can do whatever the hell they want with our applications they're like eh no you're gonna do these two things and no more than these two things and these are the primitives that we give you so they say that if they restrict the user behavior they can increase some of the resilience because they can just narrow down the API for this particular service all right and then also the consumers that although they did that they said that they found corner cases that were completely like completely unpredictable to them so again we have this problem of onk, onk popping up again all right so now Netflix because everybody has to talk about Netflix when we talk about resilience right are there any Netflix employees here awesome good all right so tons of things from Netflix they started chaos engineer and there's a lot of patterns are very good I'm not gonna comment on them yet there's a lot of links you can use all of these things that they say in here they're still applied they're still true they're very good but the thing that I find much more interesting these days is like since they started with this chaos monkey chaos engineering they're now even going beyond they're the things that originally they told us to do like these things into something that is like a little cookie and a little bit like more science fictiony where they are like all right systems are complex we can't reason about them we might as well develop an intuition and they have this thing that taps into all of the monitoring system and in real time shows how traffic flows and the entirety like that start of microservices how it looks from like abstracted away I don't have a demo of like how the Amazon regions would fail to each other and then they have the little dots are like packets and then they're like their request and have different colors when they're healthy when they're not healthy and that was pretty amazing so imagine if we can have something like that for all of our applications and we just like like see them we're better at see them and like second like into it in things that we are about like analyzing metrics, graphics and all of the things that we need to in order for a system to work correctly so that's pretty neat so since we cover Netflix very very quickly I'm gonna share a little bit of the things that we do at FASI as well so we have this system that is called Powderhorn and these two people like I work with them Tyler and Bruce gave a talk in Barcelona a few years ago describing the story and the evolution of how we created our instant purging thing so first one's B1, B2 and then in B3 we got it right after reading some literature and we got it right by using a gossip protocol and then it's called by model Montecas and then this talk is very nice because it describes the entire complexity and our problem domain is very interesting because we deal with everything that is awful on the internet so everything that is awful in the internet we have to respond and then this particular thing has to be very efficient, very fast and it has to go everywhere in the globe and then invalidate your cache very very quickly and it does so in this talk you can learn a little bit more how we do it and they use it by just using a gossip protocol so on the Nexus area which is like we have ways to programmatically like interact with the internet we have this single fail D so you have the internet and this is horribleness of everything and then traffic comes to a pop and you have this protocol that routes traffic to different servers the problem is is whenever one of the servers go back the protocol rearranges the table and things shift so if you have content in one of them like your content gets moved and you need to be able to do this so the way that we solve this problem is by just hacking the protocol and using just MAC addresses and fake MAC addresses to be able to like predictably route traffic around this is also in a talk by one of my colleagues called Joao so if you're interested in how we do that this is a trick that we also use to just like all right the current infrastructure doesn't do it let's trick the current infrastructure to do the thing that we want and abuse this open configuration things and actually get more sane behavior and more resilient behavior from how and route traffic reliably within our own pop and then like I'm gonna talk about then a little bit of my team so for the last nine months I think it's nine months right it's September so yes nine months it is September yes it is September so for the last nine months I've been like trying to start something new and then there's different challenges this thing about trying to make something resilient it's very important to me because now I'm gonna be responsible for a system that I have to support as well and other people are going to be dependent on and we're in beta and it's great and then we discover something and now I'm here and everybody's working on a problem that I just can't help so it's great I'm gonna tell you then what my system does so you have a user that just come to our CDN and then they have my service and we go to the origin and they fetch the images and then we resize them fast and then return them back to and then we resize them and then we put them on our CDN and then it goes back to whatever device or whatever like customer or whatever client you have for that particular content piece so again, because I was doing fast I want a picture of my dog my picture of my dog lives over there and I want to resize this big I have to figure out with our team that we're on we have to figure out how to do that more efficiently and then just like and very, very fast so in a way, the advantages that we have is that this system is stateless I don't have to keep the copy of the original I only have to transform it very quickly which should be like easy because like when you don't have state to deal with you have a little bit more allowances and your system can be more resilient and I don't have to keep anything no coordination, no database I don't have to use things that are like harvest and yield in the sense of like how much data I give out but I still have to be available and then data has to still come out the thing that's most difficult for us is how we thought about the request cycle through all the dependencies that we have and the dependencies are the hardest thing and they're the hardest thing because this application has to be it's everything like customer setup if you messed up where your images live I don't know you like for example screwed up your S3 like bucket permissions we already cannot access it and then that disrupts our system if the caching layer is having a problem we need to be able to know that the caching layer is having issues and then just not send traffic back to the caching layer the interaction between every point is very complicated and it's very complex and this is a thing that actually like has cost us the biggest time like our design the same time has gone up front to figuring out this thing and we even the libraries that we use can have problems so at this point we're thinking about resiliency on every interface and every area that our application covers and it's hard so I was thinking all right another thing that has been related to how we deal with resiliency is like how you define your error types and what you decide to expose versus not because sometimes you think that sometimes you could be just hiding away information and your system is less resilient than it should be because you're just simply not thinking about like exposing this problem that you're having and handling those carefully like is very important so in this case failure detection and system operability are ongoing concerns for us we have many servers, we have many many locations and if one of our servers goes down again as I mentioned the interaction between the edge and us has to be like synchronized like if we're in a state where we can't take traffic we have to signal the edge and who's talking to us to actually not send us traffic okay so this is getting boring again in a sense it's like it's complicated and it's hard and another thing that is very helpful when you're iterating this fast is to have intermediate versions for things that you save so for example we versioned the way that we represent things internally and the thing that you wanna know about that if you're ever doing that it's very good to have it from the get go as part of your design but you should like know that you're gonna you should have to support for mix mode and this means that if I have version two on this server version one is could be on the other server on both of them need to be able to communicate and that is missed very very frequently when you forget that you need to support mix mode happens like it happens more frequent than not it's like it's a huge it's a huge threat to your availability and your resiliency all right you want versioning for everything all right so we covered a little bit of literature we covered a little bit of like how companies think about it let's summarize it because I oh the person that was asleep woke up so it's great so let's then just like summary like summarize a little bit of this okay so in our application or anything else things that have to deal with redundancies are very important so you add resilience by just putting more of it and then again we saw this in the things before but you have like redundancy of like resources execution even for example like whenever we're talking about our purging mechanism this thing's gossiped so it's more than one message that goes out and this is redundancy of messages and this help you like stand up like or be able to like withstand circumstances or things that you didn't plan so all right all of those failed resiliency and also like something that is very important too is like capacity planning is still something that is worthy of an exercise and now I'm trying to even just do recent capacity planning to the point in like it's correlated even to the power that we have available in a rack how many operations can I do for the amount of power that this server that I have installed in a pop consumes and that is that is also hard and also that when you're trying to make optimizations like in the cook model say you're trying to save some money or you're trying to like that particular set of optimizations may remove some of your redundancies and then make your system may make your system less stable all right so we want to have more things and we want to be able to like to just test them and then just have them out so redundancies are key as we saw in the cook model operations are very important and when you don't know like how close you are to the error boundary it means that we're always guessing and we don't really know how close we are and if your operations are complex if for example your deploy process takes like 20 steps and they are like done by hand you're gonna get it wrong and this is gonna cost you an outage this is going to cost you a problem and the thing that is interesting too is like you said when you were like implementing your system or probably it's also something that you design so it's important and also like okay things are complex we are building distributed systems they are like they're very hard they can fail they can have all sorts of crazy shenanigans but sometimes complexity is good and then when you're dealing with complexity sometimes it's very very hard to make something very simple to operate and then in that sense that complexity is good because you have invested a lot of time and invested a lot of effort into make something simple to interact with so if it increases safety then it's good and also like you should know that resiliency is sometimes will come at the cost of other goals like if you wanna make a system that is resilient it's not gonna be cheap you're gonna have to have more of it and it may be just time consuming because it takes time to make this thing so it's like it will come at a cost of something else all right so again so we wanna like leverage engineering best practices we saw like a pattern emerging that the things that we know that we should be doing are actually helpful and they're helpful in different planes right so we wanna do them testing and resiliency are correlated you still have to do it I'm sorry that there's no magic bullet there versioning is important provides you an upgrade path is very very good okay so the upgrades and the evolvability of a system is so tricky again mix mode is important we recently forgot to account for this so but we fixed it but having been burned by this one now I'm like I see it everywhere and also I feel it's an interesting thing because we should be able to like the way that we prototype system I think is fundamentally broken like we start creating these things and there's just too much too many areas of attack and if you push something out and then like follow these patterns it's like you have to still revisit every single aspect and constantly check them and see how well you're doing and that takes effort and takes a lot of discipline and we don't seem to have like ways to construct applications better and then that's my rant on yeah okay I guess what I'm trying to say it's like yeah it's laborious it's worth doing and it doesn't seem like for distributed systems there's that much mechanization or things that get done for you so that's a little sad so let's wrap it up a little bit like let's start just bringing it home because probably like I'm running out of time oh cool all right so again this is my TLDR we've seen different things we've seen patterns in academia that highlighted the importance of code design we've seen patterns in academia or even system safety that tells us how we operate and how we interact with our applications is very very important and what we expose and what people are gonna do with our systems are things that we cannot expect and then we see in the world model that although like we may be very good at those things there's still gonna be things that we miss and there's still going to be challenges that we could have never predicted and there's nothing that past us could have done to like to help us in the moment that we're in and there's some things but this is where like I think maybe some higher level this is maybe where I think the future is gonna help us the most how we simulate, how we model the systems before we even construct them and I think that that's interesting that that's where we're gonna see the biggest returns maybe we have a language oh there was a paper where like Microsoft described this language where as you made a change it went to the cloud and run formal methods on everything you have to let you know if you have violated an assumption and that's amazing so like if we can do that as part of our like development environment or how we develop that would be great so like since we're talking about the future that would be the future that I wanna live in but all right let's summarize it then we have here the things that we could do in design and in design we could do all right we have to figure out are we favoring like harvest are we favoring yield we need to make this decision explicit we can use orthogonality as in like we can just like have different areas with different responsibilities or the composition are like for the wind things that help you do we have enough redundancies in place this happens in design are we resilient to our dependencies that's much harder to do than it sounds and also like theories matters because sometimes this problem that you're running into has already been studied and a solution has been already proposed decades before you run into this problem and it's nice to be able to borrow from that and then just start from a point where it's well known that it works operability are you providing enough control to your operators or your users and a big question that will let you know if your system is resilient or not is would you like to be on call for it so that would be really good once you're on call everything that is shitty in your system will wake you up at the craziest times of the day so that motivation is real so I think people should be on call for their own applications so especially if you're responsible for it you should rank your services in terms of like what can be dropped when you're talking about harvest and yield if you have things that are not important and you can just drop them and shed them when you are in situations that are problematic then your system can continue to make progress as well and monitoring and alerting are very important they should be in place they should be part of your MVP and when things that have to do with the unknown unknowns the existence of like stress actually like the unknown unknown like stress is how important the first two areas are and how when you cut corners on the first two you're moving closer to the error boundary and on top of that you have this extra area that is as big as the first two that is coming for you and is going to ruin your life and ruin your weekend how we run everything we can and if everything else fails you we don't do this in San Francisco yet we don't have enough intern but you can start like doing human sacrifices that works I know maybe when you don't call that is a version of a human sacrifice so you solve this unck-unck problem by putting the burden in a poor person in a poor human being more you should do these things tests, core reviews are good these stress claim behavior is good even if they are internal because sometimes you can do those your own application from your own company it's easier that somebody else can version is good remember mix mode checksums are very good and then you have things like error handle secret breakers it's like again if you can just shut down traffic to certain areas so you can still continue to make progress back pressure is important leases are important timeouts are important just make sure that those things happen while you're in design or at least you have a node to just go back and think about them and the probability automation anything that you felt to automate is going to be something that you run into when you are drunk at 3 a.m. and you get a page it's very important to do that and then release stability like it's often tied to system stability if it takes you forever to release your application or do a deploy then it's very common it's often you're gonna make it like you're gonna get wrong at some point playbooks are good you should link your alerts to things that are actionable like if you see something pop up it should have a link to how you monitor, how you debug it or how you troubleshoot it and also like how you configure your system has to be consolidated especially if you have many of them if in one system you do it with like for example a data bug in Chef on the other one you have a config file when you're moving from one to the other and it should be much more straightforward and should be able to have some common patterns for how you configure your system across your organization and also the important thing is that you should keep in mind that your operators determine also how resilient your system is how the people that are using it like the people that are like helping you just like run it like they are like also helping you determine how resilient your application is I could be that have the same the first like the most amazing system and if it's a pity to ask to monitor like nobody's going to go and read through my logs and see what happens it will be very hard sometimes alright so this was me last year it's like I was like we can't recover from lack of design design is all of the things that matter I'm not minded either harvest yield or not being aware of everything means that we're going to sign up for a redesign the moment we finish coding and I was very idealistic and I feel that I was very arrogant and stupid it's like now that I have something new this thing about redesign has already happened in different areas and we got it wrong so I was like okay fuck like this thing is like it's hard like having a good design is hard the unknowns are like super hard to predict like the dependencies even are super hard to like to to just like to reason through so I felt like my original apprehension about everything should happen in design is incorrect like I think that maybe like like redesigns are a part of having a system that evolves maybe like I think about it now in terms of like alright I touch this is my assumption still valid a few months on the road is my assumption still valid and then like maybe now like I think that the when I look at a problem or when I look at something else I just go through the checklist of everything that I have am I doing all of the things and what did I miss and maybe I shouldn't really like try to fight so much the redesign and also like think about it more in terms of the adaptability and I think that that is kind of nice and I think that is a better way or is a way that I like to live more with the terms of like there is a way that makes me happier when I think about resilience because we know that we're gonna have to change we know that a lot of things have to be reconstructed when your assumptions change or the world decides to throw you something that is completely bananas and before I was like everything should happen in design and then now it's like okay this is an evolution to this and then this is a reason also like why sometimes when you read literature or things from big companies they have decades of effort and labor trying to figure out we can start from there but in our applications we're still gonna have to like have this type of evolution this type of thinking and maturity. So that's all I got I don't have any time for questions but all of my papers all of my research are all in a repo so you can open an issue if you have a question I'm gonna be around and I hope I didn't talk too fast I look like I have so sorry so thank you


This is a different one. This is about journalists in Gaza. This is about journalists who work for the AP and Reuters. The AP and Reuters, two of the entities, we get pretty much all on used farm because they're the ones who have journalists all over the world who are actually on the ground. Well, in some cases, these journalists are literally on the ground.  And it turns out that a number of journalists, one of them photojournalist Ashraf Amra, that work with AP and Reuters were right there on October 7, filming what the Hamas was doing, partially because they were embedded with Hamas. But there's also pictures now being shown of them sitting around, in this case, Ashraf Amra, with a couple of his friends sitting around, watching some of the videos of Israeli soldiers, in this case, being slaughtered, just being brutally murdered, and laughing and having a good cheer about it and enjoying the whole thing. Another one, Abu Mustafa. Freelancer's been working for Reuters. And he was like, hey, I was just in Israel. This is on October 7. All of you, you should come.  This is a once-in-a-lifetime thing, he was telling them. These are journalists who our mainstream media uses. Is it any surprise that we get the kind of stories that we get? Is it any surprise that this is the kind of screening that our news agencies do where they actually get, where we actually get kind of objective information, not objective information? These people are part of Hamas. They are part of the terrorist organization. They are part of the Palestinian regime that is responsible for the slaughter. And they are reporting on it at the same time. Well, objectivity out the window. But I think that's true in so much of our journalism today.


I'm Elise DeLaGarza with Cal OES. Upgraded from a tropical storm, Hurricane Hillary is expected to gain strength as it moves north, bringing widespread impacts across California. In a very rare situation, it's expected to not only strengthen, but move northward along the coast, potentially reaching San Diego by Sunday night or Monday morning. And we're not just talking about a shower of rain. We're talking about several hours of rain. As the storm moves toward Southern California, it'll bring with it heavy rain, strong winds, and the potential for flooding. This type of scenario presents flooding challenges for any location along a river like the San Diego River, a small dry creek in a neighborhood, potential flooding. Those in or near are downstream of any burn scar, potential flooding, and debris flows. The strong winds associated with the storm could present problems for areas outside the storm's direct path. And it can also elevate the fire weather danger because outside of the hurricane's center, it's going to be much warmer, much more of a situation where fireweathers are concerned. So really the hurricane can change the entire weather pattern along the California coast. Cal OES would like to ensure that your family is prepared for any storm-related impacts. Sign up for emergency alerts, pack a go bag, know multiple evacuation routes out of your neighborhood, and check in on loved ones. To see more from us, visit our website news.calos.ca.gov and follow us on all our social media platforms.


What's up everybody Jason here from jazzy's case breaks calm 2019 20 panini mosaic sell a one pack break number 588 sell out this mixer where we're giving away the last six spots So first we'll do the break randomize customer names and teams distribute the hits of the cello And then at the end of the break we'll randomize all 30 customer names again separately randomize our numbers one through six Top six get into the mixer. So let's copy and paste the original list of customer names. There you go And let's do the break really quick guys. Let's roll it And we got a two and a three for five times One two three four and five Matthew D down to Ryan Harold one two three four and five Spurs on the Knicks All right, so there you go Just some more notables noticeable like bigger teams just to keep what you stick what you got Build at the pelicans Ryan Harold got the Grizzlies Chris got the Lakers A lot of the good teams of course, but snooker said stick with what you got All right, here you go We got Alfred Payton and Darius Basley Orzingis Mulmury and RJ Barrett All right you guys so now Let's see who's gonna get spots Yeah, Joe. It's broken on IG so at Jasper's breaks Uh sometimes when Sean does super super late night here and like I'm off on Tuesday Wednesday Thursdays He will sometimes do personals here, which I would too of course, but um When I'm on if he's on then he doesn't and then for some reason if he's off I'm still on and you guys want to do personals vice versa. It's the same thing All right you guys so good luck dice roller Stephen K down to chat Roll it and it's six and a five for 11 times one two three four five six seven eight nine 10 11 11 times And it's six right Bill Sean Aaron Parker Raymond and Martin C After 11 times congratulations six different customers. Wow. There you go Squeezing in is bill Martin at the top. Appreciate you guys coming up next is the mixer


Gut, wir kommen zum nächsten Talk. Ich hatte von einer Freundin erzählt von dem Talk, den ich erzähle, und sagte Machine Learning und Deep Learning und TensorFlow.  Ich muss gestehen, ich habe keine Ahnung davon.   Und damit auch umgehen kann. Und der erzählt uns, wie man damit sauber programmiert, wie man Dinge richtig gestaltet und richtig macht, damit es auch zu einem Erfolg wird. Michael studiert am KET und macht gerade seinen Master in dieser Thematik, in Informatik. Und er möchte uns darüber jetzt etwas erzählen. Es wird anschließend noch ein bisschen Q&A geben. Dieser Talk wird auf Englisch sein. Er versteht jemand kein Englisch? Okay. So, ich repeat in Englisch. So, dieser Talk wird über Machine Learning sein. Ich habe eine Freundin erzählt, vor einigen Minuten, über diesen Talk, die Machine Learning und Deep Learning und TensorFlow.  Ich habe keine Ahnung davon, aber Michael hat eine Ahnung. Also, er studiert Informatik im KIT in Karlsruhe. Und er macht seinen Master. Und er erzählt uns jetzt, wie man diese Themen zu verhandeln und einen schönen Code zu schreiben. Also, bitte. Applaus für Michael, Applaus für Michael. Danke. Also, wie schon eingeladen wurde, werde ich über gute Patterns in Deep Learning mit TensorFlow sprechen.  Also, ich möchte mich ein bisschen motivieren, warum das ein wichtiges Thema ist. Und für das, natürlich, muss man starten mit Beispielen, wo man Deep Learning benutzen kann. Und das erste, wahrscheinlich das meiste, famouse Beispiel ist called MNIST, das betrifft, dass man humanisch betrachtet wird. Und so, dieser Dataset war schon in den 80er-Jahren eingeladen.  Und das ist ein Beispiel, wie man Machine Learning benutzen kann. Und ein weiterer Beispiel, das ist mehr recent, und es ist nur in diesen Jahren, ist für Autonomous Vehicles.  Und in diesem Beispiel, das Kitty Road Dataset, ist es um das Detektion, in diesem Bild, das du siehst, in Red, wo es eine nicht-driveable Region ist, und in Purple, wo die Vehicle kann fahren.  Und so, dieser Task ist ein sehr schwieriger Task, und man hat viele verschiedene Dinge zu tun. Und so, man braucht eine gute Code-Struktur, um diese Probleme zu lösen, und wir müssen solche Probleme in der Zukunft lösen, vielmehr als wir sie schon machen.  Und hier ist es eigentlich ein Dataset, das ich selbst begonnen habe. Und es ist für eine Navigation für blinden Menschen.  Und so, das ist ein möglicher Task, das man mit Maschinen- und Deep-Learning und Computer-Visionen lösen kann.  Und dann, finally, ein weiter Beispiel, ein bisschen mehr für unsere Deutschen.  Und das ist v.a. wichtig für die Sicherheitsprobleme. In diesem Fall auf der linken Seite, wir haben unsere ANG. Und wir wollen wissen, ob die Person auf der linken Seite auch ANG ist oder ob sie eine andere Person ist.  Zum Beispiel, wenn du dich identifizieren willst, musst du mit einer hohen Erkürzung sagen, wer ein Terrorist ist und wer nicht der selbe Person ist als ein Terrorist. Und so, wenn du dich um viele Forschung fährst, und ich komme aus einer Forschungsbewegung, und das Problem ist, wenn du dich um eine Forschung fährst, gibt es einen Spaghetti-Monster. Und ich sage nicht, dass ein fliegendes Ding mit den Spaghetti-Tentacles ist, sondern dass es ein because you have to understand so much code just to change something. And so what I want to tell you now is how you can avoid that spaghetti monster with your code. Because quite honestly I'm a bit selfish. I want to work with your code, but I don't want to read your code if it's like the code in this example. So I hope you will learn something in this talk. And then there's another reason why you should care. And there's this quote by Steve Mcconnell, that there are 15 to 50 errors per thousand lines of delivered code. And well, you can do the math for this example. And we actually found some bugs in that code. So yeah, you should reduce your amount of lines of code. So okay, then the question is, let's solve that problem. But how do we ... And I will tell you in this talk some ideas. Those ideas are not the gold standard yet. There is no gold standard. We have to evolve a gold standard. And so I'm just telling you ideas. And I hope you can evolve those ideas and take them into your own code and make your code better. And so yeah, let's tell you. So first, how does deep learning in a nutshell work? Because our code should reflect the actual thing we are solving to be understandable. And so with deep learning at the beginning we have a problem. I used Angie in this case. We want to detect if that is Angie or not. And then when we have a problem, we want to develop a model. That model is our problem. In the case of deep learning, the most popular model nowadays is a CNN. A convolutional neural network. I'm not going to explain how it works. You can Google that later if you don't. And that CNN basically what it does, it takes in an image. And then the output is a prediction. In this case the prediction is it's Angie with 99% accuracy. Or it's 99% sure that this image shows Merkel. And while in this example it's correct, maybe if we feed in another image it would give a wrong result. And we don't know that. So we have to check the quality of our model somehow. And this is not the deep learning terms yet. This is just the intuitive way of seeing the problem. And for checking the quality usually we want some error measure. And with that error measure, then later we want to somehow improve our model to be better because we want to reduce that error. And as it turns out, this intuitive view on the data is the same as actually, if you're looking at it with the terms of machine learning, the first one is defined by your data. Your data basically tell you what your problem is. And so that's the first stage in the case of detecting Angie. That would be images of Angie and images of someone else. And then the label, if that image is Angie or if it's someone else. And so then the model, it's still the model, it's a CNN. The error or the quality measurement is called a loss of learning. And then the step is called the optimizer. And so what you can see here actually is that if we look at this, we can see already that the problem is separated in four different categories. We have the data category, the model category, the loss and the optimizer. And so why do you put all that together in the file when you're coding? Why don't you put it into four different files? And so my first design pattern I propose and it's really simple to actually implement, is not to write that everything in one file but split it among four files. One file which handles the data loading and management. The next file which actually then handles the model implementation itself. So that's just the neural network in this case of deep learning. And the next file would be the loss. And what you come and see if you look on GitHub, that people mix up the model and the loss especially. And that code tends to get unreadable. And then the last step would be the optimizer, which we can also separate. And so that basically should help you to get rid of the spaghetti monster to some extent. But then once you started doing that, you're getting quicker in development, your code gets better readable and you can start evolving beyond that. And then you will run into the problem that you're really quickly developing new experiments on your data, new model types, new loss types. And then you're running all those experiments with different configurations and you start to get your losing track of your experiments. You don't know what experiment was wrong with what. And so the next pattern is actually using a hyperparameter file. And in this hyperparameter file, you put together all your hyper information about the network. In this case, this is for MNIST. I put in there, you can see the learning rate on the top. Then you can see there's some other parameters for the optimizer, like decay and more stuff. I put there where my stuff gets saved in which directories. And so later, when I'm looking at my results of the experiment, I can put that parameter file next to the results. And when I've done like two weeks of experiments, then I can look at the experiment which got the best results. I don't remember what the code for that result was, but I can look in the hyperparameter file, which tells me what configuration I ran. And so, while this is a bit, well, you can think of it's unnecessary, but I can tell you from experience that you really should do this. And the other hand is if someone else wants to try your code and tinker with your code, then he can look into the hyperparameters file and really quickly change like the data input directory or where the checkpoints are stored and he doesn't have to look through all your code and find, okay, where does he save his stuff on slash magic slash stuff with deep learning. And so it's far easier for someone else who sees your code on GitHub to use it. And so then another pattern I want to propose is functional programming. Many of you might don't like functional programming, but I think in the case of deep learning, it actually fits the mental model. Because I don't know who of you has heard a lecture on deep learning, but the first thing they will tell you is that a neural network is a universal function approximator. Already the definition of a neural network is function. So why do you use object-oriented code when it's a function and you can compose functions? And so then you use a new way about your neural network and your code better matches your thought. Because you can think of like I've shown on the bottom. If you have a detector network, it's composed not of one single network, but actually it's different functions. On the first layer you can have, or on the first function you apply, could be a feature encoder. In this case I chose VGG. And what that feature encoder basically does, it takes in the image and then it turns that image into some set of features and then you can apply the next function and that function transforms those features into the labels. And this sounds a bit like a traditional computing pipeline for computer vision. But actually all you do is make your code a bit more structured so that in your code you can understand which layer of your network is responsible for what to do. And the thing here is that if you, in this case I chose the encoder, but funnily I can swap that out for the Google Inception V3 encoder or something else. And if I have this functional approach, all I have to do is just swap out one function and I can concatenate all those functions together. And so coming from the background that a neural network is actually a function approximator, if your code reflects that function you start to think that way. This might be a bit overwhelming at the first to understand and grasp, but if you start writing your code in a functional way you will see how it changes your thought pattern compared to object oriented programming in this case for the model. And the same is true for the loss. Your loss is basically also a function and your loss just transforms the label that you are getting in. The label is the ground truth and the prediction of what your network said. In the case of ng that would be the 99% and the true for its actually ng. And what this function does it transforms those two inputs into an output which just tells me I'm off by 001%. And you can also stick together if you're getting to more complex stuff, more and more loss functions. And so this might sound a bit scary but we'll see some code in the soon. So let's get back to our first design pattern and actually look at how we can do that stuff. Because to now it's just like ideas and I thought well at a hackerspace I need to show you actually how to do this stuff. And for the data stage the data stage usually you have some data set and your data set consists out of some features. In this case on the left hand you see the images, the raw images and some labels for those images which are basically the ground truth that you want to as a result output for your network. And your data set only ever consists out of features and labels. So there is no other configuration of your data set if you think of deep learning. There is just features and labels. So you can write your code in a way that it supports inputting arbitrary features and accepting arbitrary labels and then you have general code that loads your data set and you don't have to change it for every single data set that you want to feed into your model and that saves you a lot of time and it will make your code more readable. So in this case if you have some kind of loading the data set then you will understand every other way of loading the data set because it's the same basically. And so as I told you they are the same basically. So what you can do then is you can go ahead and create a data lake or some call it the data store and that's basically a central place where all your data that you have in your company put all that data together in a central place with a unified API like I told you it would be something that outputs a feature and a label and you can gather all the data you have in that way and then if you want to do explore some deep learning you can just take that data out of the data lake without the need of complex data transformations and if you look at the existing data sets in the private person who wants to tinker around there are also data lakes for example in the Keras library that you can use to load some data from other people over unified access and then if you have loaded your data this is one important step that most people forget and it's so important it's not really a pattern but I want to point it out in this example have a look at your data is your data making sense do you have enough data is your data balanced or unbalanced for example if I only have images of Angie and then I have one image of let's say Obama and I want to try to learn what is Angie and what is Obama that's almost impossible because I only have one example that makes sense and the other thing that you can detect if you inspect your data is if you have any bugs in your code because sometimes it happens that your image loading is bugged somehow or that you loaded an image flipped or the colors are not in the correct order and if you inspect your data you will see those errors so don't skip that step and it might take some time but it's worth it down this line if you have a data set or a data lake if you want to implement your own because I guess most of you want to tinker some and don't want to use a pre-existing one and so one way I found it it's very convenient and easy to use is that you have a function that loads your data set and this function outputs you a generator and the parameters for that generator and I don't know how familiar you are with Python but I guess generators are a concept that's in many programming languages but there's one thing that's special to Python Python has a weird way of multiprocessing and TensorFlow is a library for Python so we have to deal with that weird way of multiprocessing and as it turns out one of the most efficient ways of multiprocessing is using pooling and for those poolings you can have real multi-threading otherwise it's a single threat and to not run into the scary word you can Google that if you don't know it the global interpreter log you have to make sure that your functions are not relying on one shared object so the way I found that works best in Python is having a generator a set of parameters that is just a dictionary or something that can be easily copied and then having that parameters generate a threat separately and not sharing any data between them and your generator of course because it's multi-thread needs some sort of stride offset and probably it's useful for you to have a function that you can call an infinite time amounts so yeah and what can you do with that generator how do you plug that generator now into your training because TensorFlow has an API which only either accepts one generator or which accepts a completely different time of data and the completely different type of data is TF Records TF Records is basically some database and this is very specific to TensorFlow TF Records is some kind of a database which is designed to be readable very fast and the data can be almost directly transferred to the GPU with the least amount of delay because in modern systems the GPUs are so fast that the bottleneck if you don't write your code carefully will be getting the data from the memory from the hard disk to your GPU and TF Records is a data format by TensorFlow which is optimized for speed so that this bottleneck doesn't appear in your code and there are simpler ways to get your data to the GPU if you're really serious about deep learning I recommend you invest the time into understanding how TF Records work and use them and what I did because I don't want to write that same boilerplate code all over again I wrote myself since I have a unified API with the generator and the parameters a function that's called writeData and this function does write TF Records they're stored on the hard disk and then later I have a function that I call readData which just opens those TF Records and then outputs me a tensor so I only had to write that write and readData once because I realized that there's only the label and feature tensor that you should care about and I can reuse it for every project and the implementation is agnostic to how my label tensor and my feature tensor actually look and so that's also something I would recommend you either use the implementation I found or find your own implementation that's agnostic to your dataset so you can use it for any dataset the same implementation and this will also reduce you a lot of headache because writing and reading those TF Records is not as straight forward as you might think and you will have weird bugs if your implementation with writing and some of these bugs might only appear after you have already trained 5 hours and you don't want to have a bug that only appears 5 hours after you started your work and then you have to start over all again and that just is well annoying so yeah and just recently maybe some of you know there's Google I.O. currently in the USA so it's sort of a difficult spot to give this talk because they are just now announcing a lot of new stuff for TensorFlow and there turns out to be a new guy TF dataset which actually also supports loading those TF Records in a very efficient and fast way and I'm not yet sure how useful that guy that method is going to be but maybe you should keep it on your radar in the future because it's so standard for loading data into TensorFlow and so I just wanted to point it out here and it basically will replace the read data part so now we have loaded our data we have it in some tensors and tensors are basically a variable in TensorFlow on the GPU and what we now can do is we can actually define our model in the image here you see one of the most famous models of the modern age and as you can see that model is quite complicated only if you look at the graph already and now if you try to write some code that's well not really straightforward and then if you imagine down here there's some loss this is the loss of the network in this white box here is another loss and on the right hand you have three losses in those network and if you now don't do the splitting pattern I explained you at the beginning then you will have a super difficult to read code because you will have well somewhere here in the middle a loss then you will have somewhere in the upper end of your network a loss and at the end and if you want to change your loss to fit your own problem then you maybe find the loss at the end but you don't find the two losses and then you wonder why nothing works for you and that's very disappointing so in the model you should just write your model so everything in here that's colored basically and for your model you should look at reusability and that's the part that's also addressed with the functional programming style because if you put every part of your model as one purpose put it in a different function so that if you want to compose a new model that's built off parts that you have already written for other models like for example you want to have a Merkel classifier at the back and you know how to write that and now you don't want to classify Merkel but you want to classify Trump then what you can do is use that same code and use it in another place for your different network so that you don't have to rewrite all the code and so keep an eye on reusability and put everything into separate functions into as small steps as possible and the next thing that's a bit more difficult maybe to explain why that would be important is making your model multi instantiational usually you could think you have one model on your GPU and you use that model and you train it but if you just train your model you won't ever know how good your model is because if you show the model some data and then later ask it what it thinks of that data it's like if you're learning for an for an Exem and you're just keeping going over the same old Exems and then you realize oh well I'm perfect at doing those Exems but actually you just learn them by heart and so to avoid that with the deep learning you need one model that's learning and then you need a copy of that model to be executed on data that the learning model has never seen to make sure that it's like a test situation when you're writing an Exem for example you're seeing data that wasn't in the learning you're having new tasks to solve and we want to test that generalization so you need to be able to multi instantiate your network and there's a simple function call in TensorFlow and that's called reuse weights and all this does is it if your weights are called the same if you call your two networks the same then they will share the weights across them and so you can instantiate your model one time then you can instantiate your model a second time and you could even do that a third, a fourth or even more times and then one last thing if you remember the 2800 lines of code at the beginning make your code simple write simple code, don't write too complex code and also think of your network not being your network if you publish it somewhere on the internet for example on github but think of it make it easy for others to wrap your network to use your network give them API to use it for example if you have a special encoder just put that encoder in an extra function so that they can use that module separately because we are software engineers basically and we want to engineer solutions for problems and so write simple code and wrap it into other networks to reduce bugs and now finally I'm going to show you some code on how to do that I won't say that this is the gold standard but that's the way I found most useful and it works for me and so I basically have a function that says create model it has some input at the top the input tensor that's basically what you give in to your model so you don't have a mode that's basically telling the model if it's in training mode or evaluation mode because some layers might be different for those and then you have your hyperparameters from the hyperparameters file which configure your network so you don't have a constants in your code but you actually use your hyperparameters to trigger your if statements in your model and then there's some technical bits so I'm not going to go into too much detail why you use it but just use it because if you're inspecting your network in a tensor board and you want to see the structure of your network you can then there search for this variable scope in this example MNIST network and you can easily find again your network because if you're having large networks or even medium sized networks it tends to get messy in that view and then the next thing the scope reuse variables basically tells you if I'm not in the training mode but actually in the evaluation mode use the variables of the training network and then at the bottom you want to define your network in this case I left a blank here because that would be a little bit more code and basically what you then want to do is that I found the easiest way to make your model reusable which contains every important layer of your network and one important layer is your logits that's the layer that's before the softmax layer then of course the softmax and it's the probabilities that your network outputs but also if you have a feature encoder in your network it might be the feature encoder that's an important layer that you want to return in that dictionary and what this enables is if someone else is using your code he can just call the create model method and then he can print that dictionary on the console and dig into it and look what tensors do I have what shapes do they have do they work for my problem or not and so this is a very easy interface I think everybody can implement dictionaries it's nothing too difficult or scary and so I think this is a simple way of doing it and you don't need complex solutions and also you have to think of we want to write kind of functional code so we want to output every side effect that we have and the model is the side effect of the create model function and so then on this slide is what was on the previous slide the to do it's actually quite readable but you don't have to read that the code is on github if you're interested in the details but basically this is the complete implementation of a network and the most important thing on this slide is that if you can you should use the tf.layers API the tf.layers API is an API that's well tested and you have lots of lots of predefined layers that you can just stick into your network for example convolution layers dense layers, max pooling layers and you don't have to reimplement them with additions and multiplications from scratch and it's well tested and much used library so then we have defined our model we have our create model function and that model was the code from the previous slide and now the next step would be looking at the loss at evaluating how good our model actually is and the loss function you see on this slide is the loss function from the previous model and if somebody of you has listened to a lecture on deep learning this is the kind of an optimal curve how it should look it should go down at the beginning very very fast and then it should stay low and as you can see here our model starts to converge at about a thousand iterations already and then later on it stays low no matter if I'm feeding it the training data or the validation data and the validation data is never seen except for the validation so it's not in the training process so in this case basically if you have such a loss function your model is well prepared and has generalized in a good way what you shouldn't see is that your loss function your validation loss is going up here at the end and that's the typical case of overfitting you looked at too many examples and then your network just knows all the exams that it has already written but doesn't know how to write a new exam so it doesn't really learn how to add 1 plus 1 but it learned that 1 plus 1 is 2 and if you give it 1 plus 2 it wouldn't know what the solution is and so how do you engineer that loss function and that loss function is very important for your network and you have to engineer it in a way that your network will converge in a good way and what it basically is it's the clue between your model and the dataset because in the model definition we didn't have any knowledge about our dataset except that we have an input tensor and that input tensor was the image but we don't know anything about the labels yet and so what the loss does it glues together the predictions of your model with the labels of your dataset and those are the 2 inputs that your loss function gets and it doesn't get any more inputs than the parameters and it contributes a lot to the quality of your model in the case of MNIST we will see that the loss function is quite simple but if you are training more complex tasks like for the case of autonomous driving then your loss function will not be simple but it will be like 100 or 200 lines of code to actually implement a loss that works well and that's robust to different situations and so again for the loss as for everything I don't know why I have to tell you that but people on Github seem to forget about it reusability is key you want to write your loss once and then apply it to as many models and datasets as possible and there are some losses that are supported by tensorflow like the cross entropy loss and then the L1 loss or L2 loss that's the L1 norm they are not directly implemented by tensorflow it's like two lines of code but you have to write them always again so you might want to wrap them in a function that's called L1 loss and just use that all over the place again and then there's more complex losses like alpha balancing or focusing the loss which are state of the art in papers and those losses are a bit more involved to write and you don't want to rethink how to write them every time again and if you have a network that has problems with learning because your dataset is a bit unbalanced you might want to consider a focal loss and then if you wrote it in a reusable way you can just stick that focal loss in there and 5 seconds later you have implemented your model with focal loss and you don't have to read the paper again understand how it worked implement it again and you will spend like one or two days until your focal loss works and if you don't write it in a reusable way you will waste those two days every time you write it again and also the problem is that there is an open source implementation by the authors of the paper but it's in that cryptic form that I showed at the beginning so you can't use it in your own code because you don't understand how it works because it's not reusable and so if you discover a fancy loss by yourself write it in a nice contained function as least inputs and outputs as possible and so how does the loss function actually look like if you want to create a loss in the case of MNIST it's quite simple the inputs are our model and the labels why do we need the model and not only the predictions it's because as we have seen in of inception our model doesn't only have one output but it had three outputs also relevant for the loss and so we need the whole model and all its endpoints and then what we do is the first line is just putting the node the mode of the network in a nice string format so we can use that for our output printing and then the next step is in the case of MNIST we have to make our labels fit the output format of the model and then we can calculate a softmax cross entropy with logits that's the tensor flow implementation of the cross entropy loss for this talk it's not important to understand what actually this function is you should put some function in there in this case it's just the cross entropy but it could be some more complex function like the L1 loss or the focal loss because you feed your network batch sizes you don't feed it one image at a time but you feed it 16, 32 or 100 images at a time and so you get a 100 loss outputs but your loss only can be one number and so what you have to do is you have to reduce that into one number and that's the loss operation which you can see here and that loss is then put into a tf-summary scalar and that tf-summary scalar is basically so that tensorboard can nicely visualize your stuff and the matrix is another way of putting it into a way that you can visualize it we will see later what the matrix are useful for and so the two things that your loss basically returns is first the loss operation and then you have to optimize that's how bad your network is the error and then the matrix those are you can don't only have one loss but you can have all kind of other matrix that later will tell you how bad your network is performing because the loss is only one number and as you might already know from traditional debugging it's good to not only have one number of your code and the optimizer happens to be in TensorFlow the simplest part that you can imagine is just one function call there is the tf-train library where there is the optimizer all common optimizer are implemented there they are in a nice clean interface you just give them the learning rate some optional parameters and then you tell them what they should minimize and in this case they should minimize the loss so the optimizer is no magic at all so now we have seen how we can implement those four stages the first stage was with tf-records the second was the model cnn the third the loss that we've seen and the optimizer was that one line of code and then with that optimizer we have to call that optimizer and the way TensorFlow works it first creates a compute graph and that graph is not executed it's just defined nothing happens so you have to execute that graph and what you can use to execute it is the tf-estimators API that's pre-written for you but it doesn't support very well the way of evaluation and training at the same time but they're changing the estimator RP right now so maybe in a month or so the estimator RP might be a very great way to do this so I wanted to point it out here so here's some code that you can also see in the GitHub but for the sake of time I won't go into detail this is just the proof that all the patterns that I described to you earlier are compatible with the estimators framework and the way I did it was because I didn't like the estimators and they weren't giving me the flexibility I started customizing them via callbacks or I wrote them in such easy way that I could reimplement parts of it as I need it so my implementation of the estimators is just 77 lines of code to train a network and I guess 77 lines of code is quite easy and quick to understand instead of like 1000 lines of code that are split upon dozens of dozens of files like the estimator and what I have in this estimator package by myself is just a train and evaluate method and an easy version of it that's even simpler we'll see an example of the easy version and so what does this train and evaluate do and what does the estimator do what they do is they merge those summaries that we defined in the loss if you remember we had that tfscalar thing where we told it we want that loss to be visible to tensorboard and they have to be merged so all those values and then they have to be written into a log file that's the first few things you have to do and then you have to save your hyperparameter file into the log directory because if you don't do that the hyperparameter file is completely useless because you have it when you start the train but not later when you look at your results so you should that third step is the really important step and that's the step that the tfestimators API doesn't do for example and then the last step is a training loop which is basically just a for loop which calls your training operation and your summary operations and maybe some callbacks that you passed in and so here's all the code that you need to write in your network it's that clean and simple you just go at the top and define your you import the helper functions the easy train and evaluate the hyperparameter loader i've pre-implemented them i will put the github link at the end and then you import your own create model function that you have written somewhere for example for MNIST then you import your loss function and after that you then load your hyperparameters using the loadparams method and you call the easy train and evaluate using your hyperparameters your model and your create loss and i guess everybody in this room even if you don't really program python will understand what those lines of code do and they're simpler and less code than the 2000 lines of code we saw at the beginning and also we have seen how the complete loss was implemented that were like 10 lines of code we have seen how the model was implemented and so all those pieces it's quite little amount of code that you have to actually write to do deep learning with those patterns so you don't need 2800 lines and so to sum it up what patterns i found the most important there are the first three patterns the first was splitting everything into data, model loss and optimizer and treated as four separate parts so you can reuse them the second was functional programming because a neural network is a universal function approximator keep that in your mind when you're developing it and using a hyper parameter file and then when looking at the data set we saw that you should use the tf records they are much faster and that you should probably in the future consider using the data sets once the api gets stable and then for the model you should use that reuse variables method so you can have a training and an evaluation model to see if your model is actually overfitting or not so that you know if your model generalizes well or it doesn't and then the sixth step that was a bit between the lines is creating summaries and matrix and those summaries and matrix they are plotted in the tensorboard and you can also plot them in mudplotlib and make them available to you to inspect what your model is doing because those summaries they are basically your debugger for what's going on in the neural network you have no chance to look into it except for those summaries so take them seriously and now after I've talked so much about deep learning here's a quick disclaimer by Occam's razor the simplest solution is almost always the best not every problem is suited for deep learning some problems are better to use with linear regression in the first step when I told you to analyze your data and you realize it's some kind of a linear problem use a linear regression don't use a deep neural network it's overkill and the same goes for decision trees, K&N there are so many normal deep learning methods consider them first and look at your data what fits your data the best and if none of those traditional approaches fits then the solution is probably a deep neural network and so there's one last thing that I showed you is all available on GitHub in my GitHub repository StartTF I have on the main readme the examples, the exact same examples that we saw in this slide and there's also the implementation of the write data the read data that you can dig into into detail if you are interested or just reuse it if you want to and if you want to know how to actually implement something visit my workshop later and so with that I'm open for questions wow, I'm deeply impressed I understood nothing are there any questions come on no one understood anything there's no dumb questions you can also ask me anything you want if you want to know something about deep learning in general and not how to program it okay so in my experiment okay okay usually I don't have enough dataset on the internet is there any way that we for example I only have hundreds samples or thousands samples not so much as on the internet how is there a method that we can get more samples or yeah I understood your question what you can do in the first place is when we're looking at that image for example when you're looking at your data and you realize you don't have enough data or if it's unbalanced or something you can try data augmentation and the most steps there are rotating your images a bit moving them around cropping them, scaling them maybe changing the illumination and if your data is unbalanced you for example if we only have 10 images of someone as than Merkel we can duplicate those images or reduce the number of images of Merkel and try to balance our data and then there are of course some things that you can do with your model if you say let's have you only have a thousand images for a segmentation problem what you can then do is and that's why reusability is so important um you can for example go ahead and use the inception model and you take this model and cut it off right here just before the the the final classification layers you take those weights that were trained on millions of images by Google for you for free and you use that encoder and stick it in the front of your and solve your problem and so that's actually a good question because it demonstrates how important modularity is because you can just now use a module written by someone else and import it in your code and you should also check out tensorflow.hub it's a new API that was just released today, yesterday or somewhere around that time so any more questions what about Keras how does that how that can can that be used these high level APIs or how do they fit into this schematic yeah Keras is an API that's designed to make but the problem with Keras is that it makes deep learning simple because you can't create your own optimizer in such an intricate way or your loss as you can with bear tensorflow and what you can then do is specifically for that question there's in Keras you can use all those layers like the dense layer and instead of putting them into let's say a sequential model you can just give them at the end as an extra parameter the input of the previous layer and then as the output you get your next layer and then you can stack them together like normal tensorflow code and you can then put them right in this slide where the to do is you put your first Keras layer and as an extra parameter you feed in your input tensor and for the next you and you create your model in this way and then the last output is fed into the prediction step but you can't use the sequential model wrapper so any questions anymore okay you showed us how to separate the encoding after training we have some nets that detect several features and they are currently going on for example low bay to reuse parts of a train model to use the features are you working on that as well or do you have any ideas on that how to play that out in the future what you can basically do is this create model function I've shown it here as if you have one create what you can do is you can call another model function inside your create model function and this way your network the feature encoder part we are talking about feature encoding you have that as a model which you then input here as a create model function you just call that one and then you have your separate part on putting heads at the end if you want I can show you some code on that later I have code for that and so that's actually one of the things why you should do these create model functions because you can then do that very easily without any hassle is it possible to extract the classes also next to the weights and so on or I forced to provide the classes I'm sorry I don't understand your question can you rephrase it somehow or you can also ask in German if you want to ich möchte versuchen problem zu lösen von dem ich die Klassen vorher nicht kenne die möchte ich zusammen mit dem neuronalen Netzwerk trainieren ist es möglich oder ist es aus Sichtlos das ist eine gute Frage also das Framework ist nicht darauf spezialisiert, dass man unbedingt die Labels haben muss also man muss ja im Prinzip wenn man hier diese App anguckt, müssen wir nur irgendein Model zurückgeben und in diesem Model haben wir in dem Dictionary irgendwelche Tensionen drin stehen wo ich mir bei TensorFlow gerade nicht so ganz sicher bin wie man da Tensionen erstellen kann deren Länge man gar nicht kennt also würde ich mir vorstellen dass es vielleicht ein TensorFlow Problem ist aber im Prinzip ist diesem Pattern dieses Pattern das so zu teilen ist es egal was das Model ist und wie das aufgebaut ist und auch dem Los ist egal wie das Model aufgebaut ist es muss nur irgendwie wohl definiert sein oder es muss nur irgendwie TensorFlow formulierbar sein und ich weiß nicht ob man das Problem in TensorFlow formulieren kann ehrlich gesagt weil auf der Grafikkarte braucht man ja fester Arraygrößen und das ist dann ja schwierig wenn man die Labelgröße nicht kennt wenn ich die Frage richtig verstanden habe ich könnte vielleicht einschränken wie viele Labels maximal sind aber das was ich gerne wissen würde ist welches Datum welches Label bekommt das würde ich gerne zusätzlich zu den neuronalen Netz trainiert haben wollen ist das möglich also das wäre dann im Prinzip ein Ansupervised Learning Prozess das müsste eigentlich auch damit möglich sein ich muss zugeben ich bin kein Experte bei Ansupervised Learning alles was ich in der Regel mache ist Supervised Learning deswegen bin ich dazu den Details leicht überfragt aber ich sehe nicht warum das mit dem Pattern ein Problem haben sollte ich muss nur sagen ich weiß nicht wie man es in TensorFlow dann tatsächlich implementieren würde also eine Variante wäre natürlich dann mit Autoencodern aber im Autoencoder habe ich ja auch wieder ein Label also wenn ich zum Beispiel ein Autoencoder mache wo ich lernen möchte zum Beispiel ich gebe lauter Bilder von Katzen rein und ich möchte jetzt implizit Features von Katzen lernen was für Features eine Katze hat und weil ich dann machen kann die Autorencoder schreiben und diese Autorencoder kriegt als Input der Bilder von Katzen und den Output den er produzieren soll sind auch wieder Bilder von Katzen und das ist ein symmetrisches Netzwerk und was man dann nach dem also der Trainiervorgang ist dann wieder Supervised weil ich habe ja Input und Output sind bekannt das heißt ich habe Katzen und Katzen also Input und Output als Katze und was ich dann am Ende machen kann ist ich kann meine Bitte durchschneiden und dann ist der Layer an dem ich es durchgeschnitten habe und dahin ist dann mein Feature Encoder und da habe ich dann meine Katzen Features und dann kann ich nachgucken da gibt es zum Beispiel DeepDream kann ich gucken was diese Features bedeuten und kann gucken dann kriege ich zum Beispiel wenn ich das eine Feature aktiviere kriege ich dann, das wird nur aktiviert für Inputs die zum Beispiel Nasen sind oder die die Nasenhaare sind und so könnte man also das wäre so eine Art von Anzugbeweist Learning die mit der ich mich ein bisschen auskenne aber technisch programmiert technisch ist es halt auch wieder ein Superweist Problem ich habe mein Anzugbeweist Problem so umgeschrieben dass ich Label und Feature beides kenne A practical question your workshop where is it taking place in an hour and is it going to be in English or all German I planned on doing it in German but if there are some people who only speak English I can consider doing it in English too that's not the problem with the language the original plan was holding it in German and for the answer where it is we have the far plan here now it's in the half gay workshop room I don't know where that room is yet I have to explore that myself but yeah wow thank you very questions thank you for this talk applause