I'm going to be talking about generics and the reason I'm talking about generics is I believe they can help us to write more understandable, robust and reliable code. The reason I believe that is because I've worked in other programming languages where generics has been part of the language and I've seen the benefits. Hopefully by the end of this talk you'll agree with me that they would be useful. Now, generics isn't part of the PHP language at the moment and I doubt it will be for a while. But what I am going to demonstrate is how you can use existing tools to almost give us the benefits of generics now. I just want to make sure you're going to invest an hour of your time wisely. I've marked this as an intermediate talk and my assumed knowledge, my starting point is that you understand the type decoration, the type hint, in that method signature, that function signature for process to understand what the user is doing there. You can see that if we use this code there would be an issue. That's kind of my starting point. Conversely, my ending point, if you already understand the annotations here, things like app template and app return class string and you already use some or PHP stand, maybe you won't get much out of this. But if you're somewhere in between, stay on and hopefully you'll learn something. So anyone wants to move now is a good time. Oh wow, brilliant. OK, so the journey we're going to take, there's four main areas. We're going to talk first of all about what are generics. Then we're going to do a deep dive into generics. We're then going to see how we can use them now and I'll finish with a few thoughts. So what are generics? I just want to talk very briefly, just before we do generics, about the benefits of type safety. So specifically with this example, there are in the context of this talk three areas, three benefits of this. The first is that we have, by having that type declaration of user, it gives other developers clear unambiguous type information. We know if we're going to call process, we have to call it with a user object. We have a run time check. If we call process as we are here with something, a string, anything that's not a user, that code will fail and it will fail when we try and call process. So we've got a run time check there to protect us against mistakes. But best of all, we've all just done static analysis right now. We've looked at that code, we don't have to run it and we can see that there is an issue. So static analysis is another major benefit and the reason these are all benefits, if we think of the cost of a bug, generally the later we find a bug, the more costly it is. So if we have some kind of continuous integration that may be running some kind of static analysis tool, it could potentially find a whole class of bugs to do with type errors at the time we run CI. And better still, if we have an IDE that can give us real time static analysis and providing we've added type information, it can tell us about mistakes as we're writing them. So we can fix mistakes almost at the point we add them in. So what about generics? Imagine we have a class that represents a queue. Now a queue could contain anything. It could be delegates waiting to get their lunch. It could be jobs waiting to be executed. It could be a queue of strings, a queue of users, a queue of books. So for this very simple queue, we've got two methods. One is a way of adding something to the back of the queue and getNext takes a thing from the front of the queue. And the issue we have here is at the time of writing the queue class, we don't know the type of data that that queue will hold. And there's no way of knowing because queue could be used in many different ways. On one day it will be holding a string and some other time it will be holding a book and some other time a user. So if you look at this function call to get queue, it returns our queue, but we don't know anything about that queue. If we think about what types of entities are in the queue, that's just not known. We'd have to go and look at a lot more of the code to find that out. There's no runtime check that we can add in to make sure that our queues are holding the correct information. And we can't do any static analysis. We can't look at this now and know if there's an issue or not. So one way around this is we could make a typed queue class. When we construct it, we record the type of the object that we want to store in the queue. We also construct an instance of the queue behind the scenes. And then our add method, when we add the item, the first thing we do is we check that the item we are adding to the queue is of the correct type. If it isn't, we throw a type error and if it is of the correct type, we add it to the queue. This is our private queue behind the scenes. So this typed queue, there's no way we can get something in that is not of the right type and our getNext method is just a wrapper. So we would instantiate this queue like so and we would say I'm making a typed queue and in this case, I want it to contain objects of type user. So this means if we ran some code like this where we're trying to add something of type user to the user queue, this would all be fine. But conversely, if we tried to add a string to our type queue, our user queue, this would fail. So if we think about this as a solution, the same code works for any type. We need another queue and that holds books. We just create another typed queue and we do book, colon, colon class in the constructor. We have runtime checks. If we try and add something to the queue that is not of the right type, then we will get a type error thrown. But there is a problem with the solution. We can't do any static analysis on it. We can't just look at something without looking at the context in how that queue was instantiated to know what type this queue is expecting. So there is another solution. We could make a specific user queue object. Again, behind the scenes, it has this generic queue and we've got our add and getNext methods. But importantly, we add type information to the add and the return type. And we've constructed it like this. User queue equals new user queue. And again, if you try to add a user, that would all be fine. But if you try to add something that wasn't a user, it would fail again with a type error. So let's think of the pros and cons of this approach. Unfortunately, the same code doesn't work for any type. So we have to rewrite, we have to create our own class for each type of data we want to hold. But we do get runtime checks and we can apply static analysis to this. So both these solutions are non-ideal. Now in languages which support generics, we can do something like this. We can have our queue and then we can put this thing in angle brackets, a T. And the T just means that at the time we write in the queue, we don't know what type this queue will hold. It's kind of a template for later. And then when we add, we say we don't give the specific type, we say we're adding something of type T. And our return type is something of type T. So at the time we write queue, we don't know how it's going to be used. But the time we instantiate queue, we will give it that missing information. So instead of creating a queue like this, in the other languages that support generics, we can do something like this. So we're saying we're a new queue and in the angle brackets we're saying it's the type user. So that means behind the scenes, wherever we had T, that gets replaced with the type user. So as before, it will correctly know, it will have the runtime checks to know if we're adding something that's a type user. And if we try and do something that isn't a type user, it will fail. And the benefits of this solution is the same code works for any type. We only have to write the queue class once. We have runtime checks and we can apply static analysis to find bugs upfront. So here's our three possible solutions. The first two are possible with PHP today, but they have some problems. The final one, it would be great if PHP supported it. But it's not yet possible. So you might be thinking, I've got a bit of deja vu. I'm sure I've seen something like this before. What about when we do things like, say, at return and then we've got user open and close square brackets? Is that not generics? And it kind of is. So if we look at this code here, we can see get users is returning an array. And from the dot block, we can guess that we can conclude that what we're returning is something of type user and an array where all the values in that array are type user. So when we iterate over that and we're calling process user, we would guess that the object we are passing to process user is of type user. And looking at this, we would say, well, that's all fine. But the problem is this is not actually checked by the language at all. So if we have a function process users, which is expecting to take an array of users, and we call process users passing in an array where one of the items is a user and the other item is a string, the code will not fail at a time we call process user. It might fail later, but it won't fail at this time. There will be no type error. So that's a bit of a shame. You might say, well, are there tools that can enforce these checks? And the answer is there are tools that can enforce these checks. Tools like SAM, FAN and PHP STAN. If you're not using these tools, I would strongly recommend investigating them. They are incredible. So if we pass this code into SAM, and it would look at it, and it would find some errors, and specifically it would highlight the error here and say we're calling process users, and we should be giving it an array where each entry in the array, or each value in the array is a type user, but it's correctly highlighting here that it isn't. So how does this work with our queue example from earlier? What we can't do in PHP is we can't use this way of documenting templates because it's not syntactically correct. So if we gave code like this to PHP, it would just fail. It would fail at parsing time. So instead, we move the type information to dot blocks, and for the class we might say at sign template and then T. So this is telling our static analyzers that the queue is a template. It will hold something that we don't know yet. We will find out later. And then we do at pram T, and then we do at return T. So what we can't express in the language, we have to express through dot blocks. And then how do we tell the static analyzer that we're dealing with a queue that contains users again in the dot block? We can just do at var queue and then the angle brackets with a user. So this tells the static analyzer that our queue object is actually a queue of users. So let's have a recap. Sometimes we write code for objects like queue where at the time we write it, we do not know what information, what type of data it will hold. So we can tell static analyzers later about this by using the annotation at template. And when we create our queue, we give the static analyzer the information it needs again in the annotation. Unfortunately, this isn't supported by PHP at the moment, so instead we use tools like SARM and PHP Stan to validate that everything's correct. Okay, so that was our first section. What are our generics? We're now going to do a bit of a deep dive into generics. We're going to look at five areas. Sometimes this gets a bit complicated. If you get lost, don't worry, I will share with you at the end a Git repo which has fuller examples of all these code snippets so you can actually have a look at them for yourself. So stick with me, but if you do lose it, then just come back for the third section where we'll show you how this all fits together. Okay, so the first of the five, I'd like to talk about our collections. So imagine we have some code like this. We've got a foreach. We're calling the get employees method of the business class. We can see that this get employees, it would turn something of type array and we can look in the dot blocks to see that it's an array where all the values are of type employee. So we know the type of employee in our for loop is of objects employee, but we're not sure what the name is. It would either have to be an int or a string in this case. But we're calling welcome passing in the key of whatever's coming back from get employees and we are expecting that to be a type string. Now an array could have keys of either ints or strings and it could actually have a mixture of both. So looking at this code, we can't guarantee that it's correct at static analysis. Maybe it's correct, maybe it isn't. And if we gave this code to Sam to have a look at, it would complain, it would say that we're calling welcome and we don't know what type, we cannot guarantee that we're calling welcome with type string. So how do we get around that? What we can do is we can change the information we give slightly. So instead of saying out return employee open and close square brackets, we can say out return array and we've got the angle brackets string, employee. And what that means is the array keys are all of type string and the array values are all of type employee. So that means we now know that name is of type string, we're calling welcome with something that is of type string so we know that is all correct. And our static analyzers would look at what's going on in the get employees method and if you ever try to put something into the array that it returns where the key isn't a string or the value isn't an employee, it will complain about that. So with arrays we can express the information we need in a number of ways if we're just interested in what the value type is, we can just do something like this. We have array, angle brackets and v, it's just telling us what the type of the values are. If we're interested in the type of the keys and the value, then we have to give both the key and the value types. Now for arrays the key values can only be of type string or type int. And of course we might be using some kind of collection library, for example Drin's collections library, they have an array collection object and you would give the additional information about that array collection in the same way here. And I've done things like t and k and v, they're just conventions, you don't have to use those, they're just like variable names, you can name them however you like but the convention is there's normally t for key and value, k and v, that's where they came from. So that's the first thing, that kind of covers off arrays collections. We can also use generics just on functions, so we've got a very simple function here, I think it's only value really is to demonstrate generics, it takes a value and then just returns it. Again we've got template t, so at the time we've written mirror, we don't know what it will be reflecting. But we do know that whatever type value is, is the same type that we return. So if we have some code that looks like this, we're calling mirror, we're passing there five, so we're passing an integer to it, so we know that the value we get back from mirror will also be of integer. So another slightly more advanced version and perhaps something that has a slight bit of use, we could have a function called as array, it takes a value and it returns an array of those values. So again we can see that we're passing through it something that's an integer, so we know that t in this case will be of type int in the case it's used here and we can see that what we're returning, we're returning an array of something of type t, so we know in this case values will be an array of integers. There's also something called a class string, so we can represent classes, we can either have their fully qualified name or we can take the class name and do colon, colon class and behind the scenes that results in the same string and my preference is always a second one on that list. So imagine we have a dependency injection container, we're only interested in for this one method which is called make, we pass in the name of a class that we want it to create and then it returns an instantiated version of that. So if we call this code here person and we're passing it person, colon, colon class, what we expect it to return is a person object. The issue we have here is there's no way a static analyzer can assert that's really happened. I kind of just have to hope it's the case. The best we know here is person is a type object. So once again we can add the at template t and we have this new thing here which is class string and then angle brackets t. So that means we're expecting the string that we pass in not to be any old string but a string that represents the fully qualified name of an object. And what this is telling us is that t will be whatever that object is and we're returning something of that type. So now we've got enough information to say to our static analyzer we're asking it to create something of type person, person, colon, colon class so it knows in this case that person will be the dollar person that the make method returns will be in this case of type person. And we get that from here. Another thing we can do is extend templates. So in the same way in PHP you can extend classes, you can extend classes and also extend the templates. So a place where this is done quite commonly is repositories. So if you're using doctrine repository you might see something like this. So our abstract repository class, it could be of any type of a repository you could hold anything. It could hold users, it could hold books, it could hold talks. And we've got a couple of methods on here. The first is find all and that just returns an array of all of the items in that repository. And then we've got a find by ID. And just like, so in this case we're going to return something that's either a type T or if it's not a repository it'll be null. So we can do that just as we would in normal if we knew the type in advance. So we've got our abstract class repository and we know it is a repository and will hold something of type T which we don't know at the moment. Now imagine we have a use of a repository, the use of a repository extends repository and the interesting bit here is in the dot block. We can see we've got something that says at extends and what this is telling our static analyzers is that we're extending the repository and we've got the user in angle brackets so that is saying that the repository T in this case is a type user. And what that means is the find by ID that was a method in the parent class and we know that what we will be returning is something user in this case will either be a type user or it will be null if there is no user of that record ID. And that's kind of how we get that information, that's how we know the type of T of user. So there's one final thing I'm going to do in this deep dive, I'm going to talk about how we restrict types and you can probably explain the whole of generics with cats and dogs and animals. Imagine we've got an animal, we've got dog which extends animal, the dog can bark, we've got a cat that extends animal, the cat can meow. Similar to what we had last time, we've got in this face called animal game and again we've said it's a template T and then we've also got this single method which is called play and it takes something of type T. So we've got dog game and this instead of extending animal game, animal game is an interface that implements animal game and when we were extending classes, we had at extends, now we have at implements. So this is very similar to our last example at the moment and what this means is that we've got an animal game and it's the dog specific version of it. So when we play with our animal, we know in this case that animal must be a dog so it's okay to make the dog bark. And if we did something similar where we've got again a dog game, but we mistakenly think that the animal is a cat, our static analyser will tell us we've made a mistake here because a dog can't meow. But here's the new thing. What if we have a class called car which has nothing to do with animals and then we say car implements animal game. It doesn't really make sense and we've got our implements animal game car. So the problem is car isn't really type of animal but actually our static analyser wouldn't complain at this moment in time. So what we need to do is we need to restrict the type that T can be to just animals in our animal game interface. So that's what this template T of animal, that means if we try and instantiate something, if we try and extend something where the animal is of type car, it will go well car is not an animal so this is not allowed whereas if we try and extend something that is of type animal, in this case cat that is allowed. So you might be thinking well that's all very well but how does this actually help us? And it helps us just like it did with the type information on the process method. We are communicating additional type information. So this is to other developers and in this case to tools like static analysers. So if we look to this we know here that our array, the array key is of type string and where whole and the value is of translation. So that's useful to other developers. It is useful to static analysers because they can find bugs and sooner and remember the sooner we find bugs are to fix. So if we could get generics into language we already have the type system we currently have offers benefits in finding bugs earlier and the bit that is missing currently of the PHP type system is the generics. So this is why this talk was here and what I hope you agree with me now is that using generics can help us write more understandable, robust and reliable code. So if I lost you in the deep dive if you can come back now we are going to talk about how we are actually using generics now. So as I said it is not possible at a language level. So typically what we do now is we write code and then we can run it. But all we need somewhere in our process is this extra step where we do the static analysis.  If we want this to work if we want our static analysis to give us correct results we have to provide type information for everything including generics. And we can use SAM or PHP Stan or FAN probably for this kind of, I think SAM is probably the most advanced of the tools when it comes to type checking but these tools are constantly evolving so maybe that advice won't be right next week. Installing SAM is pretty easy the other tools are similar you just do a composer require in fact all of these tools require some kind of config SAM you can just create it yourself you just run it the first time with a minus i to initialise you tell it where your code is in this example it's the source directory and you give it a number and I think SAM is between one and eight or one and seven and one is the strictest and if you want to do what I'm advocating here these tools have to be run on their strictest settings and then you just run it. If you do this on an existing code base you might get a lot of issues but anyway and the other tools are very similar to this so the next thing we have we might have our perfect code base we might have put type decorations everywhere we might have put in all the dock blocks we need to give all the information we require about generics and the third party code probably doesn't sit in isolation we are probably interacting with frameworks and third party libraries and third party code so what do we do with these things? there's good chance that some of these frameworks and third party codes won't have the additional information you need to safely do static analysis and for good reason a lot of these a lot of this is quite new so maybe the frameworks were written before this became not even mainstream but before a lot of this was around and third party libraries might say well I don't want some of these annotations because they're not really part of the language there's not even a PSR that supports them so I don't want my code polluted with that so those might be reasons why people aren't going to engage or don't engage the best thing we can do is try and get those who are writing third party libraries on board some are already doing this, doctrine, PHP units and the web moat site assertions library are all examples now if you go and look at their latest versions they have these additional annotations in if you are going to try and get third party libraries on board before you do any work at least engage with the maintenance to see if there would be any interest in doing that and I think it's a two-step process the first one is to add in the additional annotations and even better it would be to introduce the static analyzers to their build process now if for whatever reason third party libraries aren't going to get on board or for whatever reason then we have to work with that so one example I had is I was using a tool which you gave it, you could give it ints, you could give it strings you could give it anything and it returned a hash of whatever you gave it and if we look at the if we look at at this, the way I was using it I was always passing an ID and if we look at the method signature for encode it looks like it doesn't take anything it's not expecting any parameters at all so a static analyzer would say I'm pretty sure you're not calling encode correct so one way around this is we could create a clean int space to whatever we're using so we've got our clean hasher we inject the actual hasher in Instructor and then we have our own encode method where we're specifically saying I'm expecting something of type int and I'll give you something back as a string and then this means that when we call this our static analyzer will look at it and go yep I'm expecting clean hasher to be called with an integer that's what you've given it and it is returning a string so this is one way of approaching our interface between our perfect code and other bits of code which from the perspective for static analysis is not perfect another way we can do this is using stubs so imagine our dependency injection example from before if we looked at our third party library and this is all the information they give we need to provide more information so we could make a stubs file and we would save this in our project and maybe like a stubs directory so something away from the source code and we would copy out the namespace and the class declaration and for every method we're interested in we would copy it into our stub and we don't give it any implementation details at all but we can give it additional information in the dot block and what this means is the static analyzer will look at it, you would configure Sarm and say this is where you'll find all the stub files and Sarm would take this over the other information that it finds but you have to be incredibly careful doing this because if you put some information here that is wrong then the static analyzer will take that information so it's garbage in, garbage out we have to make sure that we're giving the static analyzer the correct information all the static analyzers allow you to write plugins this is probably needed where some of the frameworks have got a bit of magic going on the problem with this is they're specific to each tool so if I wrote them on for Sarm it wouldn't work on PHP Stan and compared to all the other solutions I've proposed this is certainly an order of magnitude more difficult so kind of as a summary for this section we would add in static analysis to our process so we'd write the code, we'd have some static analysis and then we'd run the code and our static analyzer needs to know the types of everything so if we are interfacing with something else like a framework or a third party library we might have to give that information somehow so I'm going to wrap up with some thoughts so the title was PHP Generics Now Brackets Almost and you might be asking why almost because it seems like I've proposed something but I don't have the power of generics now and if you tried doing what I've suggested there are a few friction points and probably the biggest one is that the IDEs don't necessarily understand this additional these new types of annotations so for example here I've got the way of expressing the key and the value information of an array and the problem with this is that this tool doesn't understand that and what's worse is before it knew at least the type of the value now it doesn't even know that as well and this is a bit of an issue really there are some ways you can mitigate it so what you can do is you can use the normal at return annotation to say this is something of employees and you can give the additional information that the static analyzer requires by having a SAM-return at PHP-stand has PHP-stand-return and actually SAM and PHP-stand understand each other's annotations you can do this in a lot of cases but there are times when you can't so this doesn't completely solve the problem essentially what we really need is a standard for this and once we've got a standard all of the tools can adopt that standard and that will make creating code easier and then the standard will also help all the libraries get on board as well so thoughts about how we could implement the standard the best one would be full language support there's a link later which goes into details why that's potentially difficult maybe a PSR or maybe some other way of all the interested parties sitting down and saying let's agree this is how things are going to be done more crazy blue sky thinking which we should probably completely disregard but maybe the language there are RFCs for implementing I say proper annotations language level annotations in PHP and if that happens then we could properly define these additional annotations and then the static analyzers could use them so this would be better than a dot block solution because they would be more formally defined so in this case instead of having a dot block that is one of the proposed methods of doing annotations in PHP an even more crazy suggestion is you could add the syntax of generics to the language and you would take that information just as far as the AST so let's explain that just quickly when you've got your PHP code there's a number of steps it goes through before it gets executed so at a very high level you've got your PHP code it then goes through a tokenizer stage and that would need modifying if we were going to add it into the syntax of language either of these two solutions from the tokens you create something called an abstract syntax tree and that is a graphical a graph of your entire code base and the static analyzers that I've talked about PHP Stan and Sam I imagine PHP Storm are not sure they probably well first to definitely look at the AST to the abstract syntax tree to get all that information so if we could get all the generic information into the abstract syntax tree it will be in a consistent way that all the tools can use it but what would probably happen in these proposals here we're not going for full language support is that information wouldn't get through to the bytecode so the VM would have no knowledge of it so you wouldn't get your runtime checks now if we're doing templates we might be sacrificing runtime checks it's quite possible that our code at the moment doesn't have runtime checks in anyway but we're certainly removing any of these runtime checks so that means we have to make sure that our tools are correctly reporting everything and the problem with that is all bets off is that any missing or incorrect type of information so if you have a legacy code base it's going to be quite a lot of efforts to get all the information you need to successfully do this kind of analysis but if we're starting from scratch I'd definitely recommend this approach and if we were in a position where we could, I mean we're essentially trusting our static analysis analyzers to do all our type checking for generics well if we're trusting it to do that why stop there, we could actually remove type checks completely, we could have this stage where you've got our code and before we actually deploy it to production we run our static analysis it's type checked everything, it says everything's all right and then we remove type checks and well we remove type checks where there's a performance penalty in checking them, I think if you leave in int type hints then I think there is a performance boost but I believe a lot of the other ones actually slow things down having these type checks so you might say well why don't we just use Java or some other language and there's lots of good reasons to use PHP one of the major benefits of PHP is you can pick the level of code quality you want you can go and do all the things I'm proposing which is quite a high level this is quite a high level of engineering involved or you can just make some really simple, if you just write a script you're only going to run once you don't need to worry about doing all of this all of this stuff the other thing is languages like Java have this compilation stage and one of the great things about PHP which was alluded to in the keynotes is you can just make a change hit F5 and you can see it and you don't have any of this long process we kind of skip this whole compile process and that is a really great thing about PHP and of course PHP developers are much happier than Java developers I can say that, I've been both so that's kind of what we've discussed today we've gone live into generics we've looked at ways we could actually add in static analysis into our build process now and you kind of get the powers of it and you've kind of seen maybe some of the limitations and maybe what has to happen next in the community to get closer to generics and the benefits are we're adding clarity to our code and all range of type-related errors earlier which then reduces the cost of static analysis and that's kind of how we use it now we write our code, we put it through static analysis and then that gets run but yeah, the next step really I think is either standards or full language implementation I've been Dave Lidmouth, thank you for listening I said we've got any questions we're using static analysis right now and one of the things I really want to do is get more with the generics and stuff but colleagues are a bit hesitant because it removes runtime checks so how would you convince them? well the first question is are some of these runtime checks actually there? because if you look at the example I had right at the start where I'm returning users and I've got open and closed brackets in that code there were no runtime checks so I would check if there really were and that would be my first point and then probably it's something there are times you just have to remove the runtime checks but there are probably ways around it which is less optimal code to put them in so that might be a discussion unless you've got something that's really easy to explain over we might need to sit down with a bit of paper and the answer is you might not be able to remove those runtime checks so it depends on if you use something like Salm it tells you how much what percentage of of all the types it has inferred or understands so if that's a high number probably what you're doing it is safe to remove those type checks and if that's a low percentage then you're probably going to have to have some kind of compromise which probably hasn't really helped you but now thank you hello a couple of questions one of them is which ideas support this right now and the other one is if the syntax is more or less the same for all the tools sound, php, sound or are there major differences so the first question is unfortunately no ideas support certainly none of them support the template none of them support the array note annotation where you've both got the key and the value so if anyone is working for an IDE for php then please the second one is I do these tools so on the face of it they both understand each other's annotations and they act largely the same even in the examples I've done there's a github repository third one down there there's about I think there's eight or nine examples and both Salm and php Stan have these online playgrounds so you can take the code snippets and put it into the online playground and even in the relatively simple examples I've got there are a couple of differences between the tools and these things so Salm has had templates for over a year php Stan added them in at around December so they are it is still quite new they are really responsive if you raise issues on them and they believe they are issues they are really responsive about fixing them so I think they will normalise towards a standard eventually but there are subtle differences even in code that looks the same so but because they are so responsive for fixing these things next week they might be more aligned and I certainly know that the authors of Salm and php Stan are on the same Slack channel and they often discuss things with each other has that answered your question? Thank you very much Any others? I was one at the back Am I right in thinking that there is a way to add the static analyser to an existing code base where you can say only run the static analyser on newly committed code so you can run it moving forward but it doesn't particularly fall over when you are working with old code Yeah so Salm and php Stan have base lining facility and I've actually written a base liner for tools that don't have base lining facilities so you can do that and you can progressively improve the quality of your code base in that way so you can run your analysis you can fix whatever issues are most critical and then you can create a base line so you won't see those problems again and as a static analysis approach I would advocate doing that the issue of and it kind of goes back to the first question and if you want to apply this if you want to apply this kind of generics analysis to be 100% sure that the static analysis tool is telling you the right answers and there are no false positives and no false negatives and you have to have the complete information so if you have a legacy code base then you can certainly put this information in but I'm not sure I would trust it completely because there might be bits of information that are missing and what you can try and do is if you've got a legacy code base and you're working on a contained bit of that code base and if you work out the boundary so if you work out all the entry points if you've got a new bit of code and it's almost completely separated from anything you've done if you work out that boundary of that code and what are all the external entry points to that which will probably not be all the public methods it will probably be the smaller space in that then if you kind of protect your boundary to that you can apply this analysis to just that chunk of code and you just need to make sure that any bits where you're communicating from legacy code base into this new bit you can use things like assertions to check the data really is on the genox the data really is what you expect it to be but if you've got a huge code base and you're just adding little bits here and there this lining will definitely help but you can't guarantee what I'm proposing here will work but you should still do it does that make sense? any other questions? there's one on the back just a small comment I'm also working on a legacy code base and I'm using FAN it also has pretty good directives it's enabling lots of its rules on a per directory basis so that's a really nice way to gradually adopt it in your code base so you can say ignore everything in my 10 top level directories and then sort of slowly enable one rule at a time for some one small sub component of your application yes and you can do an approach I haven't used FAN myself does it have a baseline facility? so can you take all your results that you have now so let's say it looks at your code base and it's found a thousand issues can you say I want to make that my baseline and I only want you to add in new tell me about new issues so you can store an issues directory where they'll tell you these statements and lines and anything new will mark as a regression of that and again this is an approach you can do but it's kind of like the last it's a bit like the last thing you still unless you're entire, if you want to do what I'm proposing with the generic everything needs to be known so you'll always have these issues with legacy code bases where you're ignoring bits of it or you're ignoring existing issues I don't know if that answers a question so I want to find it okay I mean I'm happy to discuss this if you want to ask any questions okay there's a joined in talk please let me know how it can be made better or come and see me the slides are there on my website have a look at the code snippets in the github repo for this is of interest and definitely check out the static analyzers if you haven't already okay thank you very much


Hello, my name is Steve Cezina. I live in Colorado and I'm a UX engineer at CrowdStrike. This is a talk about memory leaks, specifically memory leaks and JavaScript single-page apps, like the kind of apps that we write with Ember. I'm going to talk about how we, as a community, can put an end to them. First, I want to make sure we're all on the same page by defining some terms and concepts. First is the heap. Every physical device has a certain amount of RAM. The device allocates this memory to the browser, which then allocates memory to the JavaScript engine. So the heap is what we call this finite chunk of memory that is dedicated to the browser tab running our app. This determines how much memory our apps have to work with, and it varies by browser, OS, and device. So how do our apps use this memory? Well, every object in JavaScript is automatically stored in the heap as your program runs. This includes objects, functions, arrays, strings, numbers, booleans, and DOM elements. These things all take up memory while they are in use by our app. DevTools provides us the heap snapshot tool for visualizing the memory allocated in the heap at any given moment. You can see here that after capturing a heap snapshot of this chess app, we can see the heap size measured in bytes, and we can search the snapshot to see various classes from the app that are currently present in the heap. Although we can see what's in the heap, we have basically no control over memory management. Memory is automatically allocated by the JavaScript engine. So if memory is automatically allocated, that means memory must also be automatically freed up. Otherwise, we would eventually run out of memory. Garbage collection, or GC, is this process used by the JavaScript engine to free up memory. It does this by removing objects from the heap once they are no longer in reference, and it does this as your app is running. As soon as the JavaScript engine decides it needs to reclaim memory, it will briefly halt the execution of your program to run a GC. We can see this visualized here using another DevTool called the memory allocation timeline. As we use the app, we can see memory that is allocated represented by the blue bars, and memory that is reclaimed by the garbage collector represented by the gray bars. Normally, we don't have to think about this process. It just works. Well, most of the time. The rest of this talk is about the times where we do have to think about it. All right. So if the JavaScript engine automatically reclaims memory, how does it know what it can reclaim and what our program still needs? Well, GC roots are objects that the JavaScript engine uses to determine what can safely be reclaimed. Each JavaScript environment has its own set of GC root objects, but you'll probably recognize them. Node.js has the global object, the browser has the window object, and the document DOM tree, and DevTools also creates a GC root when it is active. These GC roots are separate, but they're all part of the same heap. When a GC needs to run, the garbage collector begins at the each GC root and recursively traverses all the objects in reference, and it marks everything it finds during this traversal. Then it sweeps back through the heap and reclaims anything that wasn't marked. So you can imagine we might end up in a situation where we have object references that hang around unintentionally, preventing garbage collection from reclaiming objects that our program is done using. This situation is called a memory leak. When this happens, the leaked memory stays around in the heap indefinitely until the object's dominator is garbage collected. Dominator is just a scary word for some other object in the heap that's still holding reference to the leaked object. Now, imagine that the dominator is a long-lived object like an Ember service or controller or worse, one of our GC roots like the window object. Since this dominator will stick around for the life of the app, so will the leaked object. If a memory leak is exercised over and over, say like a leaky component being rendered many times, the heap will fill up with these objects that we're no longer using, reducing the amount of memory available for our program to use. This is a problem because as the heap grows, GC will begin to take longer to execute and it will begin to run more frequently. Longer, more frequent GCs will cause poor perceived performance for the users of our apps because more time is spent pause for GC and less time is spent running our code. Given a bad enough leak, the heap can also run out of memory causing the app to crash. In all of this varies based on the device, browser, and OS. Devices with lower overall memory or lower CPU speeds will experience memory issues much sooner, and mobile browsers will often aggressively kill tabs that begin using too much memory. So hopefully you have a basic understanding of how the browser manages memory and how things can go wrong. Now I'm going to share a story about memory leaks and me. Before CrowdStrike, I worked on a very ambitious Ember app. It was the kind of app that users leave open in a tab for days on end. The app had a lot going on. Hundreds, maybe thousands of components on screen at a time, drag and drop, keyboard shortcuts, in-app video recording, real-time data updates. These sorts of apps are sometimes called productivity apps or workspace apps, and they can be especially sensitive to memory issues. Apps like this often deal with lots of data and components, meaning lots of objects on the heap, and long sessions, meaning lots of time for memory issues to compound. So out of the blue one day, our customer support team began receiving reports from users that the app was randomly crashing. The crashes were unpredictable and there were no clear reproduction steps. At first, it was just a few customers reporting this, but after a few weeks, the volume of complaints grew. Users were experiencing multiple crashes per day, often losing work that they were in the middle of. They began to perceive the app as unreliable and unstable. We even began to see people complaining about this issue on our NPS surveys, and our NPS scores were being dragged down. NPS is a measure of customer satisfaction, and everyone in the company paid attention to it. As you can imagine, fixing this issue became a top priority for engineering. There was high pressure to fix the problem, but no reproduction steps, no clear indication of what the problem was or whether it was even a problem with our app. This was a recipe for disaster. We had a vague idea that this might be a memory-related issue, but it just as easily could have been an issue with an add-on, with Ember, or even with Chrome. And no one on our team, including me, was an expert in memory. I was like Ed here. I knew some of those words, but I had never faced down an app-breaking memory leak before. But we had to do something, so our team decided to dedicate two developers, I was one of them, for a full sprint to do whatever it might take to fix the issue. So we read up on memory leaks. This repo, memory leaks examples, became our guide. If you're not already familiar, it's a great resource and goes into more detail on how to actually find and fix memory leaks than I have time for today. So without reproduction steps or a clear root cause, we set out to fix every memory leak we could find in our app in hopes that that would fix the problem. We captured countless heap snapshots. We recorded countless memory allocation timelines. We would run a test. We would open DevTools, capture a heap snapshot. Then we'd search the snapshot to see if that part of our code had caused a leak. Then we'd fix whatever code we suspected was causing the leak. And then we would do it over and over again. This is all very manual slow process. And heap snapshots are definitely not easy to interpret and make sense out of. And once you find the leaky code, it's not always immediately obvious what's even causing the memory leak. But we were making progress in fixing leaks. So one sprint turned into several. We merged PR after PR, hoping that each memory leak fix would be the one to finally fix our problem. But every time we fixed a leak, we would uncover another. This process led us to find and fix memory leaks and some of the add-ons we used. We even found a leak that was introduced by a recent upgrade of Ember itself, which fortunately had already been fixed in a newer release, so we just had to upgrade. But after all of this, we were still getting reports of the crashes. We had ruled out third-party browser extensions as a cause. We had ruled out low-end devices as a cause. We even watched customers use our app over Zoom, hoping to see it happen, but it never did. And we even thought about just giving up and engaging a consultant. Eventually though, we realized that continuing without finding a reliable reproduction and root cause would continue getting us nowhere. So our last idea was to install a session recording tool in our app in hopes that we could capture a recording of a session that crashed and finally get something to go on. So a couple of weeks go by and nothing. No crash sessions had been captured and we had stopped receiving complaints from customers about this issue. Months after we started fixing memory leaks and weeks after our last memory leak PR had merged, the problem just mysteriously went away. We suspect the issue was probably a bug in Chrome that had been patched in a recent release. So what's the point of all this? The point is memory management, when it goes wrong, is hard to deal with. All of the memory leaks that we fixed were very subtle. They were easily introduced by well-meaning, talented developers. They snuck through PR reviews and QA. They were present in our app, our add-ons, and even in Ember itself. And we as a community don't have a good way to detect memory leaks. Find them, fix them, much less prevent them in the first place. And until a user complains that the app is slow or randomly crashes, we might not even know that there is a memory leak in our code. So what do we do about it? Well, we start by using Ember. Ember happens to give us a lot of tools to avoid memory issues. And although I happen to run into a framework-level memory leak, they are actually very rare and the core team cares deeply about preventing them. In fact, Ember is one of our best tools for avoiding memory leaks. There are entire classes of memory leaks that Ember developers just don't even have to worry about. For example, Ember's declarative template rendering system means that we pretty much don't ever have to think about leaking DOM nodes. The elements we render in our templates are automatically torn down by Ember once we are done with them. And we never have to think about it. These days, Ember isn't unique here. Other frameworks have the same benefit, but this hasn't always been the case in past eras of JavaScript development. Similarly, Ember's on modifier used for adding event handlers in our templates automatically takes care of removing event listeners. Forgetting to remove event listeners is one of the most common sources of memory leaks and Ember handles this for us. And even when using our own custom modifiers to interact directly with the DOM, we are given a nice API to clean up after ourselves. So good memory management hygiene is built into Ember's common constructs like components, routes, controllers, templates and services. And the same memory management tools that Ember uses internally are now available by the Destroyables API introduced in version 3.22. Destroyables give us a robust set of APIs for managing object life cycles and cleaning up objects once we're done using them. So Ember does a lot for us when it comes to memory management, probably more than most of us realize. But as we saw earlier, there are still opportunities to introduce memory leaks in our code. Here are some of the most common causes. We won't be covering these in detail, but it's likely that you have one or two of these occurring in your app right now. Once these issues find their way into our code, we're pretty much on our own as developers when it comes to dealing with them. Manually capturing allocation timelines and sifting through heap snapshots is basically the state of the art for finding and fixing memory leaks in our apps, assuming that we even know we have a leak in the first place. Like pretty much any other bug, memory leaks are much easier to introduce than they are to find and fix. And it's best if we can just avoid introducing them in the first place. That's why we have tools like automated testing, TypeScript and linters. These tools have eliminated entire classes of bugs from our code bases. But this sadly doesn't exist for memory leaks. We often leave it up to our users to tell us that we have a memory leak, although they don't know that's what the issue is. We can do better. So out of frustration with the experience I just shared, and after some long thoughts in the shower, I realized that all of these problems could be solved. Thanks to Ember's strong conventions around testing and some open source packages from the Chrome team, I realized that we have the technology. So I wrote an Ember add-on called Ember CLI Memory Leak Detector that solves some of these challenges. It works by recording a heap snapshot after your test suite finishes running and the application has been torn down. The heap snapshot is searched for any retained instances of ES classes defined in the host app or add-on. And if it finds any retained classes in the heap snapshot, we know we have a memory leak. And the test suite fails and a report of the retained classes is logged in the console output. This tool can be used after a full test run via Ember test, for example, in a CI environment, ensuring that you never introduce a new memory leak in your code base. It can also run on smaller sets of tests via Ember test dash dash filter or repeatedly while you're actively writing code via Ember test dash dash server. This means you can quickly tell if a test module or even an individual test exercises leaky code. This dramatically speeds up the developer feedback cycle when you're fixing memory leaks. The process of running a test, opening dev tools, capturing a heap snapshot and searching it for your code now takes seconds instead of minutes. And it's integrated right into Ember's test harness. Since the add-on tells you which classes have leaked, you are able to narrow down to a set of test modules, then a set of tests, then a single test, then you just change code in the leaky class until the test passes. It's like test-driven development for memory leaks. I'm really excited about this tool and the potential it has to benefit the Ember ecosystem in a few ways. One, it gives every Ember developer, including you, the tools to find and fix memory leaks quickly without needing to be an expert in memory allocation or heap snapshots. Second, it gives add-on authors and users of add-ons a shared way to understand, report and fix memory leaks in add-ons. And third, it can prevent Ember developers from introducing new memory leaks into their apps. I would love to eventually get this add-on added to the default blueprint for new Ember apps. Between Ember's strong memory hygiene and Ember CLI memory leak detector, the Ember community has the tools to work toward a future where every Ember app is free of memory leaks. There's still work to do here, so let me know if you're interested in helping out. Before I end, I wanna say thanks to Alex Ford and to TrueCoach for making development of this add-on possible. Thank you.