Okay, so let's start. So keep calm and trust your model. So I'm Praveen Sreedhar. I'll be talking on explainability of machine learning models. So I've been working as a machine learning engineer as and as like machine learning R&D roles with multiple companies in Kochi, Bangalore and the US. And I'm also a co-founder of a nonprofit called Tinker Hub which essentially is for college students to learn about new technologies. So let's start with a story here. So suppose you are a very recently hired data analyst at a financial services company. Okay, and your boss here, he has actually given you this loan default data set. Okay, which looks like this. You just query the data set using your favorite library pandas and see that you have these features here and this is the target variable. Okay, if the person has defaulted or not. Okay, and then you run to scikit-learn and train directly a random forest classifier. Okay, and give it to your boss and say that your job's done. Okay, you also find out feature importance is because you know that your boss is very concerned with the interpretability of the model of what features are important in your model and how it is being predicted and all. And you show him that the amount of the loan, the term and the age of the person who's taking the credit grade and all of these are important features.  The model makes a wrong prediction.  All right, and you have no answer because random forest model, you don't know which. So if it was a normal decision tree model, you know, like you can go through the tree and see which like parameters were essentially important for predicting this feature, but your complex black box models not allow you to do that. Okay, and you're given a big lecture on how like it's very important in regulated industries like finance, defense, medical diagnosis, insurance and all that to have this, this interpretation layer on top of your normal black box evaluation metrics and things like that. Okay, so that's why interpretation is of very paramount importance in these industries. And the problem that occurs is that as and when we have like higher and higher accuracies with more complex models, explainability of the model gets hit by a large extent. So you see here, decision trees are very explainable. You can see at each node what is happening. You can go through the tree and all, but as you go up, you go up to graphical models, macro models, and of course, deep learning. You get a very high bump in accuracy, but at very low explanation capability. All right. Now what do our models learn? Okay, now interpretable models. So we all know linear regression. We know that y equal to mx plus c, you know the slope and the intercept. So you know like how the line is, how you can exactly, you exactly know the function. So it's very explainable. You can tell your boss what is happening in the model exactly. So those are applicable only in linear monotonic functions where any change in the independent variable gives an exact and measurable quantity of change in a single direction in the dependent variable. So you see here, but the problem is that not all data sets that you get will be of this linear fashion. So you need to train proper complex models to predict the non-monotonic functions like these that you see here. Okay, so these are the ones which create the biggest problems. Now, what is explainability? Okay, now early we saw like linear models, you know the function that is being predicted. So it's like very explainable. Okay, important that would be highlighted. So we'll see an example after you get more idea. Now, this is one of the methods of doing that. Okay, it's called tree interpreter. So interestingly, this does not have any research paper associated with it. It's just a blog post by a person named Ando. Okay, and he made this library called tree interpreter as well. So for an example, see the XOR gate, exclusive OR gate where you have this truth table. Okay, and we're training a decision tree, a basic simple decision tree to model this particular data. Okay, so this is the condition at the base node if X1 equal to zero, if it is true, it goes up. Okay, so true always goes up, false goes down. So we will take this last example. Okay, then both are one, but the output should be zero. Okay, so X1 equal to equal to zero goes down because it's not zero, it's one. Okay, X2 equal to equal to zero. Again, it's not that, it's one, so it goes down. Okay, and the value is finally, we get it as zero. So the prediction is zero and how we decompose the contributions is. So the train set is given a mean value. So you see zero, one, one, zero. Mean value of point five is given at the root node. Okay, and at each traversal, okay, the value is recalculated. So X2 is zero, there's no increase from X1. Okay, and when it goes down, since it's X1 and X2 are interrelated, it gets a decrease of minus point five. So we see here, even if we train a complex model, we can average the contributions using this and basically decompose each of the values and it flows down. Okay, so I'll, so since like I think all of your minds have been blown, I'll explain this as you're a five-year-old. Okay, and that's why a team in Russia has come up with ALE five. Okay, so explain like I'm five. Okay, that's actually a library that they built for explanations. Okay, now let's see the same data set that we, I mean, the problem that we faced earlier, the lone default prediction data set. Okay, so this is a normal scikit-learn pipeline where you're training a model here, a random forest classifier. You're fitting the model, okay, model.fit, okay. You see that the accuracy is 74%, okay, good enough. Okay, you're calculating the feature importances earlier that we did. Okay, now this package actually makes it very easy. It was import ALE five. Okay, so now let's see why that wrong prediction was being made. So we're taking the fifth sample data. Okay, we're seeing that it has been defaulted. So one means default, zero means not default. Okay, and you see that the prediction is giving it as not defaulted. Okay, now if you see here, this is the contribution values.  Okay, the number of years of playing a big factor, the grade, the credit grade of the person is playing a big factor. Okay, so since the person has a very high salary, the model thought that maybe that person would not default. Okay, so now you can show that it's because these parameters and the proper contribution values that the model is throwing out error like this and you can debug it and make a better model out of it. Okay, similarly I've done for one more. Right, so I guess I'll go back. Okay, so what we just saw was a model specific approach of interpreting pre-based models. So, I mean this same method can be used for all like ensemble models of trees, like random forest, gradient boosted trees or any tree-based models. Okay, now there are many such more model specific approaches. So one of them is model specific visualizations. Now, this is the Andrew NG's famous cat neuron. I'm guessing like all of you must have seen this. And in the previous talk, we saw how easy it is to fool CNNs, right? But the same sort of a method, the gradient ascent method can be used to visualize what is happening in a CNN model as well. So what happens is essentially to get this final neuron, I mean the picture where, I mean the visualization where what sort of features actually activate the neuron maximally, you do a forward pass, you compute an activation, okay, activation AI, okay, at a particular neuron. So this is a random static image, okay. There's nothing involved in this random static that you get in a TV, that sort of an image just to start with, you do a forward pass at a particular neuron I, you get the activation value and you do a backward pass, get the gradient, okay. So here the gradient is essentially what, by what values should you change each pixel's color so that this activation is maximum, okay. So that is essentially the gradient. So if you want to increase that further, what will you do? You'll add that back to the initial thing that you got. So you repeatedly do that. You add this gradient repeatedly with a scaling factor, okay. And at a particular, I mean after doing, I mean a few iterations, you get like what the neuron is actually expecting, okay. And you can visualize it in that way, okay. So this is again like global scope. You don't have much scope in saying like you can't take a particular instance or instance and see what is happening, but then overall idea you'll get what neuron is looking into, okay. Now this is the same slide that the previous speaker showed, again it's very easy to fool CNNs. You just add the gradient to the previous image to get an adversarial example. The gradient of a given is added here and it's predicting it as a given, okay. So this actually makes it all the more important to be able to explain predictions, okay. So the next model specific approach would be attention mechanisms employed to explain a model, okay. So in this you see Salesforce has this new abstractive summarization model where essentially whatever portion or whatever like input text is actually being attended to that gets a heavier weightage and you see here. So see the summary here it has elements whichever was important and attention was paid to here are getting more importance and so you get to kind of know intuitively that all of those attention points were the ones that are used to make the final summary. Okay, so again this is another example of attention. So this is a famous paper called show it and tell it came out in 2015. And here you see like a woman is throwing a frisbee in a park. So you see frisbee underlined and you see that the model is paying attention to the portion of the image where the frisbee is, okay. So this is again an intuitive explanation that you can give for the model predicting in a particular way, okay. Now what happens in a attention model? So let's take an example where we're converting human readable dates into machine readable dates, okay. Into a particular format date. So you're converting five January or five Jan or like short words, short shortened dates into proper format like year, month and day, okay. So you have a encoder decoder framework here, okay. And so you see to predict like 2016 here, this 16 is getting maximum attention and it comes here. So the encoded, the attention mechanism what it does is it actually calculates the weighted average of wherever the network needs to look into at, I mean, maximally to get the particular output. So that is the attention model and we have like model agnostic approaches. We earlier saw partial dependence plots. We saw variable importance measures, okay. There's something else called residual plots and surrogate models. Again like statistical methods, not very explainable to your business people, but this is a new method called local interpretable model agnostic explanations. So this research was done at University of Washington under Carlos Guestrin. So I'm not sure, so he was actually the CEO of a company called this Pedro Domingos. He was the, so I forgot the name, Turi, Turi. So Turi was acquired by Apple and all for the research that they did, okay. Now what this model does is, so this is again a caption model, okay, where this is an image of a dog holding a guitar very strangely, okay. And you see that when a caption is made for electric guitar, it shows this portion of the image as an explanation. So it shows that the model is looking into that particular area to explain, I mean to predict that it's an electric guitar there. For acoustic it's looking at the wooden portions, if you see. Okay, so unfortunately it's still getting confused between electric guitar and acoustic guitar because almost the same string shape and color is there. So it's still getting confused, okay. And Labrador is explaining using this face and a small image here, okay. So Lyme is a very interesting method. Essentially what they do is they throw away the original model, okay. Your original model is thrown away and suppose you want to explain this particular instance. What is done is data points are sampled across, I mean the vicinity of that instance and locally another classifier is trained there. So another linear model is trained there. So in that particular local region, you know a proper interpretable model of why something is happening. So the original model is thrown away and you train a new model around the vicinity and in the local context, you train a interpretable model, a linear model with which you can know what is happening inside this particular, the original model as well. Okay, so essentially the idea is for every image, you take like random perturbations. So random images you just chop off different portions randomly, okay. And see which one is giving more importance in the local interpretable model. Okay, so you see here, you'll have like many instances like this, this is ever continuing, okay. Since we can't show the endless models, it's not shown. But one of these will have the maximum importance, which is this one. And that is chosen as the explanation, okay. So behind the scenes quite a lot of complex math is going on, but essentially this is what happens. Like you make a lot of random samples, you train a local regression model, which you can explain. Okay, and then whichever one of those is giving the maximum prediction, you choose that as an explanation and show it to the end user. Okay, now let's see a demo of this. So I have a Keras model here. So I'm sure like all of you must have tried out Keras deep learning models. So we have an inception V3 model that is pre-trained, okay. And let's see the prediction for a particular image here. Okay, so you see this image, okay. Which to us looks like a cat and a mouse. Okay, but to the inception model, it looks like an American black bear. Okay, strangely. Okay, American black bear and like tons of other animal names that we don't even know. Okay, so what's going on? Okay, again, lime is very easy to use. You just import lime, you have a image explainer class. Okay, you do the explanation here. Explainer.explain instance. You take the image, you take the model, and you tell it how many labels to show. Okay, and parameters like that. It kind of takes some time. That's why I'm not doing it live. So it takes like half an hour on this MacBook. You need a GPU to do deep learning. So you see here, this is the region that is impacting the model to train it. I mean to classify it as a black bear. Okay, and if you see this closely, actually even like to us, now it looks like a black bear, doesn't it? Can you say this is a cat only by seeing this part of the image? No, right?  But the model does not know that, right? That's why it's looking only at this part and classifying it as a bear. So this local interpretation gives a clear idea of like what is happening, what is happening like within the model such that like this particular misclassification is happening. Okay, so we cannot blame the inception model because even we are confused when we only look at this portion of the image. Okay, now we can do like fancy stuff where like you highlight which of the features are important, which are not and things like that. So it also classifies it as a bat. So that is this particular portion. Again, kind of like looks it's black, bats are black. So essentially your model is just learning. It's like when we do like wrote learning in our classes, engineering classes, it's just wrote learning and predicting out, throwing out values. Okay, so this is essentially about Lyme. Okay, and you can check out all the code at this link. So I've added like much more detailed intros into like what happens inside. And like there are many other models as well. We saw, so we saw, where is that? So we saw here model agnostic approaches as like these many, but we have other models like layer-wise relevance propagation. Again, local and interpretable. Again, there's like couple of models within this. So I will link to those research papers in this particular GitHub repo. And it's all thanks to these three open source packages, pre-interpreter, Lyme, and ALI-5 that any machine learning engineer can use these to properly explain and build models, debug models, and show like what is happening inside the model without having the issue that the analyst faced in the beginning. Okay, so that's all from my side. Thank you. Thank you. So my question is how to improve once you've identified the problem? How do you? How do you improve your model once you've identified? In the case of say you've identified it's a better than that. Yeah, yeah. So how do you go about and improve? So that's like a huge topic because you have like tons and tons of ways you'll have to try different approaches. So this was inception with you. But something like CNN it's like very translucent. I mean it's very opaque to actually how do you actually learn that? So it is actually because I don't think there's a hard and fast rule of like improving deep point learning models using interpretations. But I guess people are trying to build that. I'm not aware of any particular process that you can use to like improve models. Rather I mean even image net like the challenge and all what people do is like they try out different architectures keep on trying out different architectures and just like try to beat the top score. That's all right. Yeah, I think he has asked the same question. What I want to ask. Okay. So I have a question here.  But when you apply to a problem bigger than a loan prediction, you have a lot of features. You'll definitely have correlation in the feature space itself. So does it split the contribution between correlated variables or? Yes, yes. So the three interpreter package has like has like provisions where you can specify that you want feature interactions to also show up. So what will happen is that along with those individual features the interaction features will also come up.  It'll show A and B together and like show what contribution is being made because of both A and B. So you can use that. What you showed in that right variable importance and you showed the effect. That's easier for simpler models where you have less number of features. But like a lot of these models you have large number of features and there are like very specific regions where these variables exist. So these variables exist together in like groups. Right, right. Where the interactions are very complex. So models will usually pick one and then if you slightly perturb the data the model will pick another feature. But that's very difficult to explain to the business guy because he's. True, true. He's not, he's thinking that your models are sort of not robust because they are changing. That's the. Yeah, so this is not a solved problem at all. So this is like the initial stages and like I've seen like very people actually delving into the practical aspects where, so it's, so even like yesterday we had like a couple of OTR sessions where a lot of talks were going on on how interpretability is important but not much was done in the practical aspects of how to make them. So this is definitely not a solved problem. And as you said, like we still face a lot of challenges when you have tons of features in. Hi, so in these interpretation of the models, how does the confidence come in place? Like, you know, if a model is not very confident of the prediction, does it still get the same areas or does it get highlighted as, you know, model is not confident because. So in the local, local interpretation model, so the Lyme package that we saw, they don't have like the importance is shown in but for three interpreter, it includes that. The only problem is that three interpreter cannot be used for images. One question here. Yeah, so I saw that you demonstrated LE5 and Lyme as the, this thing and you took up a, and you took up image examples. So does it work for non-image kind of data as well? Because the contribution also you demonstrated as an image which highlights a certain portion. So does it work in the same way for non-image data? Yeah, it works for the other cases as well. So, so let's see. So this is the Lyme package, the image interpretation model. But after, if you see, so this one I showed earlier, right, the loan prediction. Yes, you see here, this is, so earlier I was using LE5 for that, like for contributing the feature values and all, but you can use Lyme as well. So see, you see prediction probabilities here and you see the feature importance. These are the importance, okay. So you can get it even in Lyme, that's also possible. Thanks. So you showed that black bear or that cat image. So what if it's actually a black bear that looks like a cat to us? I mean, that's possible. Yeah, that's possible. Definitely possible.  It could be that some earthquake happened and that guy defaulted on the loan. Yeah, it definitely can happen that way. The problem with our like black box methods is this. It's that like we're like learning, I mean, it's like road learning from the distribution that we have and predicting based on that. So we don't have that like intuitive logic sense to see that there's a mouse there. So maybe it could be a cat sort of a thing. No, so like it's possible that our priors are wrong because we saw a mouse, we thought it was a cat.  Like maybe you decide that what is the predictability in this use case? And if the predictability is low, then you can't blame the model. Even if you find out how it's interpreting the results, it won't be the model's fault because. Yeah, yeah. So there's no method. I mean, I'm not very aware of any method like that right now, but it could be a very interesting thing to do. Okay, thanks. Hello, yes. Extending to his question, like I was once looking into a problem where we can fill in the loop of confusion matrix. Okay, like if based on the first pass of learning, we generate a confusion matrix between cat and the beer and feed that into the CNN. Can you repeat? So, okay, so like instead, one is like we have some hierarchical classification here that at the first level, we apply the output from the CNN and then we from the second level, we apply some contextual information that if the class one has is close to class two, then they probably belong to a confusing group. Okay. Second option is that we feed that output of our confusion matrix into the CNN itself. Okay. I'll have to look into it. I didn't like get what you're like suggesting. Basically, I was looking into if you can, because we stopped at that cat and beer example, but how will you exactly solve it? I was... Okay, okay, okay. Okay, let's have a chat offline because I need to catch up on that then. Any more questions? We have a little bit of time left. All right. Thank you, Praveen. Thank you.


So there has been a lot of talk over the last couple of weeks about the Linus Tech Tips Linux challenge and even I have made a video about it. I don't want to talk too much about the particular challenge in this video, but one of the things that Linus himself has talked about is the frequency that he is forced and I use that word because that's the word he used to reboot his computer. Now there is a myth in the Linux community that is widely spread that you never have to reboot your computer after an update and this is not true. You do have to reboot your computer in order for updates to take effect. Now where Linux differs is that you're not absolutely forced to do so right away. Now for people who have used Windows before and I'm pretty much assuming that's mostly everyone, you'll know that when you update Windows you have to restart every time. There's no exceptions. Every time you do an update you have to reboot and the thing is you are forced to reboot eventually.  On Linux that is not the case. You do the updates whenever you feel like it. You're never forced to do an update and once that update is done you don't have to do a reboot immediately unless you want to. You can do so whenever you're available to do so. Now there are several things that make this situation more complicated because the truth is the only time you absolutely have to reboot your computer is when system files are modified. So the big example of this is the kernel itself. When the kernel is updated, which it probably is whenever you're running on something like Arch or the unstable version of Debian or something like that, something that's a little bit more rolling in flavor, chances are you're going to see it a kernel update and that means that there are files on your system that have been modified and in order for any of those changes to take effect you have to go through and reboot your computer. You get to choose when that reboot happens but if you want those changes in the update to take effect you have to eventually reboot your computer and the situation where system directories and files are modified is the only situation where you absolutely have to reboot after an update. It gets more complicated when you're talking about applications because you have to remember that Linux treats updates much more differently than Windows does. When you update Windows all you're updating is your operating system. Anything else is updated through an app by app basis. So Photoshop will update itself. Microsoft Word will update itself. Whatever application you're thinking of will update itself unless you've got it through something like the Microsoft Store in which case the Microsoft Store will handle the updates. The system updater just updates Windows. On Linux when you do an update it literally updates everything, whether you've downloaded that from a store, whether you downloaded it from the command line, whether it's a snap, whatever. When you update it updates everything. So what that means is you kind of have to know what is being updated because if it's a kernel update or something similar to the kernel that is necessary for the system to run, in other words it's running all the time on your computer, in order for those things to have the update take effect you have to reboot your computer but if it's an application that's not true. If it's, for example, say it's Firefox, you use Firefox as your browser and an update comes through. It will go through an update even if Firefox is running. But you don't have to reboot your computer once you're done updating Firefox you just have to close Firefox and open Firefox back up again in order for that update to take effect. So that's where it gets a little complicated because applications and this usually mostly applies to user installed applications or distro installed applications. If those things have updates really all that has to be done is for those applications to be closed and then reopened in order for the updates to be take effect. So I can see where the confusion is for Linus and Luke who are very new to Linux or their new to Linux desktop. But to say that you're being forced to update or excuse me to say that you're being forced to reboot is just not true. I've never once in four years of using Linux full time been forced to reboot. Now I suppose if you say you have to reboot after a kernel update that is technically being forced to reboot. But the thing is it's I think the definition of forced is a little bit different here because yes you do have to reboot after a kernel update. But you get to choose when it could be a week from now. Your computer is going to be perfectly fine running after that update if you don't reboot. You won't get any of the benefits of that new kernel obviously because it's waiting for you to reboot for those things to take effect because your kernel has been loaded into memory and it can't be modified until it's no longer there. But again you can go through and do that reboot whenever you feel like Windows forces you to do a reboot after an update. Oftentimes it only gives you a few hours to say no until it just forces you to do so like you have to do it. And this and it it's worse because they force you to update in the first place. Like on Linux you'll never ever be forced to update. Now there are certain more new user friendly distros that are much more forceful in terms of suggesting that you update. For example Ubuntu and Linux Mint they will actually go through and frequently suggest that you have to update your computer. But just because they suggest you do doesn't mean you have to. Whereas Windows will actually go through and update your computer whether you like it or not. And eventually will force you to reboot no matter what no matter what you're doing. You could be you know editing a document or whatever. And if the system decides you have to reboot it's just going to reboot. That's where Linux is miles and away better than Windows will ever be. So bottom line I think that everything surrounding this idea that you are being forced to reboot can be very confusing for new users. It's definitely the case. But it's imperative for people who know about Linux to spread the truth. Because yes you do have to reboot your computer from time to time. But no you're never going to be forced to do so immediately upon the end of your update. Like if you just never will be you can update or you can reboot whenever you want to. The thing is you just have to keep in mind that eventually if you want that new kernel or whatever that system file to be or dependency or whatever to be you know used in a new update to be applied you then have to reboot your computer. So it can be like I said very complicated. But again it's very important for us to be honest about it. Instead of saying well I'm being forced to reboot my Linux computer all the time which is what has been said. And that's just not the truth. I mean I think the biggest problem and I didn't want to talk too much about the challenge itself is that Linus has said he's using Manjaro which is an ArchBase distro. He's using a rolling release. So he's getting kernel updates all the time probably several times a week and when those updates come through if you want those updates to take effect he then has to reboot. So if he had picked a distro like Ubuntu like the LTS version of Ubuntu or Linux Mint or PopOS literally any Debian based distro he would have been miles away better and he would have never experienced this whole reboot issue because those distros don't update the kernel very often at all. The LTS release gets a kernel update maybe two or three times a year maybe not even that often and you don't have to install those if you don't want to. You know those are things that are completely opt in you know. So I think a lot of his problems have been because of the distro that he chose. Now whether or not Manjaro is a new user friendly distro is something that we can argue about in another video but I think that the problem there relies in the distro he chose and not in Linux itself. So that is it for this video if you want to get in contact with me you can do so by leaving a comment in the comment section below. Follow me on Twitter at Linuxcast. Follow me on Odyssey at Linuxcast or supporting me on Patreon at patreon.com slash Linuxcast. Before I go I'd like to take a moment to thank my current patrons Devon, Chris, East Coast Web, Gentoo's Fun2, Patrick L, Marcus, Meglin, Jax, Neftool, Steve A, Mitchell, Art, Center, Merritt, Camp Drash, Reli, J-Dog, and BSD's Rock. Thanks everyone for watching. I'll see you next time.