All right. Well, thanks for coming to our third tutorial session for this Q3 hackathon. I want to welcome Tim and thank you for talking about GitLab package stage. It's a new topic that a lot of people probably don't know about. So, Tim, I think this would be a great opportunity to not only give us an overview, but where community members can help and make GitLab better. So, I'll turn things over to you and feel free to share a screen. Thanks, Ray. I really appreciate the chance to talk to everybody today about some of these features. I'm going to, I have a short roadmap presentation that I put together. I cover some items that are open for community contribution as well. And just going to share my screen, and we'll do a short demo of the container registry as well. And, Ray, can you see my screen okay? Looks good. Cool. Let's start out with, okay. So, I'm Tim, the product manager for the package stage. And we are basically responsible for the container registry, the and our different package manager integrations, which includes NPM and Maven. Conan is coming soon, launching in a couple of weeks. And after that, we'll be working on NuGet, which will help support .NET developers. So, really, what is the package stage? It allows you to publish, consume, and share packages in any languages, in any language, all in one place. And you could do that using the specific client, like the Docker client, which I'm going to show off today, how to use that with the container registry. You could also use GitLab CI, CD, and be able to publish these packages. So, for us, our north stars are the universality of the product. We want to be able to support all different languages, not just Maven or not just Java and JavaScript. We want to support Python, RubyGems. There's a tremendous value in us supporting all the languages that our users need. It has to be secure and accessible. So, for us, that means providing a unified method of authentication using the GitLab personal access token. So, you could simply authenticate with GitLab and connect to any of the different package managers that you want to. And then providing a performance supply chain. So, many of the packages that users rely on are downloaded from external sources, of which there may be downtime or they may be open source vulnerabilities. So, providing caching and proxying and caching of dependencies for faster build times. And in the future, providing some more security firewall features to prevent any vulnerabilities from entering into your supply chain. Okay. Oh, I guess I'll do the demo right now. Okay. So, I just wanted to give before a little context for the container registry. I have this group that I've set up here with container registry examples. I have this project called container registry one. And just to give you an idea of overview, you know, I have this project. Many of you are familiar with the GitLab CI ammo file. This is the recipe for CI CD pipelines. It includes a Docker file. And then the app itself is really simple. It's just going to open up and say, hello, it's Tim. And I'll be able to and it'll show which host it's being hosted on. So, let's start with maybe making a small change here. Get rid of this guy. I'll just add my last name to the hello, it's Tim. And commit changes. Okay. So, now, because I have in my GitLab CI ammo file to automatically run a test and build a new image on commit, I have a pipeline that's running now. So, if I go over to CI CD, I can see my pipeline running. And this is building this image now. Alternatively, I could also do the same thing from the command line using the Docker client. So, here, I'm CD'd into my specific project here. And I could build a new Docker file just using the standard Docker build and Docker push commands. So, here, I'm going to say Docker build. Okay. And then I can also do run a Docker push here. Okay. And now, if I go back to the container registry page, which is under this packages container registry, I could see we just pushed this tag just now, this merchandise tag. And I could also run it by saying Docker run. Okay. So, this is running. Let's get a view of which containers are running on the system now. I could see here that the ID is 0AA408. So, let's see where. And here it is. Hello, it's Tim and 0AA4. So, it's the same image that is running or the same container, I should say. So, we just built and published a container. Let's go back and check on our CI CD pipeline, which finished successfully. And so, we should see another image pop up in our container registry as well. That will probably be here. And we could see latest was updated a minute ago. And this came from CI CD. So, a couple of things that we're working on here is improving this user interface. We recently added the ability to select multiple images for deletion. We're going to be adding more things like sorting and search in the future. Okay. So, I just wanted to give a quick overview of how you could use, how you could build, publish, and consume Docker images using GitLab. And let's go back into the presentation. Okay. So, I did want to talk about the roadmap. So, what are, what themes are we addressing in the coming months in the package stage? The first and most important theme for us is lowering the cost of the GitLab container registry by introducing storage management features. What does this mean? For a lot of our customers and for GitLab ourselves, we have not, we don't have a lot of features for cleaning up the container registry, for programmatically untagging images, and for running inline garbage collections. So, that's a big, highly requested feature for us. Many of our customers and users are asking for this. And I'll dive a little bit more into what that means in the upcoming slides. The second theme spans multiple container registry as well as our other package manager integrations. We want to improve usability by standardizing authentication and functionality. The idea is we want parity across all of our different integrations. That means that you should be able to always authenticate using your GitLab personal access token. It means that you should expect the same functionality from integration to integration, meaning that you could build and push images or packages using the client or CI, and all of it should just work. We've made a lot of progress on this theme in the past months. And then we want to expand our user base by integrating with additional highly requested package managers. And we'll go over what that list is in an upcoming slide. I suppose I should pause here. Are there any questions? Ray, do you have any questions? No, I think we're good. Okay, cool. We'll keep going. Okay, so what have we done in the past three releases? One of our, when I started in April, so over the past few months, when I started, I noticed that our MPM registry was a source of negative customer sentiment because authentication didn't work. Subgroups, we didn't support subgroups. So for many of our enterprise customers, they were blocked from using this because they're using groups and subgroups. So in the past few milestones, we've enabled groups and subgroups. We've improved the image deletion process for the container registry. And then we've improved authentication for MPM, adding support for the GitLab personal access token. We've improved the navigation for the overall the package stage before this. The container registry and the packages were all under separate navigation items. It's all centralized now. We are slowly adding more features to our API, both for the container registry and for the package registry. So now you can list the images in a group. That's important because next, we'll be able to create a front end for that view. So you'll be able to view all the containers or images, I should say, at the group level. And we talked a little bit about the multi-select the lead for the container registry UI. So that's what we've been working on. Okay, so what will we work on in the next six releases? It's like wherever that is, it's in the way. Okay. So for the container registry, like I mentioned, a big piece of what we're working on is the storage and management features. So we want to give users the ability to delete images from CI. The way that we'll do this is we are updating permissions for CI registry user to not only be able to build and publish images, but to also be able to delete them as well. That work is almost done, should be coming out the next couple of weeks. I mentioned the improved image deletion. We added the multi-select last milestone. Now we're fixing a bug with Docker where if you delete a tag, it will delete all tags with the same image ID, which is problematic for users in an unexpected behavior. So we're actually pushing a change to the Docker API to resolve that. We have built a few milestones ago. We built an experimental command for Docker pruner, which will clean up the container registry. So in the upcoming milestone, we want to take that from experimental to production-worthy. We've seen some of our customers be able to use this, and it works fine. There's some bugs. We need to be able to make sure this is production-worthy, and this will really provide a streamlined garbage collection process, and hopefully we'll run much faster and much more efficiently than what we currently have. Tim, I assume these also listed on your product directions page by any chance? Yes. Thank you for exactly. So all of this information can all be accessed in our product vision package, which includes what are our North stars? These look familiar. This is the information at the top of the presentation. What are our categories? And if you drill into each of these categories, you'll see all of the same information I'm showing to you now. I just wanted to make a presentation for that. Perfect. So if anything gets updated, people have the latest in the handbook. Exactly. Always go to that. This is more of a snapshot. Yeah. Thanks. No problem. A couple of other things that are worth calling out. A new API for bulk removal of tags at the group level. Right now, the API is limited to the project level. So for system administrators that have many groups and many projects, this is problematic and time consuming to remove tags across many projects. So giving some group level ability there will be really valuable for cleaning up the registry. And then some more features maybe worth calling out down here is how do we expire images from CI? So currently for artifacts, for instance, you could add on a line to your GitLab CI ML file that says expire in seven days. We'd like to do the same thing for images and tags.  And then we want to add in more retention and expiration policies which will allow users to programmatically decide how images and tags are removed and deleted. Okay. For the package registry, so this is all of our package manager integrations. Up next, we're actively working on Conan, which is for C and C plus developers. This should launch a milestone 12.3 and we're really excited about it. Right after that, we'll be working on Nougat for .NET support. And then we have a couple of things. So we want to add CI authentication for NPM. That's, we mentioned the theme earlier of standardized authentication and bring parity across these features. This is a key missing feature that we'll be working on in milestone 12.4. And then a couple of other small things. So a new API to list out the packages at the group level. That'll be, again, a nice change in something that we're doing for the container registry as well. Supporting subgroups for Maven. This is another key parity feature, making sure that we are supporting our enterprise customers or people who have many groups. And then the same thing. Maybe you're seeing a theme now, CI authentication for Conan. The dependency proxy is basically allows users to proxy and cache Docker images. It's in a pretty, it's in minimal phase right now. In order to bring it to viable, we need to add authentication support for private projects. And we need to give some more management controls to allow users to delete items or purge it entirely. So this is what we're working on over the next six releases. And again, it's really addressing those core themes of storage management, usability and authentication and net new package managers. Okay. So in the spirit of GitLab and everyone can contribute, what are some ideas that people can contribute to during the hackathon? Well, we have this issue published to a group based Maven API endpoint. Currently, we allow for pulling pack dependencies at the group level, but not publishing. So this could be a really good first contribution for people. Auto rebuilding Docker images in the container registry when the baseline image is updated. This is a highly requested feature, something that we definitely want to tackle. Maybe a little more advanced, maybe not for your first contribution, but this would be something that I think would be a really fun project. Another one for first contribution maybe is an example Docker image to check the setup of the container registry. So basically once you connect and authenticate to Docker and set up your GitLab container registry, basically that you have a Docker image prebuilt in for you to check the setup. And then finally showing container registry push events in the GitLab activity feed is would be another valuable thing. So when a new image is pushed or updated, basically showing that detail on the activity feed for people to consume. Yeah, I definitely appreciate that, Liz. And I mean, people typically ask for a starting point on where they can get started in different product stages. This is great. And in case people are watching the recording after the hackathon, if these issues haven't been taken up during the event, I mean, obviously, you're welcome to raise your hands and just mention me or Tim, and then we'll do our best to help you out as you work on an MR. Perfect. Thank you. Sure. And just real quick, I wanted to give an overview of what's coming next in terms of package manager integration. So we mentioned Conan and Nougat. After that, we're planning on PyPy to support Python developers, then RubyGems, then Helm charts are next three. And how does this compare to GitHub's package registry? They have some of these integrations, but the feature is still in beta. So we're hoping to launch some of these in the coming months. Okay, I could just go through, quicks these updates. So for the package registry, we mentioned Conan and Nougat, we mentioned subgroup support for Maven. We also have some UI improvements that we'd like to make. So allow for sorting of the packages list and allow it to delete files from the packages UI. For the container registry, we're also conducting user research. So we're currently surveying system administrators and developers to understand the required metadata for the container registry. And I can take a minute to talk a little bit more about that. So here in our container registry UI, we're showing some very basic metadata. We're not really taking advantage of the fact that this image we built a few minutes ago from CI, and we're not showing which commit it came from or which job ID it came from. So understanding what metadata would be useful in this table through survey and user research, and then actually doing the work to make sure that it's populated here and that we're helping users as much as possible. And a couple more slides here. The dependency proxy, you know, really we're just planning the next set of features for broad support and adoption of this feature. It's pretty early on. It's still in the minimal phase. It just might be more towards the end of the year that we start to work on this. And for Helm charts, again, a similar story. Helm v3 is coming out in the coming months. Azure DevOps supports announced Helm support with their container registry. We will probably do something similar as Helm v3 is changing how it operates to use a container file or container registry as storage for the Helm chart. So we'll probably do something similar. We're just doing some user research on that one as well. And then just in terms of people. So we had five new hires start between July and August. The package team today, we have three engineers. We have two managers. We have a product designer and a product manager. We have a new engineer starting in October and we're looking to hire another Go engineer. So if you are interested in, if you know Go and you're interested in working in this stage, definitely click through that link and see if you can apply. Okay. I think that was the presentation that I had and the demo raise or anything else that you think that I should cover? No, I mean, I really appreciate this. I mean, not only providing an overview of your team and the product stage, but also where and how people can start contributing. And I know there are a few other folks on the call, like if you have any questions, feel free to go off mute and verbalize them or type them in the chat window and give people a few more minutes. And yeah, I'd also appreciate on your last slide, like listing not only you, but other people in the package team that people can ping if they have any questions on issues or MRs. I mean, this is a reminder that I give probably almost every tutorial session. Pinging people at GitLab is a completely fair game, a fair thing to do if you have any questions. We're all part of one community. We don't distinguish between whether you're working in GitLab or not. If you have any questions on issues or need help on MRs, feel free to ping anybody listed here or myself. It's true. We love it. And as a product manager, I totally rely on feedback from our users and from the community to make sure that we're building the right features and that we're moving as quickly as possible and that we understand our users' needs. So definitely reach out if you have questions or feedback or anything like that. I'm always happy to talk to our users. And I was also checking merge requests in CE in terms of community contribution and I searched for container registry. It looks like there are two MRs that are in progress for 12.3 and one was merged for 12.1 and coupled before, like a previous to 12.0. So it's not a completely green territory. I guess people have been contributing. I definitely want to see more of it, I'm sure. Yeah, exactly. And we have a couple of contributions to integrating new package managers. So someone has an MR that we're reviewing now for adding in support for Composer, which is the PHP package manager. That's another one that we want to push through and we really want to make it so that other people can contribute as easily as possible and add other package manager integrations. So we're definitely interested in that as much as possible. We love the community contributions. All right. It doesn't look like there are any further questions, but obviously this slide set is already posted, but the video will be posted on the Hackathon page as well. So if you weren't able to join the session, hopefully you'll have a chance to view this and then Tim or myself with any questions. And thanks again Tim for making the time. Really appreciate it. No problem. Thank you. All right. Thanks.


Namaste, welcome to the next module of our course. In this module, we will understand under fitting and over fitting through the code. You know that over fitting happens when our model has excess capacity to memorize the entire training data. So, what happens if you look at the learning curves, we observe that the training error and validation error both reduce to begin with after a point the training error goes down, but validation error starts climbing up. If we are seeing that those kind of learning curves, we infer that the model is suffering from over fitting. On the other hand, if your model is so simple that it does not have enough capacity to learn the model or to learn the patterns in the training data, then our model is suffer from under fitting. In case of under fitting, both training and validation errors are high. In this lab, we will use IMDb movie review data set to demonstrate under fitting and over fitting. We will initially build a baseline model, then we will build a model to under fit the data and over fit the data. We will first build a baseline model and then we will build couple of model so that our model under fits and over fits to the training data. Later in the lab, we will demonstrate some of the techniques that help us to overcome the under fitting and over fitting problems. Let us begin. Let us first connect to the colab, colab runtime, we are connected to the colab runtime. Let us install the tensorflow 2.0 and print the TF version to ensure that we have right tensorflow version installed. We will be using tf.keras API as in earlier colab, so that is why we are importing keras, we are using numpy and matplotlib for data manipulation and plotting respectively. So, you can see that we have successfully installed tensorflow 2.0 through the print command over here and you can see that we have now tensorflow 2.0 installed on the machine. We have tensorflow 2.0 installed in our cloud runtime. Next step is to set up the training data, we are using IMDB dataset for training and IMDB dataset has movie reviews and each of the movie reviews is tagged with a label 0 or 1, 0 meaning the movie review is negative and 1 meaning the movie review is positive. So it is a problem of identifying whether the movie review is positive or not. So, what we do is IMDB dataset is available in the keras.datasets, so we do not have to write a lot of code to load the IMDB dataset. So we will use top 10,000 words, the IMDB dataset is available in keras.datasets, so we can easily load IMDB datasets and we will be using a multi-hot encoding where we turn the words into vectors of 0, 0 and 1 in 10,000 dimensional vector space. So if the word is present at a specific position that we will see one over there. We will be using a multi-hot encoding for our reviews where each word is converted into a vector of 0s and 1s. Concretely if we have a word whose index is 3 and 5 present in the document, it gets converted into a 10,000 dimensional vector where all the entries are 0s except for indices 3 and 5. So let us try to understand that every review is converted into a multi-hot encoding where we have a vector with 10,000 positions. So initially we convert each movie review into a sequence of word indices. So the IMDB dataset is maintained as a sequence of numbers in the keras. We convert each review into a multi-hot encoding that means if there are two words with indices 3 and 5 present in the review, only the position corresponding to those words will have one in the list, rest of the other elements will have 0 in their list. So that is how we convert the review into multi-hot encoding. So as you can see in the code, we use a multi-hot encoding with 10,000 words. We use the load data command to load the training and test data along with their labels and then we define a function for converting the sequence into multi-hot sequences. So this particular function takes into takes sequence and dimension as argument and it essentially defines or it essentially tries to fill a vector of dimension 10,000. What we do is for each of the word or each of the word index that is present, we set 1 for that word index, everything else is essentially 0. We call we essentially we first convert the training data into multi-hot encoding and test data into multi-hot encoding using multi-hot sequences function and we are going to use the number of dimensions to be 10,000. So let us run this and convert training and test data into multi-hot encoding. You can see that the data is being downloaded and getting loaded into memory and the data is converted into multi-hot encoding here. Let us look at some of the resulting multi-hot vectors. So here we have plotted the multi-hot vector for the first training example and you can see that the indices where the word is present is 1, everything else is 0. You can see that there are lot of words from the initial index are present in this particular example and fewer words from later indices are present here. Now that data is loaded, let us try to build a model. We will first build a basic model where we will use a neural network with two hidden layers, each layer having 16 units each. So let us look at how this neural network looks like. In base model, we have neural network which has two layer of 16 units and then there is a single output unit and we have a multi-hot encoded vectors. So the number of inputs is 10,000. So each of the 10,000 inputs are fed into the first hidden layer, then the second hidden layer and then the output layer which gives us the probability of the review belonging or review being a positive review. So let us see how to set this up in the code. So we build a kera.sequential model where we are going to stack layers one over the other. We are going to use two dense layer each with 16 units with activation of ReLU and the input shape, we have to specify the input for the first layer and the input shape has got 10,000 inputs. And the output layer is essentially it has got one unit with sigmoid activation. We use Adam as an optimizer and binary cross entropy as a loss because we are trying to solve really binary classification problem and we will track accuracy and binary cross entropy both during the training. And this particular command will summarize the model. The model summary is a useful way to understand what is going on in the model and to check whether model is set up as per our expectation. So you can see that there are three layers in the model. So the first hidden layer has 16 units, the second one also has 16 units and then there is output layer which has got one unit. The first hidden layer each one of them get 10,000 parameters each because there are 10,000 inputs plus a bias unit makes 160 k parameters. The second one essentially each of the unit in the second layer has will get 16 inputs which is output from this and plus 16 is to 72 and the final one has 16 inputs plus 1 bias unit. So thus there are 17 parameters for the last layer. Let us try to understand this a bit more carefully. So what we have is we have this 10,000 units. So each of the 10,000 unit will be connected to the 16 units. If we just focus on one unit and try to understand number of parameters we can readily derive it for all of them. So you can see that for a single hidden unit there are 10,000 parameters and there are 16 such units. Remember each of these units also have a bias term. So we add 16 to this. So this gives us, so we essentially get 160 k parameters. For the first hidden layer. Now what happens in the second hidden layer, let us try to understand it. Second hidden layer also has got 16 units. Let us try to see how many parameters are there for one unit in the second layer and then we can calculate for all the 16 layers. So you can see that there are 16 inputs, one from each of the hidden unit from the previous cell coming to each unit in the second hidden layer. So there are 16 units, so there are sorry there are 16 weights and there are 16 such kind of units plus there are 16 bias units. We said that each one of the, each node has, each unit has one bias, no bias connection attached to it. This goes to 272. Let us see what happens in the last node. So there are 16 units here, so this last node we will get input from each of the 16 units here and it has got its own bias term. So the number of parameters in the last layer is 17. So let us tally this with what we see in the summary and you can see that there are exactly the number of parameters that we derived. If we sum all these parameters up, we have total parameter of 160k or 305 parameters. So after setting the model, let us fit the model with the training data and validate it on the test data. So we will train the model with 20 epochs with a batch size of 512 and we are going to use Adam Optimizer for training the model. And just to remind you, we use binary cross entropy as a loss that will be optimized. Let us train the model. So you can see that as we train the model, the cross entropy loss is reducing, the validation loss is also reducing. But after third layer, it seems that the validation loss is going up while training loss is still going down and you can see the same thing happening for the cross entropy loss. It initially went down and it started going up. So what does this point to? I would encourage you to stop here and think about it. This probably points to some kind of an overfitting is a signature of overfitting. So we see that after 20 epochs, our accuracy is 1. We got a perfect classifier, time to celebrate. But hold on, the accuracy on the validation is not so great. Validation we only get 85% accuracy while on training, we get a perfect classifier. We have accuracy of 100%. Our job is to build a model that works well on the future data and not the one that works well on the training data. So there is some problem which we need to fix. So one way to address the problem overfitting is by reducing the number of parameters in the model. In neural network, it is easy to reduce the complexity. We can either reduce the number of units in each of the layer or we altogether remove a layer that will result in lesser number of parameters. Other way to reduce the overfitting is by getting some kind of training data. If training data is not available, then we have regularization techniques that we can use. In regularization, we can use either L1 or L2 regularization or a dropout regularization that is used in the context of neural network. In case of L1 and L2 regularization, we add penalties that are proportional to the weight of the model. In case of L1 regularization, we add penalty that is proportional to the absolute value of the parameter or the sum of absolute values of the parameter. In case of L2 regularization, we add a penalty that is proportional to sum of square of the value of the parameters. In case of dropout, we decide to randomly drop certain nodes from the hidden layer or input layer of a neural network. We normally set dropout between 20 percent to 50 percent that means 20 percent to 50 percent nodes will be randomly dropout in each of the iteration in the neural network training. The dropout is only used while training. So, in the same manner, even regularizations or dropout are used during training. In the test, we simply apply the test data on the model. So, the first strategy, let us create a smaller model here. So, instead of using 16 units, we used 4 units. So, actually the number of parameters will go down that we just have 4 units. So, let us run and see how many parameters are there in this. So, you can see that the number of parameters have come down at least 4 folds. However, we had 160 k odd parameters. So, just by introducing, just by reducing the number of units in each of the layer, we got it down to 40000 parameters. I would encourage you to see to yourself that whether this parameter calculation is correct on the lines of the calculation that we did in the earlier case. Let us train this smaller model and see what happens. So, you can see that even this smaller model seem to be overfitting. The training accuracy is close to 1, but validation accuracy is quite low. That means the model is probably memorizing the training data. It has got enough capacity to memorize the training data. But going back, you can see that the model started overfitting somewhere around, we can look at the validation loss, validation loss is going down, going down and around this point it started overfitting where validation data, validation loss started going up around epoch number 7. If you go back and check when our baseline model started overfitting, our baseline model started overfitting right from the third epoch. So, you know by creating smaller model, we are able to delay the overfitting of the model. So, if you use smaller model and if you stop training around fifth epoch, we should still be fine. If you stop our training after fifth epoch, we will not get the overfitting that we see up to 20 epochs. Now that we have created a smaller model, let us go to the other extreme and create a bigger model and see how fast it overfits. So, we use a bigger model, so how do we create a bigger model in neural network? We can simply add more units in each of the layer or add more layer itself. So that way we will have more capacity in the model or more parameters in the model and since we have more parameters in the model, if you do not have enough training data, the model tries to memorize the training data and gives us perfect output on our training set, but it did not perform well on the unseen data. So, let us compile this model and you can see that now we have far more number of parameters. So, there are close to 5.3 million parameters as against 160 k parameters that we saw in our baseline model. Let us try to train the model for 20 epochs with the same batch size of 512. You can see that after 4 epochs, so the model right away started overfitting. So, you can see that the validation loss is increasing right after the second epoch. So, in the first epoch it was 0.29, it went up to 0.33 and then it went, it kept going up all the way and at the same time you can see that the cross entropy loss or the training loss is coming down, training accuracy is going up, but the validation accuracy is reducing. These are the signs that this is probably the model with quite a large capacity or excess capacity and it is prone to overfitting pretty quickly in the training cycle. So you can see that after training the model for 20 epochs, you can see that the training loss is very small close to 0, but the validation loss is quite high and validation loss never reduced rather it started growing up right from the second epoch. So, this points to the fact that the model is overfitting pretty fast in the bigger model. And we will compare how the size of the model affects the possibility of overfitting. So, you can see that there are 6 line in this particular plot. So the solid lines are for the training loss and dotted lines are for the validation loss. On x axis we have epochs, epoch is one epoch is one full iteration on the training set. We are using the same batch size. So all of them will have same number of iteration in one epoch and on y axis we have got a binary cross entropy loss. So, you can see that the green line corresponds to the bigger model, the blue line corresponds to the baseline model whereas, the orange time corresponds to the smaller model. We will have to keep an eye on the dotted lines because those give us validation loss and we infer that there is a overfitting when validation loss starts going up while training loss keeps going down. So, let us look at the smaller model, initially smaller model the validation and training both the losses are going down and around eighth iteration the validation loss started climbing up. So, you know we can say that the smaller model started overfitting after 8 epochs. Look at the baseline model, baseline model started overfitting around third epoch, initially the validation and training loss were going down but after third epoch validation loss just went up and up whereas, in case of bigger model it overfit almost instantly right from the first iteration. Training loss was coming down but validation loss never came up. So, you can see that smaller the model it will take a bit longer to overfit. Now that we have visualized the overfitting of the model, how can we really prevent the overfitting? So, there are three strategies if you can get more data overfitting can be prevented. But getting more data is not always an option because getting more data is costly sometimes it is just not possible to get more data then the second resort that we have is to add regularization or a penalty term to the loss function. So, there are multiple ways of performing regularization couple of them are based on L1 and L2 penalty both L1 and L2 regularizations add penalties proportional to some function of their weights. So, L1 regularization adds penalty proportional to the sum of absolute values of the weight whereas, L2 regularization adds penalty proportional to sum of square of the weights of the parameters. So, let us see how we can add L1 and L2 regularization in the code. L1 regularization adds cost proportional to absolute value of its weight coefficients, L2 regularization adds cost proportional to square of the weight coefficients. Let us see how to add L1 and L2 regularization to the neural network through tf.keras API. So, along with each of the layer dense we add along with each of the dense layer while defining the model we add a kernel regularizer argument. So, kernel regularizer we can either use L2 or L1. So, here we are using L2 and we are also specifying what is a regularization rate. So, we have let us say a loss function and we add a regularization rate times the penalty. So, this particular penalty is added is added over here. So, the regularization penalty is 0.001 and we are using L2 regularizer here. So, we use L2 regularizer in both the layers in order to prevent overfitting. So, this is how you can define your L2 regularizer and then we can compile the model and train it. You can also add L1 regularization in the similar manner. You can also add L1 regularization in the similar manner and I would encourage you to try it yourself. So, here we added the regularization to the baseline model. Now, let us compare how the training of regularized model compares with the baseline model. Let us plot the loss with respect to the epoch. So, you can see that the blue model corresponds to the baseline model whereas the orange model corresponds to the baseline model with L2 regularization. You can see that the blue line which is the baseline model started overfitting around third epoch whereas the regularized model took slightly longer to start overfitting. So, you can see that the regularized model is resistant to overfitting for some more epochs than the unregularized model. As you can see the L2 regularized model has become much more resistant to overfitting than the baseline model even though both the models have the same number of parameters. The other way of adding regularization or other way of regularizing neural network is by adding dropouts. Results in randomly dropping out a number of output features in the layer during training. Dropout is not applied at the test time. It is very important to note that at the test time we do not drop any units. Instead the layers output values are scaled down by a factor equal to a dropout rate. So, let us see how to use dropout in the context of neural networks. Keras has a dropout layer. We can use a dropout layer with dropout rate at its parameter. So, you can see that this particular dropout will be applied to a layer that is specified just before that. So, this dropout will be applied to the first hidden layer and then this dropout of 0.5 will be applied to the next hidden layer. And after specifying the dropout we can again return the model and compare its performance with the original baseline model. So, let us compare the baseline model with the dropout added. So, now compare this also with the L2 regularization. You can see that after adding L2 regularization the model marginally improved its resistance to the overfitting. But after applying dropout you can see that there is a substantial improvement in the resistance to a dropout. And now model overfits after few more epochs than the original model. So, to recap what we did is we actually build a model on IMDB dataset. The model that we built a baseline model on IMDB dataset then we built a very big model on IMDB dataset that was overfitting. In order to prevent overfitting we first reduce the capacity of the model by building a smaller model and we also added regularization to it. We studied L1 and L2 regularization and then dropout regularization. So, these are the common things that are done to prevent overfitting in the neural network. The first strategy is to get the training data if that is possible. If training data more training data is not possible we can reduce the capacity of the model or add L1 or L2 regularization or dropout. Dropout in fact is the most commonly and the most effectively used regularization technique in neural network. Apart from that we can also use data augmentation to generate more training data from the available data and batch normalization to prevent overfitting. Hope you understood underfitting and overfitting through this coding exercise and you had fun learning these concepts. Thank you.


I've come from Vietnam. I've had 21 years of people in Vietnam hear this news. I'm so sad. And we are in this race. We have one people come from Vietnam. I'm so sad. I read this news on Vietnamese newspaper.


Hi, my name's Chris Mepperm and this is the ball swap challenge. I've got something in my head, and I'm not sure if anyone's tried it, but I don't see why it wouldn't work. It was so good! Can I have a retake of that? It would have been unbelievable! Obviously it didn't go as planned, but on the same kind of thinking behind it, just better execution. Folks, the definition of insanity is repeating the same behavior over and over again, expecting different results. My mouth's up here! Alright, bear with me. What am I going to do now? I thought that was a valiant effort. I'm going to be honest.


From Hollywood, it's time now for Johnny dollar. Oh, yes, I've been trying to reach you. Mr. Sherman That's right Well, I've already told my secretary to make all the records. It's not the records I'm mainly interested in mr. Sherman. I want to talk to you personally Because I've been informed that you're able to furnish an alibi for my number one suspect Marty Blake. That's right Who informed you mr. Dollar mighty Blake a lovely widow herself, right? That's exactly what I was thinking Tonight and every weekday night Bob Bailey in the transcribed adventures of the man with the action-packed expense account America's fabulous freelance insurance Investigator yours truly Johnny dollar From special investigator Johnny dollar location Greensport, Missouri to the home office Great Plains Guarantee Company Hartford, Connecticut assignment The open town matter Expense account continued Items five and six seventy cents a copy of the Greensport Daily Herald and the taxi fare to City Hall I open the paper and glanced over the headlines Murder of police chiefs still unsolved City Attorney's Office claims new evidence mayor Lyons demands action I could feel for the mayor my client stood to lose $50,000 life insurance payable to the dead police chief's widow Marty Blake So I wanted action too and I was hoping to get it from City Attorney Dave Sherman. I'll have a seat. Thanks Why did you get in town? Oh? Early this afternoon, and you've already met Marty Blake That figures yes, it does under the circumstances Her husband has been killed he carried $50,000 worth of insurance with my company and Mrs. Blake filed her claim less than 24 hours after his death. So that's why you hot-footed it out here from Hartford, huh? Well, the company figured 24 hours was pretty fast action for a grief-stricken widow. Oh, I Don't imagine Marty is grieving too much. Oh, she's not she told me that herself Marty's about 26 or 7 Ed Blake was in his 50s money. Maybe well not until he met her then he started to make it fast He had to that's how Marty likes things fast And then he married her well, then she married him And now eight months later. She's a widow With an insurance claim for $50,000. Well, she said she liked things fast. Why? Yeah, I know what you're thinking, mr. Dollar, but you may as well back up and start all over You're off on the wrong foot. Well, how do you mean this killing? Oh sure it Worked out perfect for her made to order a man could build a pretty good case against her especially with that 24-hour insurance claim Yes, I would really cinch it with a jury. I pointed that out to her. I figured as much Would you mind telling me what happened? Well, our first move was to start turning on the chair That's one of the easiest things she does if she thinks it might pay off And after all with $50,000 in the offing sure Did she give you any of the details of you know of the night it happened?  What would you think? Well, she wasn't mr. Dollar any more than she was sorry that Ed Blake was murdered and she's pretty cold-blooded Well, that's a moot question. Figuratively. Yes. I suppose she is See she used to be a dancer never good enough to make the big time so she had to well I live by a wits and her charm But you pointed out things that could look pretty black for if the case ever got to court. That's right What happened? She sent me to you Why so I could tell you she didn't kill him. Well, it adds up mr. Sure, I know late at night the two of them alone in the house her husband shot to death with his own I know mr. Dollar, but not by her why not that story about the mysterious prowler has been used before over and over Matter of fact, I broke one of them not six months ago But in this case, it's true. How do you know because you were wrong on one point they weren't alone in the house What do you mean? Who else was there? I was oh, I see Jesus pure as the driven snow well at least so far as Blake's murder is concerned Yeah kind of a neat setup isn't it huh instead of prosecuting attorney. You're the star witness for the defense Yes, I I support if it came to that I'd have to be I was there when it happened so that's why she sent me to you so I could hear it straight from the horse's mouth That's about it. Well, she couldn't have a much better alibi than the city attorney himself. Thank you Well, she's in the clear mr. Dollar How did you happen to be there at two o'clock in the morning? I was spending the night Edna I were planning to leave early on a fishing trip. Oh, I see Like to tell me all about it certainly We'd all gone to bed around midnight or a few minutes before to I woke up I heard Marty and Ed out in the hall just outside my room I opened my door and just then Marty snapped on the hall lights a second later the shots blasted downstairs five quick one Where was she standing at the time? By the door of her room not ten feet away from me. So you see mr. Dollar. She couldn't have done it So that's that She can thank her lucky stars you were there. Oh, Marty's always been lucky Have you known her a long time three or four years ever since she came here to Greensport You and Blake were pretty good friends. I suppose. No No, as a matter of fact, we didn't have too much use for each other. Oh, I know it doesn't make sense going fishing together spending the night in his home, but Well, I know what you're thinking mr. Dollar. Do you yes? You're thinking maybe I need an alibi as much as Marty does Well, it is that kind of an alibi it works equally well for either of you two Yeah, only I don't need an alibi Blake and I had our little differences.  Sherman 33? Why and Marty blank is 27 and a beautiful girl Her husband was around 52. I thought wrong guess again, mr. Dollar. I've known her too long And what's more important? I know it too. Well, meaning sure. She's a knockout. I was nuts about her once she's a My summer night wild honey circle and a handful of stars, but there's one great big catch to it And that is she's got a building jukebox hidden way inside of her and when you put in money it plays real pretty music when you don't Nothing had Blake found out and still went on putting in money. Yeah, you like the music And that's why he got mixed up in the rackets. Please cheese salary was Well, well, mr. Dollar, you do get around don't you was Blake running them Or just taking on now. What makes you think there are any rackets here? Maybe it was just a guess. Oh, who told you about them It's funny. I can't seem to remember at the moment.  Sherman who's behind the rackets here in Greensport just a minute I don't get as that is from or was he the one who gave you wait a minute, mr. Dollar. Just take it easy Sorry, I didn't know you had no, that's all right. Well, come on in This is Johnny dollar dollars mayor Lyons. Yeah, Mr. Dollar Mr. Dollar's an insurance investigator.  Dollar We'll appreciate all the help you can give us well so far mayor I'm afraid that doesn't amount to very much the only theory I had just blew up in my face Oh, what was this theory? Oh, mr. Dollar suspected Marty Blake of the killing. Well, Dave Didn't you tell him that you yes? Yes. He told me mayor That's when I blew up in my face Well at various times in the past mrs. Blake might have been there's one might say a bit indiscreet But to consider her capable of cold-blooded murder is utterly unthinkable, sir Particularly when she was standing just ten feet away from the city attorney at the time. Well, yes, that's true Of course just what is the official theory on the shooting gentleman while at the moment? I'm afraid we haven't any I'll tell you mr. Dollar when we find shorty wells. We'll have the killer Shorty wells a local hoodlum Blake got him sent up the river a couple of years ago and shorty swore that he'd get Blake for it If it was the last thing that he ever did oh Well, that's a fairly common threat though for a criminal to make and usually it's nothing but talk But surely was parole just last week mr. Dollar He was seen around town the morning before the killing and we haven't been able to locate him since I see Wells Did it all right? There's not the least doubt in the world Yeah, it seems to end up doesn't yes, and we've got every available man in the lookout for him But so far no luck. There's not even a trace of him. Tell me something here Would Marty Blake know anything about the rackets her husband was tied in with rackets?  Isn't it mayor as I hear at Greensport is a wide open town May as well admit it will he knows it already. There's no point in trying to lie All right, it's true mr. Dollar, but it's not wide open as you put it But there are rackets as unpleasant as it may be to admit it. Do you have any idea who's back of them? I wish to heaven I did what about you mr. Sherman Well if I had an idea strong enough to talk about I'd be talking about it in court What about this shorty Wells was he in on them before Blake sent him to prison? Yes, supposedly, but he wouldn't spill a word about the setup at his trial afraid you maybe huh probably Wells is the key to this Whole thing mr. Dollar the rackets as well as the killing fine shorty Wells and we can wrap this thing up in ten minutes Then I guess we better find it standard theory number two an ex-convict with a grudge number one young pretty wife middle-aged husband big insurance policy hadn't panned out too well and I wasn't too sold on the second one. It was just two pats somehow And the fact that shorty Wells was missing didn't mean much If I were an ex-con just out of prison and had threatened a man who was later murdered I'd be missing too And yet the pan answer was sometimes the right one the case was still like the town itself wide open I Won't out of city halls started down the sidewalk toward the taxi stand. I didn't know what moved to try next Technically, of course, I was out of it since the beneficiary had an ironclad alibi She'd get paid regardless of who actually did kill Blake I didn't pay any attention to the horn the first time but when it sounded again. I turned to look She was parked at the curb a few yards away Marty Blake I've been waiting for you Johnny. Have you yes, I wanted to talk to you you didn't earlier Well, I guess you just rubbed me the wrong way Weren't very nice to me, you know, sorry You've talked to Dave Sherman, I suppose. Yeah. Yeah, he gives you an airtime alibi Why didn't you tell me he was there that night when I talked with you this afternoon? I might have If you'd been nicer to me, I see you'd have checked it with him anyway Yeah, I suppose I would have done. Well now that you know I'm not a murderous Maybe we can get to be friends Johnny. You think so You're still not being very nice. I am always politely respectful toward new widows, Mrs. Blake At least until after the first week.   Of course, I know him Yes, I used to know it. Where is he now? How should I know? What are you trying to pull if you think for one minute that you're down quick? Are you all right, Mrs. Blake?  How do you know it was me that we're shooting at right?  Won't you yours truly Johnny dollar? Yours truly Johnny dollar starring Bob Bailey is transcribed in Hollywood written by Les Crutchfield It is produced and directed by Jack Johnstone Be sure to join us tomorrow night same time and station for the next exciting episode of yours truly Johnny dollar Roy Rowan speaking