Hey, folks, one of the really powerful things about working in the tidyverse is that it's so straightforward to use tools like dplyr and ggplot2 to explore your data. The data exploration tools are just really just so powerful. And while I just geek out on making pretty figures and trying to make things look publication quality, the really good thing about ggplot2 and dplyr is the ability to quickly iterate through a number of different types of analyses to better understand what's going on in your data. That's what I want to show you today is a few strategies that I would use to look to see if my data are correlated with each other. In this past several episodes, I have been looking at local weather data from outside of Ann Arbor, Michigan near where I live to try to get a sense of kind of global climate change as it affects my local area. Some of the data that I've been working with are precipitation data as well as temperature data. And so what I want to look at today is what have the annual trends been like in precipitation and maximum temperatures over here in our studio, we're going to get going with a new R script that I'm calling tmax prcp correlation, tmax and prcp are the variable names, the columns in the data frame I'm working with. When I run this source function, it runs an R script called local weather.r in my code directory that goes up to the NOAA website and pulls down the data for my local NOAA weather station. If you want to get a copy of your own data, then I encourage you to go back and look the previous episodes in this series, and also to go ahead and get a copy of this project. For instructions on doing that, you can go down below in the description to a link to a blog post that will get you everything you need to get up and running. And of course you can go into local weather.r, insert your latitude and longitude to figure out where you are. All right, so this has loaded and run, of course, it also loads the tidyverse lubricate and the glue packages. And so if I look at local weather, I see that I get a data frame with the date the tmax precipitation and snow. Some of you I know don't know what snow is, but trust me here in Michigan, we get a lot of that. All right, so I'm going to go ahead up here and take local weather. And what I want to get is the average annual temperature and the average or the total precipitation by year. Okay, so we've seen this a number of times already, but we can do a group by. And I want to group my data by the year, but to group it by the year, I need to extract the year from the date column, right? And so nicely, we can use the mutate function to do year and run the year function from the lubricate package on the date column. Great, that gives us our year. I can then pipe this to a group by a year. And I can do a summarize tmax for the average tmax, and we'll do mean tmax. And then we'll do PRCP as the sum of precipitation over the year.  And that is because when we looked at the previous version of the table before we aggregated by year, we had NA values. And so if you calculate the mean of a column that has NA values or the sum of a column that has NA values, the functions don't know what to do with that, right? So what you could do is go ahead and remove that NA before you calculate the sum or before you calculate the mean. And so you could do that with something like na.rm equals true. So let's go ahead and do that approach. Alternatively, what I could have done would have been like drop NA on tmax and PRCP, right? I'll go ahead and leave that in there. But what this would have done is to remove those rows where tmax or PRCP were NA values. I'm going, I'm not going to do this drop NA approach. The reason I'm not going to do this drop NA approach is that this removes the entire row if either tmax or PRCP were NA values. And so we might not have precipitation data, but we might have temperature data for that day. And so what we do here with mean tmax NA, NA.rm equals true is we remove the NA value before we calculate the mean. So let me show you a quick example. So if we do define a vector x, and let's do NA 234, right, and I have x, if I do sum on x, that gets me NA, right? That's what we'd expect. But if I do sum x na.rm equals true, so you'll see there in the pop up that the default is false. Then again, it removes the NA values before calculating the sum. And there we get nine. And it does the same type of thing for the mean. Running that, we now get the year, the tmax, and the precipitation. Now what I'd like to do is look at a couple different ways of visualizing these data to get a sense of whether or not the average temperature and total precipitation for the year are correlated with each other. So I will go ahead and call this tmax prcp. And we will use that data frame now for a variety of different analyses. So the first thing I like to do is let's go ahead and make a facet where we give them the common x axis by year. So we can do tmax prcp. And again, what we have in here are three columns where we have the year, the tmax and the prcp to make a facet where we have one window for temperature, one for precipitation. I need these numbers for tmax and prcp to be in one column. And I need tmax and prcp the names to be in one column. We'll do pivot longer on everything, but the year. And so now this gives me a year column, a name column and a value column. We've seen in other episodes how you can specify the name and the value. But let's not worry about that so much right now. We can pipe this to ggplot, aes, x equals year, y equals value. And then we can add to this would be geomline. And we can then add facet wrap. And I'm going to go ahead and then facet by name. This then gives me two side by side panels, prcp and tmax. That's not exactly what I want. I want them on top of each other, because if I have a common x axis by year, then I can kind of more evenly more easily see the change in these two variables over time. So to facet wrap, I could add n call equals one. So now we get one column with precipitation on the top and tmax on the bottom. Of course, we see that both panels are getting the same y axis scale. And that doesn't make sense for tmax, because we would never have 1000 degrees Celsius, right? I'm not here on earth, hopefully for a very long time. So what I can do is I can add to facet wrap scales equals and then free underscore y. And so this will free up the y axis scale to vary by the observed data. And so now we see that tmax has a much more plausible y axis scale. And we can kind of see how that changes over time relative to the precipitation. One thing I noticed is that I have my 1891 and 2022 data on here that I would probably like to go ahead and remove because we don't have full years worth of data for those. And so the total precipitation and the average temperature for those years is going to be a little bit screwy. So I'll come back up here and I will then do filter. And I'll say year, not equal to 1891. And year not equal to the year on today. So the year on today should give me today's this year rate 2022. And so now when I feed this into make remake tmax PRCP and regenerate my plot, I no longer have those kind of funky values at the end. And so what I see, I don't know that we've looked at the precipitation data this way, is that it's relatively flat, and then increases in total precipitation over time. Maybe we could see this a little bit more easily. If we add geome smooth, and I'm going to go ahead and remove the standard error clouds, I'll do SE equals false. And so again, we can see that in general, the tmax has been increasing over the past 130 years. And the precipitation has really been going up over the past 60 years as well. So this plot gets me thinking about something that's generally considered a no no in data visualization, which is having a dual y axis. And so we could put tmax on the left and precipitation on the right. Let's try this and see how good or bad it might look. So we'll come down here and we'll do tmax PRCP. And I'm going to start by making the temperature plot, right? So the ggplot aes, and on the x axis, we'll put year on the y, we will then put tmax. And I will then do geome line. And I'm going to make this color blue. Very good. There's my temperature on the y axis. I will also call this tmax plot. And now I want to take tmax plot and add to it data for the the precipitation. So we'll do geome line. And we will do aes y equals PRCP. And I will then do color equals red. And so of course, we saw this before when we made the facets, is that the data are at very different scales, right? Our precipitation is you know, maybe 50 fold higher than what we saw for our temperatures. And so what I could do is perhaps take PRCP here and divide it by 50. So that gets the precipitation and the temperature data to overlap, their averages are close to about the same. Of course, the variations are quite a bit different. I think what I'd like to do is scale them between zero and one. To do that, let's come back up here to tmax PRCP. And so again, we'll take tmax PRCP. And I want to get the min and the max. And so what I'm going to do is I will do a mutate on tmax to subtract out the smallest value of tmax, right? And so then we'll do min on tmax. Maybe I'll call this tmax tr. And so that gives us a transformed version of tmax, right? And then we will then that's the bottom. So we'll have a value at zero. And then we want to get the maximum to be one, right? And so then we will then take all of this, and I'll put that in parentheses. So I'll scale this so that the maximum of this is one, right? And so now I can divide this by the max of tmax. And I need this in parentheses, right? Okay, minus the min of tmax. And so now if I do a summarize to do like min on min of tmax tr and max to be the max of tmax tr. Yeah, it goes from zero and one. Cool. So this is my transformed tmax. So it's between zero and one. I would like to keep track, though, of the max and min values. And so what I will do is this is going to get a little bit messy, I know. But I will then do tmax min as the min on tmax. And then tmax max as the max on tmax, right? And this then gives us our min and max values. Great. Now we want to repeat it for the precipitation, right? So I'm going to go ahead and copy this. And if I was thinking ahead, we'd probably make this into a function. But hey, you know, we're not. So we'll go ahead and change all these pure tmaxes to prcp's get that loaded. Great. So now we have all that. And I can now feed this as my tmax prcp. And so I'm going to call this scaled tmax prcp like that. Get that. And so now I can take scaled tmax prcp feed that into the rest of the plot again is going from x equals here to y equals tmax. But that needs to be tmax tr, right? So underscore tr like that. And so now I see I go from zero up to one, that's great. And I can now do the same thing we did before with prcp, except we will then do underscore tr. And so now we can see that our red and blue lines are snugly on top of each other. So what I will do is I'm going to come back to my tmax plot. And I'm going to add a y axis to this, right? And so we've seen this before in the last episode, I did something like this, where I converted the millimeters to centimeters within scale y continuous. So we'll do scale y continuous. So I want my labels to go from 10 to 30 by fives, right? And so then I need to figure out the breaks that I want to put those at. So if I then do breaks equals seek 1030 by five. And I'm going to basically take this and run it back through the transformation that I did before, right? And so I'm going to subtract out 11.7, right? So we'll do minus 11.7. And then we're going to divide this by 17.5. So that looks pretty good. Maybe what I'd like to do is change my limits to go from like 10 up to 32. And again, I can do the same type of thing I can do limits equals to 32. So again, I'm going to subtract 11.7 divide by 17.5. And I need to open parentheses up here. And I think I need to close parentheses to close out the scale. And so now if I look at tmax plot, I see, yeah, I've got a pretty good range from 10 up to 30 for my temperature. And now I want to do the same type of thing for adding on the line for the precipitation. And so that line now is red. But what I'd like to do is to now get the same type of scaling for the right side. So to get the right side, I'm going to do the same type of thing that I did here. For the tmax plot, but I'm going to do scale y continuous. And I will then do sec dot axis equals sec underscore axis, and then I'll do trans equals tilde period. And so that basically means I'm not going to transform the data. I'm going to doing the transforming in the labels and the breaks. So I will go ahead and copy down labels and breaks down here. And let's go ahead and just make sure that we've got all our parentheses in the right place. It gets a little bit confusing sometimes. And I'm going to want to go from 300 to 1300. And I'm going to go in 200 millimeter increments, right? And so we'll do the same thing down below, right? Add that in. And then we need to give our transforming values. So again, if we look at scaled tmax PRCP, we have 420 and 1298. So I'll do 420 and 1298. And it's telling me that the scale for y is already present, adding another scale for y will replace the existing scale. So it doesn't like that I have two scale y continuuses. So that's cool. I'll go ahead and grab that first one. And we will merge it with the second one, right? And so we'll go ahead and grab that. And just make sec access a second another argument, right? And I need to reload tmax without that first one, reload that now the warning messages go away. And we see that they overlap on each other fairly well. I am noticing that this should probably go up to 1800. So let's go up to 1800 and we'll go by 300 increments. 1800. Yeah, that looks pretty respectable. We can then add to this a name. And so I'll come to the end and do name equals total precipitation in millimeters. And that gives us total precipitation millimeters on the right. We could also add name here to be average annual temperature. And so now we have both labels. And so it might be confusing, which is which, right? And so what we can add to this would be a theme. So we'll do theme access dot title dot y dot left equals element text color equals blue, I think that's what we used up here yet blue for the left. And then we'll do the same thing for the right, right. And this will be red. And so now we have the blue label for the blue line and the red label for the red line. And we have our data right on top of each other. And so the question then is, do you prefer this? Or do you prefer them faceted on top of each other? In general, people prefer them faceted on top of each other. Because this just gets really hard to decipher what goes with the left, what goes with the right. For data like this, where they're right on top of each other, it's really hard to kind of decipher again where the lines are. And it just gets to be a bit of a jumbled mess. Also, there's, you know, funny things you can do if you're like compressing your data too much for one variable one side of the y axis versus the other. So again, that's why I would personally prefer to do things with the facet that we saw up here. I think this is easier to look at than this. There are people that really like to have multiple y axes. In general, the data visualization field I think is kind of sour on double y axes. But know that you can do it in ggplot. It's not totally straightforward to do. If it's not totally straightforward to do in ggplot2, kind of like making pie charts, that's generally the developers telling you, this is a bad idea, right? So again, the alternative is to stack them as separate facets. This would be my preference because then it's more clear what is what. And then you're keeping the values on the y axes for the two different charts separate. This just gets to be a bit of a jumbled mess. And it gets to be confusing to have to look left versus right for which color and remember which color looks which way. And so I think that's why you'll find a lot of people in the data visualization community are a bit sour on this approach. All right, so we've looked at faceting, we've looked at double y axis. Let's go ahead and put each variable on a separate axis. So we'll go ahead and take tmax prcp, which we've been working with, right? And again, we've got the year, the tmax, the prcp. And let's do ggplot aes. And on the x, let's put tmax y prcp. And then I'm going to color it by the year. And let's do a geom point. Very good. So we see that the lighter points are for more recent years. And those tend to be in the upper right quadrant of the plot I feel. And so there is a bit of a temporal progression in the data, things getting warmer, things getting wetter, right? We could but for prcp and tmax themselves without looking at the year, if I go ahead and turn off the color equals year, we find that looks like a big cloud, right? And so this is kind of the more direct test of a correlation, we could add to this geom smooth. And we see that it's like a straight line through it. If we wanted to force a straight line through it, we could do method equals quote lm. And that's like bang on flat, right? So to do the statistical test, we could do core dot test. And we could then do tmax prcp, dollar sign, tmax comma tmax prcp, dollar sign prcp. Again, that dollar sign is another way to get the column out of tmax prcp. This then I'll go ahead and open this up, gives us the Pearson's product moment correlation. We see that we get a p value of 0.76 correlation of 0.02. Basically, no correlation. If you wanted, that's this, that's the Pearson, if you wanted the Spearman, which is the non parametric, you could then do method equals Spearman. This then again gives us a row a correlation coefficient of 0.018 p value of 0.83. Again, you would only do one of these for an actual study. If you wanted to get rid of this error message, you could add to the second one exact equals false. And again, that gets rid of that warning message and doesn't calculate the exact p values. And it doesn't really matter. It's 0.838. It's not significant, right? And so this relationship between average temperature and total precipitation by the year is not correlated with each other. Again, these are a variety of visual ways to look to see if different data are correlated with each other. The first we did was to facet them to take each variable, give them a common axis like date the year on the x axis, and then put them on top of each other and kind of visually inspect whether or not there's a correlation. Similarly, we also looked at how we could go about making a double y axis to plot the data on top of each other to see if there's any obvious correlation that didn't appear to be. But we also talked about the downsides of having that double y axis. Finally, we then took one variable put on the y axis, another variable put on the x axis, plotted the data as a scatter plot. Didn't really see much of a correlation. We could use GM smooth to fit a line through that, we could do core dot test to do a statistical correlation using Pearson or Spearman correlation. Again, in this case, there did not appear to be a correlation. Again, I hope you appreciate the flexibility and power of these tools to do some very quick exploratory data analyses, as well as some of the hiccups you might run into along the way when it comes to thinking about how to visualize these data and whether or not they appear to be correlated with each other. Well, encourage you to play around with this for your data. If you do this with data from where you're from, let me know if you get a different result. If you see some type of correlation, it certainly is interesting that the trends in general show that as we've kind of moved through time, southeastern Michigan has gotten warmer and wetter, even if there isn't kind of a one to one correspondence in those values, the general trends hold, but the kind of year to year correlation doesn't seem to be very robust. Like I said, let me know what you find, and I'll talk to you next time for another episode of Code Club.


You know, that's about as good a defense as you can play on Marcus Howard trying to force him into a tough runner McClung Knocks down his first shot of the night and it was a three. They'll take another look at it at a dead ball Bowser and Howard Sam Houser with his brother in town now Matt McClung no doubt about that one. That was a deep three do McClung trying to go around chartooning over the top of Theo John And go back to this little floater from McClung I mean listen chartooning was the steels leader in the country, right? So he gets past the steels leader Pick it with a good ball fake pick it now at three each to short McClung with a loose ball McClung with a tear drop off the window And you're seeing a lot of guys step up for the Golden Eagles right now 18-8 run over the last five minutes over the Hoyas McClung underneath left-hand nice drive McClung Great vision from Mack McClung Whoop fake the defense out It pick it right where he needed it Houser had seven quick points in the first half McClung top of the key Answers right back. Well When you're scouting a player Seven for their last seven the Golden Eagles Now McClung needed a big shot and McClung delivers again. He's got 23. That's a nice decision


Hi, I'm Pratibha and welcome to my talk on Python memory problems and how garbage collection happens in Cpython. I have tried my best to simplify it as much as possible. So let's get started. I hope it is helpful in understanding how in general the object get allocated Now memory problems are the worst nightmare of every developer whose code is serving large files or who is having a lot of computation in the production environment. If you have ever faced the issue of memory leak in application or out of memory exception and you are using Python and you are banging head that everything is working as it's supposed to be but you are still not able to figure out what is happening. So maybe this talk help you in understanding how the underlying architecture of garbage collection works. So this talk is aimed to summarize how garbage collection that is being of the memory works in Cpython and because it is overwhelming to see the memory issues let's get started because there are many scary issues and it's difficult to fix them because they have dependencies. So let's get started. In recent years we have seen many improvement in Python garbage collection but there are still some instances when we are not getting back the memory that we are freeing it or we are not getting the memory back as we bring up the large variables. So this results in memory current for the applications which finally crashes. Although there are multiple ways to overcome the memory challenges sometimes it is difficult to find what we can improve in our code and infrastructure that can make their memory efficient. In such cases it helps to have an understanding of what is going on behind the curtain at the lower level where memory is being managed. So this presentation aims to give you a quick overview of we'll discuss some common memory errors that we see in our day-to-day lives and how the Cpython manages garbage collection in general. So let's list down some of the memory issues we usually see in our day-to-day environment and sometimes we have large objects. Let's say we created a very big list, very big dictionary. For any computation maybe we are passing a large log file but even when we are out of that function it is lingering in the memory and that memory is not being released and this is hanging around for no apparent reason. Now there can be other reasons like reference filing in your code. Assigning references don't read distinct duplicate objects but if an object is no longer used and cannot be marked for garbage collection because it is being referenced in other place within the application it results in memory leak. In fact these kinds of referencing styles are one of the main problems of memory leak in the application field. Now this one was something which I which I faced very recently in last few months they just got unexpected memory error and there was no solution there was no answer to why it happened. So even if you have in a Fram you can get this unexpected memory error after a lot of looking around after a lot of going through the articles I realized that the pattern that was installed in my system it was 32 bit and it has used up all the virtual address space that is available to it and because 32 bit applications are limited to maybe 2 or 4 GB of user mode address space that was leading to this kind of issue but the biggest one that we usually face is out of memory. Now this one is something which we can regenerate on our own using assigning large memory chunks to the big files or big objects or this is something which just from the surface because we don't know what is happening maybe there's a memory leak. So usually when this happens what is happening is when an attempt to allocate a block of memory fails most system returns this out of memory error it's a generic one but the purpose of the problem really has to do with the actual out of memory maybe your actual memory space is not full that's because the memory manager on almost every modern operating system use the hardest space for storing memory pages that doesn't fit in perhaps just called swapping. So in addition to that the computer can usually allocate memory until the disk is filled up and this results in fact an out of memory error that means your swapping space is also filled your memory is also filled so there is a chance that your memory is not actually filled but the space which is for swapping and everything that is filled and your swapping event has reached but what if everything is in place and working as expected can there be another reason for these errors this was the main thought in my mind when I was preparing for this talk that what can be the reasons and that leads me asking the question how actually memories being allocated and de-allocated inside the C Python. So let's have a look at the memory allocation and de-allocation that is garbage collection of Python and I hope that helps us in answering our question. So the most common explanation of memory is thinking a computer's memory as an empty book this is the most common explanation that we get when we try to explain it to some of the people who don't know what it is and it is intended for short stories which are the short applications. Now there's nothing written on pages yet different authors come along which are processors and each author wants some space to write these stories and since they're not allowed to overwrite each other they must be careful about which pages they write and before they begin writing they consult the manager of the book the manager then decide where in the book they are allowed to write this is the standard explanation of how the memory allocation works in general in the computer memory. So in fact in in correct term computer memory is it's common to call it fixed lens 20 years blocks of memory pages so this analogy holds pretty well for both of them and authors are like different applications or processors that need to store data in memory the manager who decides where the author can write in the book and plays the role of memory manager of sorts and the person who removes the old story to make the room for new one is garbage collector. Let's have a quick look into C Python's memory structures in general there are layer of there is a layer of abstraction from physical hardware to the C Python usable hardware. The operating system abstracts the physical memory and creates a virtual memory layer that applications including Python can access so this whole block that you see it can be considered as the virtual memory layer top of the actual physical memory and always specific virtual memory manager it's a very long name let's call it this virtual memory manager comes out a chunk of memory for Python process so whenever there's a new process it goes to the memory manager saying that hey I have this process I want to run I want some space so what they do it look into its continuous memory space and say that okay this particular set of memories for you. The dark gray box and the image below are owned by Python processes and C Python has an object allocator that is responsible for allocating memory within the object memory area which is the blue box here this object allocator is where most of the memory happens. Here if you see there are two parts one is object specific memory where your actually objects lies and the Python's non-object memory is where it takes care of the processing and all those all the data that is required for the processing not the data but yeah the stacks and other things that it need to process it. Now this object allocator which is blue is it gets called every time a new object needs some space to be allocated or an existing object is deleted now the question is how will somebody know that object how will a process a member knows that a particular object in memory has to be deleted because it is not marked anywhere it is not how should how should it know so that's the question we are going to answer now so let's have a look into the garbage collection of C Python. Now let's revise the book analogy and assume that some of the stories in the book are getting owed no one is reading or referencing the stories anymore if no one is reading something or even referencing in their own work you can get rid of them to make room for new writing that's where garbage collections come in there are two aspects of garbage collection one is reference counting another is generation garbage collection let's have a quick look on how does a python object looks like in physical memory or sorry the virtual memory let's say you assign x variable and it's let's sorry I'm babbling let's assign that you have let's assume that you have a variable x and you have assigned value 20 to it so how will it how will it looks like you will have a memory space which will store the actual integer of 20 and it will have a label which is x and it will be pointing or referencing to that memory location actual python object that will be holding this 20 the integer value will be look like this it will have a type which will tell that it's an integer it will have that object will have value which will hold the actual value and then it has a reference count now let's see what does this reference count mean in our next slides so the main garbage collection algorithm used by c python is reference counting the basic idea is that c python counts how many different places that have less reference to an object so if we go back to this one the reference count here is one because we don't have only one reference to it that is x variable if you if you have more variables which is pointing to this one it will get implemented now such a place could be another object or a global static variable or even a local variable so when an object's reference count becomes zero the object is marked for garbage collection or it is deallocated if it contains reference to other objects their reference counts are decremented those other objects may be deallocated in return so it can have cascading effect if this decrement make their reference count zero then of course and so on the reference count field can be examined using a function called getref count it is part of our assist module which is available in python and notice that the value written by this function is always one more than as the function also have reference to object when it is called so when you assign x equal to object the reference count was one when you call the function get reference count it is actually referencing the variable x so it has incremented to two now when you assigned a new variable which is y and you assigned its value as x so now you have three sorry so now you have actually the two again now again you want to check how much is the reference count to it so now it has become three sorry let me explain again when you have x created the reference basic count is one then you have incremented the reference count by using the function get reference count because it is reference index now you assigned the variable y as a value of x so now both of them x and y are pointing to the same memorial and the reference count of that particular python object that resides in that memorial location incremented to three now you deleted variable y so when you delete a variable y and if it is referencing to any of the other object then the value of the reference count of that variable will decrement so when we again run the get reference count it comes back to two from three that's how in general the reference counting works now there's a problem with this how should I say reference counting scheme it is fine when you have simple way of assigning a variable declaring a variable you have very simplest script everything will be fine for this case but the main problem with the reference counting scheme or relying completing on reference counting is that it doesn't handle reference cycles now let's see what reference cycles are so for an instance consider this code in this example x holds a reference to itself so even when we remove our reference to it the variable x the reference count never falls to zero because it still have reference to its own internal self therefore it will never be cleaned just by simple reference counting for this reason some additional machinery is needed to clean these reference cycles between objects once they become unreachable this is the cyclic garbage collector usually called as just garbage collector even the reference counting thing is also form of garbage collection but garbage collector is comprises of both reference counting and the actual mechanism which takes care of the objects which have reference cycle now there are there is a way of thinking that why do why do we have reference cycles maybe we can avoid having reference cycles which is fine you can leave that to your code and everything will work but sometimes it happens that it is required so let's have a look at that algorithm that's back and uses to detect these reference cycle is implemented in gc module the gc module is part of your python core internals and available just as os and this module is available the garbage collector only focus on cleaning the container objects that contain the reference to one or more objects these can be arrays dictionaries list custom class instances classes and extension modules etc one can say one could think that cycles are uncommon in the kinds of objects but the truth is that many internal references needed by the interpreter create cycle everywhere let's have a look at some of the notable examples the exception that contains trace is that objects contains a list of frames in the exception itself it's a very wide it's a very widely available example of reference cycle now module level function reference the module cherry which is needed to resolve globals and which in terms contain entries for module level function itself instances have references to their class which itself reference its module and module contain references to everything that is inside or maybe other modules and this can be back to the original instances so like this even if we don't want reference cycle is part of our code part of our implementation and we have to learn how to deal with it in long your run now let's have a look of garbage collections additional machinery which takes care of reference cycles because by now we have understood that it is part and parcel of our package in order to limit the time each garbage collection takes garbage collector in C pattern uses a popular optimization which is called generational just for the record this image was taken from a blog post it is a very clear image and really loved it and there's a credit to it in the end in part inside useful links and credits but this this is a very clear and if I created my own image it would have been this so I have reused the same image now the main idea behind this concept of generational garbage collection is the assumption that most objects have a very short lifespan and can be collectively up to their creation this has proven to be very close to the reality for many pattern programs as many temporary objects are created and destroyed very fast the order an object is the less likely it is that it will become unreachable to take advantage of this fact all container objects are segregated into three spaces which are called three generations each new object starts in the first generation which is generation zero list the previous algorithm is executed only over the object of a particular generation and if an object survives a collection of its generation it will be moved to the next generation so whenever new object is created it gets into generation zero list then garbage collection will run over it time to time the objects which have no references will be moved to discard list and they will be discarded the objects that have references will be moved to the generation one list and when the garbage collection of generation one live the generation one list will be run same mechanism happen and survival surviving objects will be moved to generation two list in generation two list you will have objects which are going to survive till the end of the your program so using this mechanism it it is easier because in generation one garbage collection can be run less often than generation zero because generation zero is supposed to carry the objects which are going to be created fast and deleted fast then generation one the garbage collection has happened less often and garbage collection in generation two happen less often compared to generation one so this way we don't have to keep on running garbage collection on the complete list we have segregated them into generation based on their references but this is one of the mechanism or this is one of the optimization that helps and limiting the time spent by our policy pattern in garbage collection now let's have a look how we can actually look this data of generation in our python code so generations are collected by number of objects that contain reaches some predefined threshold which is unique for each generation and is lower for the older number of generations these thresholds can be examined using in GC module you have get thresholds if you use this you will know how much is the threshold after which garbage collection will run on a particular generation list by default python have a threshold of 700 for the youngest generation and 10 for each of the two older generations you can check the number of objects in each generation using get down like here in in generation zero you already have 596 objects but in generation one you have two in generation two you have one now as you can see python creates a number of objects by default before you even start executing program you can trigger a manual garbage collection process by using GC collect method so here you have 595 2 and 1 which which existed even just you type python in the console and you will get that now you will run GC collect method and running a garbage collection process clean up a huge amount of object because if you run the if you run the collect and then you again run the GC count you will get how much objects are remaining here there are 577 objects in the first generation and three more in the older generation which are cleaned up the best part here is that you can alter the thresholds of triggering the garbage collection here this is something where you cannot change how the reference counting reference counting work but you can change how frequently the garbage collection run in these generations now how are you going to modify the values here there is a function in GC called set threshold using this you can modify this value in the example here we increase each of our thresholds from their default increasing the threshold will reduce the frequency at which garbage collector runs this will less competent this will be less computationally sensitive in the program but the catch here is it will keep the dead objects around longer in the memory so maybe there is a possibility that some here like I said like we have said the threshold that after 15 objects garbage collection should run on generation two and if there are there are objects whose reference count has gone zero in generation two they will still lie around unless either the threshold of 15 is reached or there are some which keeps checking that okay there are you can just delete those objects now one of the catch that I have noticed in not the catch I would say sometimes not always it is it happens very rare that people turn off the garbage collector all together and manually manage it that is also feasible but it is advisable not to do that with these we conclude our talk I hope it is helpful to you garbage in the summary garbage collection is implemented in Python in two ways reference counting and generational when the reference count of an object reaches zero reference counting garbage collection algorithm cleans up the object immediately if it has cycle reference count doesn't reach zero you wait for generational garbage collection algorithm to kick in and clean the object while as a programmer you don't have to think about garbage collection in Python it can be useful to understand what happens under the hood because maybe maybe you need some manual garbage collection to be run in your program so with this thank you for your time thank you very much thank you very much for your talk we can still take a few questions if somebody has one so let's have a look if somebody is going up and yes making the way to the microphone so please ask your question many many things for your talk I finally understood what is generational garbage collection I never exactly understood it I have one question we we use Python in control systems and we tend to run Python processes for very long times months and sometimes even a year or two and we have observed that whenever we have a high peak of CPU usage in the machine where it is running it's like the garbage like if the garbage collector stops working and we saw that we have a huge increase of memory and then it never cleans up do you have an explanation for that so this is my assumption because based on what the data you have provided your whenever you have spike in CPU you notice that the garbage collection doesn't kicks in am I right doesn't excuse me what you said the garbage collector doesn't I said that whenever there is a spike in CPU that means CPU is getting overloaded garbage collection doesn't have don't happen and the memory doesn't get free exactly yeah exactly right because so here's the thing when you run a garbage collection you have this memory to be freed up and it needs its own CPU cycles if you already have CPU cycles which are using use for computation somewhere else then of course it has it will get queued up in operating system that hey wait we have this computation already overloading the CPU and you have to wait for that so that's one of the possible reasons for it so but yeah that's what I assume happening here okay and as long as the CPU cycles are available the operating systems say that hey garbage collection let's go and clean up your garbage so that memory get free but yeah it can be a cyclic you know deadlock CPU is not free the computation is blocking it garbage collection was not able to clean up that is filing up the memory and filing up the memory the computation is not able to finish and CPU is getting spiked up so in that case you have to figure out there must be some kind of memory leak or you have to restrict how you are how the CPU utilization is spiking up because in production environment we say that if your CPU utilization is more than 70 percent that means you have to spend a new instance so that the traffic get even redistributed or the load getting redistributed okay thank you very much that's all we have time for today so let's have another round of applause for Patiba thank you