It really is amazing that some people are so desperate and so desirous for people to be able to lose their salvation. And because there are no examples of anyone actually having salvation in the Bible and losing it, that people will even turn to someone who is called by Jesus a devil who was prophesied to betray him. They would say that that person was indeed saved and then lost his salvation. Obviously, I'm speaking of none of them Judas and there were those that would go on to say that Jesus, you would have to conclude that Jesus was wrong, that there is a contradiction in the Bible because that's the only way that you can say that Jesus has chosen him, but then he also lost him. Or that Jesus says that I will lose none, but then he lost the one. So let's go ahead and look at the scriptures and let's find out for a fact that Judas was never saved. Judas was never a Christian. Judas never had the Spirit. The Bible never says he does. As a matter of fact, the Bible even says the opposite. So let's go to some of the relevant passages. Let's go to John chapter 6. Let's start there. So in verse 37 of chapter 6, Jesus says, all the Father gives me will come to me and no one will come to me, and the one who comes to me, I'm sorry, will certainly not, will certainly not cast out. Now, someone will say, yes, he won't cast you out, but you can cast yourself out. You can leave. But the problem is Jeremiah 32 verses 39 and 40 says that when he puts his Spirit in them, that they will never turn away from him, nor will he turn away from them. So it could not be that a person can have the Spirit of God and then turn away from God or God will ever turn away from them. That would be, my friends, an example of a contradiction. And so he says, I will not cast him out. And so you couldn't come back and say, yes, he won't cast him out, but you can cast yourself out. You can leave yourself. And that's what Judas did, that it's not what happened with Judas, continuing. He says, for I've come down from heaven to do my, not my own will, but the will of him who sent me. And here it is. Jesus says, this is the will of my Father, of him who sent me, that of all that he has given me, that all that he has given me, and notice this is perfect tense, which is a completed action in the past. So he has given all of those that he's given, whoever it is that he has given, all of them, they have been given in the past. And notice what he says. He says, I will lose none. Not only did he say that they will come to me, he says, I will lose none but raised on the last day. So if Judas is one of the all that was given, then Judas would have to be one that would not be lost. Judas would have to be one that would be raised on the last day. However, that's not the case for Judas. Judas is not saved. Judas was never one of those. But wait a second, didn't Jesus choose Judas? Well, he says so. The question is chosen for what? In John, in that same chapter, chapter six, let's go to verse 70. He says, no, you know, let's go to verse 69. He says, we have believed and have come to know that you are the Holy One of God. Jesus answered them, speaking to the 12. He says, did I not myself not choose you? The 12 and yet one of you is a devil. Now he meant Judas, the son of Simon and Scarlet, for he was one of the 12 who was going to betray him. Did Jesus know this? Of course he did. Jesus is God. This is not something new. Jesus calls him, calls him a devil. Yet one of you is a devil. This is before the betrayal started. Jesus knew of it. He says, I chose you. The question is chosen for what? Well, before we get there, because it's going to come up again in Acts one, when they're choosing a replacement for Judas and they said they're doing this to fulfill the Psalms. Well, what in the Psalms needs to be fulfilled? What is the Old Testament prophecies that need to be fulfilled? Well, let's go to a couple of Psalms. Let's go to Psalm 41-9. Even my close friend whom I have trusted, who ate my bread, look what it says, has lifted up his heel against me. Well, this same passage comes up again. We'll see it in John chapter 13, where this is also applied to or ascribed to Judas. Also in Psalm 109, look what it says here. Appoint a wicked man over him. Let an accuser stand at his right hand. Well, who was that? Well, that's going to be Judas. When he is judged, let him come forth guilty and let his prayer become sin. Let his days be few. And he says, let another take his office. Well, where do we find that? Where is this? Where is this found? This is found in Acts one, verse 20. Before we go to 20, let's start in verse 16. This is Peter speaking. He says, Brethren, the scriptures had to be fulfilled, wish the Holy Spirit foretold by the mouth of David concerning Judas. Where is that? In the Psalms that we just read and other part passages in the Old Testament also to speak of Judas. But by the mouth of David concerning Judas, who became a guide to those who arrested Jesus. Again, what did Psalm say about someone that would betray him at his right hand? We'll deal with this even more so about his hill being lifted up. And also what he says, for he was counted among us and received his share in the ministry. Now this man acquired a field with the price of his wickedness and falling headlong. He burst open in the middle and all of his intestines gushed out. And it became known to all who were living in Jerusalem so that in their own language, that field was called a dachma, that is the field of the blood. For it is written in the book of Psalms, which we just read, let his homestead be made desolate and let no one dwell in it and let another take his office. So the Bible has spoken of Judas in the Old. And so obviously Jesus knows exactly about this prophecy. Jesus is not clueless. Jesus knows who Judas is. Jesus knows that he was chosen for this. So the question is what was he chosen for? For this. He is going to fulfill the prophecy. He's going to fulfill what it stated of in the Old about someone who is named among them, who is called a friend, who is going to betray him. And even for the named amount, 30 pieces of silver and a field will be purchased from the money because of his betrayal. And the Bible says that there will be another that will take his place. Now, let's go back to Jesus and Jesus is going to say a few things that is very, very interesting. Now, remember what we can say is Judas never had the Spirit. He never had the Spirit breathe upon him, never had the Spirit in him, not once. How do we know? So, well, let's go to John 13. Jesus is speaking. Jesus is getting ready to wash their feet. And let's start in verse 8. Peter always being Peter says, Never shall you wash my feet, Jesus answered.  So what does Jesus say?  Peter said, Lord, then wash not only my feet, but also my hands and my head. Here's Jesus's response. He said to him, He who has bathed needs only to wash his feet, but he is, but is completely clean and you are all clean, but not all of you nuts. Look at the words that's used here. He who is bathed, this word right here is from the Greek word luo, which is to be cleansed. And this word is noticed in the perfect tense. He says, le lu menos, which means that he has been cleansed in the past. It's the perfect tense. Don't miss this. He says that all of you have been or whoever has been cleansed in the past is perfect tense, only needs to wash his feet, but is completely, highlights, completely, totally clean and you are clean. So Jesus claims and states that all of you are clean. All of them, is he saying all of them are clean? No, because he even tells us that not all of you are clean. He says, but not all of you. Well, who's the one? Who could it be? Who's the one person that Jesus is referencing that he says not all of you are? He says, for he knew the one who was betraying him. So obviously Jesus knows about Judas' intended portrayal. It's spoken of, I clearly in the Old Testament prophesied Jesus knows this, for he knows or knew of the one who was betraying him for this reason. He said, not all of you are clean. Not all of you are clean. So who's not clean? All of the disciples, except for one person, Judas. So what was Judas? Never, never cleansed. He was never cleansed according to this passage by Jesus. So you couldn't come back and say that then Judas was saved, but then lost his salvation. Go to John 1712. Jesus makes a statement. He says, while I was with them, I was keeping them in your name, which you have given me and I guarded them and not one of them perish, but the Son of Perdition, so that the scriptures would be fulfilled. The scriptures must be fulfilled. And so this person was chosen for that reason. Now I've kept all of them, not the one who I was not supposed to keep, not the Son of Perdition. He was never intended to be kept. As a matter of fact, Jesus just mentioned that he had even washed him. He had not been cleansed. So when people hear this part, except the Son of Perdition, as though that Jesus tried to keep all of them, but one got away. No, because if he did lose the Son of Perdition, if the Son of Perdition was one that was saved, if Judas was saved and he was chosen by God, and when Jesus says that I will not lose one of them, but then he loses one of them, just one, then we have a contradiction. We also have Jesus clearly being wrong. That is not the case. And so if we go to John 18, verse nine, this is when his betrayal comes about. He says, to fulfill the words which he spoke of those whom you have given me, I lost not one. Jesus said, now this is as he's being betrayed. Judas is on the other side. Jesus is saying this, knowing that Judas is betraying him. And what does Jesus say? Of those whom you gave me, I have lost not one. So he's clearly making a distinction between Judas and the other disciples. How wrong would it be for him to say I lost not one, yet the one that he lost is literally there betraying him. Clearly, Jesus knows that he has not lost one. Judas was never one that was found, one that was never saved. He was never cleansed. He was never part. As a matter of fact, what does the Bible speak about Jesus, about someone like Judas? The Bible calls someone like that an anti-Christ. He says, children, in 1 John 2, 18, he says, it is the last hour. And just as you've heard that anti-Christ is coming, even now, many anti-Christ have appeared. From this, we know that it is the last hour. Now, if someone were to say or to submit that Judas was never an anti-Christ, nope, that's not what an anti-Christ is. He's not an anti-Christ. Not the anti-Christ, but an anti-Christ. Well, that person's got some explaining to do. If you're going to make the case that Judas is not or was not an anti-Christ, he says, they went out from us, by the way, the same language is used in Acts, they went out from us. They went out from us, but they were not really of us for if they had been of us, they would have remained with us, but they went out so that it would be shown that they are not all of us or all are not of us. So, here we see Jesus even making the point, I'm sorry, John making the point, that those that were with us, you can be numbered among us. That means that you are part of us. Maybe most of the people there can't tell, but God can. Certainly, he can. And so, Judas was certainly not because Jesus always made references. As a matter of fact, even when he is speaking about, there's one here who's going to betray him. And when Judas asked, is it me? What Jesus said, it is as you speak. Yes, it is you. You're the guy. Jesus does not alert everyone to his impending betrayal because why? That's what he's there for. So that the scriptures will be fulfilled. And so, Judas, no one could even make the case that Judas was ever saved. Now, Judas never had the spirit. The Bible says in Romans 8-9 that however you are not in the flesh, but in the spirit, if indeed the spirit of God dwells in you. But if anyone does not have the spirit of Christ, he does not belong to him. If Christ is in you, though the body is dead because of sin, yet the spirit is alive because of righteousness. Judas never had the spirit in him. No one, because as a matter of fact, no one at the time had that. But at least, at least through some sort of working by God, the other 11 were cleansed except for the one. And so, guys, it should be painfully obvious, clearly obvious, abundantly clear that Judas was never saved. No one ever thought that he was. As a matter of fact, the scriptures prophesy his coming and what he would do and what does Jesus call him? A devil. And so, it would be kind of hard. And I think it speaks to some people being overly desperate to try to find somebody, an example of someone who was saved, who lost their salvation. There is not one example of a person having the spirit in them, being a believer and then losing their salvation. Certainly not Judas.


Hello everyone. My name is Julie Garden Robinson and I'm your host again for today's Field to Fork webinar. This is brought to you by North Dakota State University Extension. In fact, this is the ninth year we've done the series and we're really glad you joined us. We have archived all of the webinars, including the two previous ones and all of those years of webinars, so you can watch all day long. The link is always on the Field to Fork webinar page. The next slide shows the upcoming webinars and we hope you join these as well. You will get reminders every week and it'll tell you how to sign in. The next slide shows our webinar controls. Because we have a lot of participants, we invite you to post your questions and comments in the chat. So let's practice and find and use the chat box to tell us where you are. What city and state are you coming from? I'm going to ask you to ignore the Q&A box, the question and answer box. We're going to just use the chat. The next slide provides an acknowledgement. So keep telling us where you are, but I do have a special request. These programs have always been sponsored with grant funding from USDA's Agricultural Marketing Service, and I ask all of you to complete the short online survey that will come right to your inbox after today's webinar. And as a thank you, we are going to continue to provide prizes to the lucky winners of our random drawings. So be sure to include your complete address on the follow-up form. And one more note, as you go to sign in to do the quiz, you will need to scroll down to today's date, the ones from 2023 or first, and we're adding all the data together. So scroll down until you find our 2024 dates. Again, I welcome you to today's webinar, and I'm very happy to introduce today's speaker. He's a three-peat. He's done three webinars for us in the past. Byron Chavez is a food safety extension specialist at UNL University, Nebraska, Lincoln, where he provides training and technical assistance to food manufacturers in food safety, sanitation, and regulatory compliance. Byron is a native of Costa Rica, and he obtained his PhD in 2015 from Texas Tech University. So welcome, Byron. Thank you so much for the invitation. And like Julie said, this is not my first time in this Women Are Series, so I'm very happy to be back. And I hope that today's information will be useful to what you guys do. I'm very impressed seeing where people are located. I see California, I see Arizona, I see Alabama. And of course, lots of people in North Dakota. So like Julie said, I am Byron Chavez from the Department of Food Science and Technology here at the University of Nebraska, Lincoln. And the topic for today will be understanding, product dating, food safety versus food quality. And so something that I'm going to use today is a program called Slido. You will see it in a little bit. But Slido is a software that we use to keep people engaged. And so if you are able to do it, I will want you guys to use your phones or your devices to answer a few questions and we will see in a little bit how that works. Just to keep it a little to keep it a little more interesting. You know, it is Wednesday when I was going to say Wednesday night. It's not it's Wednesday at 2 p.m. I am on my third cup of coffee, but apparently it's not working. OK, let's talk about food safety and food quality. Right. So when we think about food quality, that is really the sum of all of the characteristics that we can measure about a food product. We think about shelf life. We think about sensory and organo-elected traits of that food product, including texture and flavor and all of those things that make food delicious. OK, but we are also thinking of interactions of the food with the packaging material and with the environment, as well as nutritional composition and many more things. So food quality, very complex term. It has so many different angles and many different perspectives. And of course, we separate those, right? So we typically talk about nutritional quality. We talk about sensory quality. We talk about many different aspects of food quality. Now, food safety is, of course, a portion or a part of food quality. But when we think about food safety specifically, we are talking about the assurance that the food will not cause illness or injury in the consumer when it is consumed or when it is used as intended, right? So if a product was intended to be consumed, cooked or processed, then we are not expecting that form of the food product to cause illness or injury in the consumer. Now, a question that we can ask is can a high quality product make you sick? Right? And so remember that we have many different aspects of food quality. And so something that it's delicious and something that is nutritious can still make you sick, right? And so there is this little bit of divide between food safety and quality. But I just want you guys to keep in mind that food safety is just one of the many aspects of food quality. So now let's get into today's topic, which is product dating. And I am talking about dating on the labels, right? I'm not talking about dating in any other in any other aspects. Let's talk about product dating. Two types of dates that we can find on food products. And so if we take a look at labels, which hopefully you guys do, reading labels is one of the most interesting things that we can do as consumers, but also some of the most one of the most educational things that we can do, because we can figure out allergens that can be present in foods. We can figure out so-called expiration dates that we will see in a little bit, but that there's no such thing as an expiration date. We can take a look at ingredients and figure out what are the things that are given this product, a specific color or a specific flavor, those sorts of things. So I encourage everybody to read labels. OK, two types of dating that we get of types of dates that we can find on labels. The first one is called an open dating system or an open date. And this is typically a calendar date or, you know, something like what you see in the picture for milk. This one is a best by December 30th. OK, so that is an open date. The date provides information on the estimated period for which the product will be best will be of best quality and to help the store determine for how long that product is going to be on display. So very straightforward, we have a date. Now, there is another type of product dating that it's closed, that it's called closed dating, and this is a code that consists of a series of letters and or numbers that are applied by the manufacturer to identify the date and time of production. And it is very common in canned foods and box foods. We will see that we will see an example later, but you can see the comparison between open dating and closed dating in the picture in your lower right and the lower right side of the screen. So open dating, very straightforward, that is a date. And in the closed dating is a code. And more often than not, we will not know what the code means. Only the manufacturer and sometimes the retailer knows those things.  You can take a look at the screen once you get on to Slido.com. It will ask you for a number and the number is also on the screen 3960321. So I'm going to give you guys 30 seconds, well, maybe less than that, to get on to Slido.com. I see responses, responses coming, so that's good. There's one response. We still have many, many more responses. OK, people are participating. I'm excited about this. And I'm sure that some of you are already familiar with Slido, but I really like this program. It helps us keep it a little more interesting, keep it a little more dynamic. Thirty people have responded. Wow, OK, good. Yeah, keep them coming. And yeah, questions as we go, please put them in the chat. And Julie, we will be monitoring those just in case I miss any of those questions and when we can discuss at the end. OK, I'm going to let people continue to respond. But so far, OK, the question is is food product dating required by federal law? OK, so a lot of people are saying yes. Some people are saying no. The numbers are still changing a little bit. But let's look at the answer. OK, 80 people responded. Most people are saying yes, but we're going to move to the next slide and this is going to give you the answer. So unfortunately, the answer is no. Well, for those that got the answer correct, that is very fortunate. But dates are not an indicator of the product safety. So product dating is not required by federal regulations. There's really nothing in the regulation that will tell you that you have to put a date on your product. Now, there's always exemptions, of course, exemptions. And the one assumption in this case is infant formula. So, of course, it's a very high risk food commodity, so it has to have a best buy date, but that is really the only one that shows up in the regulation, whether it is for FDA or for the USDA. Now, for FSIS inspected products, which will be meat, poultry, processed eggs and catfish, dates are voluntary, right? So all of them can be voluntarily applied to a food label, but they have to be truthful and not misleading. They have to have a calendar date that it's expressed as a month and day of the month. And if the product has a really long shelf life, like a frozen product or a canned food product, then the year must also be added. OK, but the reality is that we don't need dating on the labels. Now, dating can be used for many, many reasons, but it is not required for federal law. It is a little bit of an expectation that us as consumers, that we as consumers have. OK, so let's move forward with this. So you might be wondering what factors influence the dating process, right? So if you've attended any of my webinars before, you've heard me talk about intrinsic and extrinsic factors that determine microbial growth and microbial stability of food products. And so when we think about intrinsic factors, those are factors that are characteristics of the food product, including the pH, the water activity and the chemical composition, but we also have extrinsic factors such as temperature, relative humidity and the packaging conditions, right? So whether we package something in, let's say in a vacuum versus a modified atmosphere environment, if we temperature abuse a product, if we do an abuse of the relative humidity, all of those things will have an impact on the quality of the food product. And of course, may have an impact not only on sensory quality, for example, but also on the microbiological safety of the product. For anybody that is developing food products as food entrepreneur, starting their companies where they are producing food, I will always tell them the same. The first thing that you need to characterize is your product. And that means, in general, measuring pH and water activity. OK. Now, the quality of perishable products may deteriorate after the date passes. And I will show you an example in a second, but they should still be safe if handled properly properly. So basically what this means is that even if we go past the so-called expiration date of a food product, the product safety may not be compromised. There are many other aspects of quality that we've already discussed, whether it is nutrition or flavor or texture that may be impacted. And so by the date that the product is to be used, it doesn't necessarily mean that microbes have started to grow or that there is a risk associated with consumption of that product. Now, there are, of course, situations when that is the case. We'll discuss those later. But you guys can take a look at the image on the screen. And so, of course, those are eggs. And so we know that the fresher the eggs, the yolk will be significantly more contained, and as the eggs age, the yolk and the whites are going to be a little more runny, so the yolk will not be as contained. But that doesn't necessarily mean that it is unsafe to consume that egg. So this is a really good example of just because the product is past the so-called expiration date, it doesn't necessarily mean that it's going to make someone sick. Now, in what cases would would the product make somebody sick? Well, if we see that there is microbial growth, that it's evident, right? So think about molds in bread. So if the bread is full of molds, obviously, we don't want to consume that because we run the risk of getting contaminated with microtoxins that may have been produced in that product by the fungus. And there are other examples where clearly it's not a good idea to consume that food product, but if the sensory characteristics haven't fully changed, if we don't see evident microbial growth on the food product in general, the product is safe to be consumed. The use by date, which is typically what we call the expiration date, is selected by the manufacturer, packer or distributor based on the product analysis to shelf life test and other technical and scientific information. So determining what is a use by or a best by kind of date is actually fairly complicated, especially for the variety of manufacturer foods that we have that are distributed under so many conditions of distribution or transportation and storage, and that is why every time that we enter a grocery store, for example, the environmental conditions are similar because we don't want the conditions at retail to affect the shelf life or the sensory characteristics or the safety of many of those products that are in display at retail. Right? So we walk into a supermarket. Typically, the relative humidity is roughly 45 percent, right? And that 45 percent, it's not a magic number, but it's a number that will prevent dry foods from absorbing water and it will prevent moist foods from dehydrating very quickly. And so we are preserving the sensory and other aspects of quality of those food products. But there's many other factors and we'll discuss those in a little bit. OK, when we think about descriptors for product labeling, we usually think about expiration date. And I already told you guys a couple of times that there's no such thing as an expiration date, but we have other descriptors. We have a best if used by or best if used before whatever date. But this is a date that indicates when the product will be of best flavor or quality. So this is, again, very driven by sensory characteristics. It is not a purchase or safety date. We don't have any of those things. The only one, like I said, is for infant formula. So this one, it's the most commonly used, right? It's the best if used by or best if used before whatever date. But we can also see a sell by date that tells the store how long to display the product for sale for inventory management. It is not a safety date, right? This is just telling the store. You should probably sell this product by this specific date. And so that's when we go to the supermarket and we see things that may be on sale, right? Because they have two days. There's one day or two days before the established sell by date. And so the retailer wants to make some profit out of that product. And so it is now on sale. There's also a use by date that it is the last date recommended for the use of the product while at highest quality. It's not a safety date, except when it is used on infant formula, like we've expressed before. So the use by is just for quality, right? This is typically what we will see on eggs because of the chemical transformations that happen to eggs during storage and refrigeration. We typically have a use by date, which is when the eggs are going to be best for consumption, just from a quality perspective. No one wants to break an egg and find a super runny, runny content. Or at least I don't want to. So we take a look at those dates. OK. And then the last one, it's a freeze by. Right. So we typically see consume by or or or a use by or freeze by kind of combination, right? So the freeze by is if you didn't consume that product that was in refrigeration, then by this date, you should consider putting it in the freezer so that it lasts much longer. OK, these are the four most common product label in the scriptures that we have out there. Like I said, there's really nothing that calls a product expire, right? So a product doesn't expire. This the quality of the product, the subjective quality of the product may expire, right? So for not for everybody, a product is not hasn't lost. It's appeal or doesn't lose its appeal for everybody at the same time. Right. So there are lots of variations there. OK, another slide.  That is a question that I have for you guys. At two twenty p.m. on a Wednesday. It's still a Wednesday. Yes, it is. Right. Yes, I'm not crazy. OK, maybe a few more. A few more are. Responding. I'll give you guys 10 more seconds. OK, so I see that we are leaning towards the yes, right? Or as sometimes. Some people would never eat a food that has expired. OK, like like I said before, this is all very subjective, right? So we all have our own biases as to what is high quality foods. And so we make that choice, right? OK, so most people do eat food past their expiration date. And I would say that in general, this is good, right? Because when we think about dating food products, putting a date on the label, this also takes care of a lot of the food laws and ways that we find in the food production chain. It is estimated that 30 percent of all of the food supply, that is all of the meat and poultry and eggs and produce and everything that we find is lost or wasted at the retail and consumer levels. This is literally hundreds of thousands of tons of food that it's wasted on an annual basis. And yeah, one of the comments is actually funny. A lot of a lot of the people changing their minds on the I would never write. And, you know, again, we are all allowed to change our minds or change our opinion in, you know, in the presence of new evidence that something is safe or not. OK, but going back to this point, of course, we know that food is lost and wasted severely in the United States and many other industrialized countries. But the reality is that foods not exhibiting obvious signs of spoilage should be wholesome and maybe sold, purchased, donated and consumed beyond the labeled best if or used by dates. Right. So we know that those are dates that are recommended for quality of the product and not necessarily for safety, except, and you guys know already, for infant formula. OK. If the date passes during home storage, a product should still be safe and wholesome if handled properly until the time spoilage is evident. Now, sometimes when a product is manufactured and packaged, we don't know if there's any spoilage in that product until we open it. And of course, if we open it, we have to consume it, especially if it's way past the expiration date, because then now we do risk that the product is going to get spoiled. So think about canned foods, right? So if you think that a can has been abused or that might have signs of spoilage, that is a big no, right? Like those are things that we shouldn't consume. And I think that we will talk about that in a little bit. But if the product doesn't have obvious signs of spoilage or it may we have fresh produce that is a little spoiled, but there's a portion of the product that it's salvageable, I would do it, right? We are avoiding or we're preventing food loss and waste in that way. OK, food spoilage can occur much faster if the food is not stored or handled properly, right? So we have abuse temperature that it's the number one, number one driver for spoilage. Or if we abuse relative humidity, right? So if we put the product, if the product is supposed to be kept in a cool, dry place and we have it in a place where it's exposed to moisture and things like the example that I have here is that a changing color of meat or poultry is not an indicator of spoilage, right? So even for fresh products like meat and poultry, sometimes some of those sensory characteristics are rarely indicative of true spoilage of the food product. OK, let me take a look at the chat for a second and see what's in here. OK, there is a huge disagreement. Let's see, do federal food programs accept these definitions regarding whether food can be transferred to consumers at a food bank or pantry? The answer is yes. And I can direct you to some USDA guidelines for food donations. OK, there is a huge disagreement between friends that make salsa. OK, let's put this in context, right? So salsa typically made of tomatoes, spices, acids, right? So it is a typically a high acid food. High acid food. Some say that it is good for a year or so. And others say it is good forever. This is the right answer. OK, very specific. You guys must really be into salsa. So I would say it depends just like any food microbiologist is always going to tell you it depends because microbes grow under so many different characteristics and circumstances, right? So assuming that the pH of the product is below four point six, which is the pH that effectively prevents the outgrowth of clostridium botulinum, which is a very dangerous microbe and that the process that the product has been thermally treated to eliminate all of the vegetative cells of pathogens, then the product has, I wouldn't say any definite shelf life, but has a very long shelf life, right? Especially if it's canned or maybe it's in a glass and was maybe a septically packaged. So basically it was put in the in the glass container hot and then it got cooled down after that. So there's many different circumstances now for the salsa that we can make at home without having a pH meter, I would say, probably serve and consume, right? I wouldn't keep that product for for much longer. But we can touch on specifics later on. Yeah, and that is a good point, right? So check local requirements because there are different requirements that are local county or local municipality or health department may have. OK, so let's start moving on here. We've already talked a little bit about this closed dating that happens on cans and pouches, right? And so we can have like the first example that you guys see there. It's closed dating. It's a code and we can typically decipher the code for the one at the bottom. It does have a used by or a or a best by date, right? So that it's open dating. So we can see we can see both of those things on cans and pouches. Those numbers typically refer to traceability, right? So that we can trace back if there's a problem. Unless otherwise stated, the codes are not meant for consumer to interpret as a best if used by, right? So those are just for logistics and distribution and lots of other things that happen in the food industry that are not related to food safety necessarily. Now, this takes us back to the comment with salsa. Of course, salsa is not necessarily canned in the in the traditional way of using a can like what you see in the example. But we know that high acid canned foods such as tomatoes and fruits will keep their best quality for 12 to 18 months, after which we may not see a complete deterioration of the product, but the can may actually start deteriorating. And so that that limits the shelf life. But however, for low acid foods, so that that would be foods that are closer to neutrality in pH, they can last for much longer, typically for up to five years. Right? I've had cans in my pantry that I've had for five years and they are still good. Now, do I want them after that much time? Right? Because the way that I think is, well, if I didn't eat this for five years, do I really want to eat it now? And so then, of course, that is a contribution to two food laws and food waste and so then keeping in mind the amount of food that we keep in a pantry is important. Now, one thing that I really want to say about cans right here is that cans that are dented, rusted or swollen should be immediately discarded. We don't want to consume that product. Can anybody place in the chat maybe? Why is it that we don't want to consume product from cans that have been dented, rusted or that are swollen? What is the problem with that? So in the meantime, I'm going to check the chat. I would say in the case of the pasta sauce that expires, is that true? As it is mostly tomatoes, I would mostly think that that product is not acidified to a point that will maintain microbiological stability for a long time. How long is it to say consume salad dressing once they are open? OK, you know, that is a really good question. Salad dressings like ranch and things like that are very stable. But you will start seeing if you don't consume it in a period of time, I would say, without, not without within four weeks is two things. One for quality, right, where the faces separate. So you will see separation of the separation of the faces where some of the solids will be will just separate from the liquids or the oil and water mixtures will start to separate after some time. And so you will have to just shake it vigorously. And the other one is that on the very top, the one that it's closer to the opening, you will see yeast that will grow, right? So salad dressings, the most common spoilage organism is yeast. Yeah, botulism, great. So botulism is so many people telling me, botulism, you guys are on point. So botulism is the problem with dented cans. OK, let's keep going. So otherwise, I'm never going to finish. And I'm sure that you guys don't want to listen to me all afternoon. OK, coats on egg carnage. Eggs are another perfect example. There's no expiration dating that it's required, right? We've already established that by federal law, but many states may have different requirements for eggs. OK, so check your local and state. Well, check your state, local and tribal regulations, depending on where you are located. Now, what we see on eggs is the example that you have on the screen. So there are multiple numbers. One is the pack date. One is the sell by, which we know is, of course, for quality. And the other one is a planned number if this is a facility that is inspected by the federal government. So the pack date is the day that eggs were washed, grated and placed in the carton using the guidelines from the USDA agricultural marketing service, which is the agency within the federal government that creates eggs, right? So grade A, grade A eggs that are the ones that are placed in retail. OK, why do we refrigerate eggs in the US? This takes takes us to something that it's not necessarily related to to labeling, but why is it that we refrigerate eggs in the US? Pretty much anywhere else in the world where we go, eggs are just kept at room temperature. Yeah, great point on the seam. It's the most hazardous, that is for sure. Because that is where potential openings have been made. OK, yeah, our eggs have been washed, right? So in the US, the way that eggs are processed is that eggs are washed. And in the washing step, the outermost protective layer of the eggs is compromised. It's called a cuticle and that cuticle has a lot of protective effects, right? So we wash the eggs, sometimes we wax them. There's a lot of things that happen. And so in order to prevent introduction of microbes into the egg, then we put them in the fridge to delay that process of microbial proliferation on the surface and preventing the eggs from spoiling, right? We also prevent or delay the exchange of gases, which is what happens when the egg inside becomes really, really runny. It's because there's there's been a lot of release of carbon dioxide. And then the contents of the egg just become really runny. OK, maybe you guys can also tell me in the chat, what is the microbe that we care about the most when we think about eggs? I should have put that on the slide also. Salmonella, yeah, OK. Wow, you guys, Julie, you really have the best audience. My audience is never as engaged as this. You guys are very knowledgeable. OK, excellent. Salmonella is a hazard recently likely to happen in egg products, right? And so it is it is definitely a hazard to consider. OK, let's move on here. Other codes that are used in packaging. I'm not going not going to go into detail of this, but there are many other codes that we can see on packages and that is typically related to logistics of the manufacturer and the retailer. One of those is the universal product code and the other is the stock keeping unit. And we may start seeing many more types of barcodes or numbers that will help us trace product or that will help us, let's say, identify quality characteristics. Or if there is a recall of the product, there will be other codes that will be added in the future. And so manufacturers have already started playing with this where you can just scan a code and it will tell you if the product has been subject to a recall or if there's an outbreak or if there's any issues with that with that product. But in general, many of the things that we see are honestly related to logistics and manufacturing and transportation and have absolutely nothing to do with safety. So talk about product dating and you, you as the consumer and you as food entrepreneurs and you as people that may be thinking about developing food products. So the first thing that we want to do, like I said, is to know your product. We want to know the intrinsic properties of the products such as pH and water activity. And you may be wondering, where do I do that? You can send samples to a lab, right? And us as extension specialists, Julie or myself or anybody in extension that you reach out to, should be able really to point you in the direction of where to send your samples to be tested. That is number one, right? We want to know what it is in your product. I forgot to mention this or to put this on the slide, but you also really want to have a consistent recipe, a consistent formulation of how you make your products. Let's say the example of salsa. You change the variety of tomatoes and your pH is always going to be different. There's so many varieties of tomatoes. And once you process them, the pH will always be different. And so you really want to standardize those characteristics. The second thing that we want to do is to determine the conditions of storage, transportation and common handling for the food product. How are people using that product? How is it going to be transported? What are the conditions that will affect quality during transportation? Could it be that the product is going to be temperature abused? How long is it going to be in transportation? All of those things we really want to know, right? Even if we just take it to the convenience store down the hallway or down the street, right? Where what is the condition of temperature and relative humidity in that environment? Now, something really important about that, that bullet number two is including the unintended uses. And as food entrepreneurs and food producers typically fail to consider the conditions under which the product is not supposed to be used, but the consumer will use it in that condition, right? So consumers may do something to the product, may, I don't know, open it and not refrigerate it or the consumer may abuse the product in some some specific way or the product is supposed to be cooked at certain temperature, but the consumer doesn't like it at that temperature and they will cook it at a lower temperature. All of those things happen. I'll get back to the question in a second. So we need to also understand the unintended uses of the food product. Number three, prioritize your own quality parameters and conditions, right? So under which characteristics of temperature and relative humidity is your product best, right? So you made your product, you know, what are the sensory characteristics and what is the expected high quality, relative expected high quality that you have for that product? So you can also start setting specifications on how you want your product to be stored and be handled. You can run your own observational study. And of course, this is just observational and it's a little bit it is a little bit anecdotal, right? It's a little bit of just an anecdote of things that you see. But you can process your food product. You can leave it at the normal conditions that you would use for storage. And you can also temperature abuse that product or do an abuse of relative humidity or damage the package in some way and see what are the things that are happening to that product. Of course, at the end of the day, we would want to have a shelf life study that somebody else has done for us, whether that is a third party testing company or a university consultant service, those sorts of things where where we have maybe more objective information of what are the quality parameters and how conditions of the environment and temperature are affecting the product. Let me take a look at the check. Well, you guys maybe think of other questions or anything. OK. Do you know where to find information referring to the codes on cans means you said most are not correlated? I would say that a lot of those things are a lot of those things are company specific, right? So lots of internal traceability, lots of internal internal logistics codes related to the country of origin. There are some codes for country of origin, but in general, we find on the label if the product is imported or most of the contents are imported, the product will say product of Costa Rica or product of Canada, right? So those things will typically be on the label and not necessarily on a code. OK, so if there is a scenario where a family restaurant wants to produce and wholesale their special sauce, do they need to be processing the recipe in a manufacturing plant in order to bypass the seven day requirement that the FDA or local health departments would look for? I would say I'm not familiar with the North Dakota food code, but more likely, yes. So if the moment that you start actually commercializing a food product, well, you put it on display at a store, even if it's just the front store of your restaurant and sell that and assume that the stability of the product is higher, yeah, you will need to check your regulators and make sure that the product has been adequately treated. Elevation, it's a great consideration. I totally agree, right? And there's multiple guidelines out there on processing temperature and time, right, based on elevation, because those things, of course, have an effect. OK, I have something else on this slide, and it is that keep in mind that product dating is not mandatory unless you are producing infant formula like we've said a number of times. But I would encourage you that if you are developing a recipe, developing a product that you follow these steps, right, that the first thing that you want to do is to test. And there was a question in the chat about where to test, I believe, do you need to test pH when canning jams and jellies? Oh, that is a good one. For me, that's a good question. Number one, because I don't know exactly if you need to do that. So I'm going to I'm going to I'm going to play it safe and I'm going to say I don't know to the question of the jams and jellies, the amount of sugar that it's present in those products. It's typically very, very high. And so the microbial stability is high. Maybe Julie knows the answer to that. And so she can help me out. Yeah, OK. OK, very good. OK, let me see what else I have for you. OK, this is not necessarily related to product dating. But Julie did mention that you guys or some people in the audience have been curious about what vegetables can be stored together. And really, the answer to that is when I started looking this up is which, by the way, I took this from a Purdue University Extension Service. Really, the answer to this is the things that we shouldn't mix are things that are high ethylene versus high versus low ethylene. So ethylene is the maturation hormone of plants. And some fresh produce produce much more ethylene than others. And so in general, fruits produce much more ethylene than vegetables. And so we don't want to keep those things together right side by side, just because of the amount of ethylene that it's produced in the fruit that it's going to, you know, mature the vegetables, it's going to reduce the shelf life of the product and can lead to spoilage, even in refrigerated state. OK, let's see. The guidelines here are ripened on the counter first and then refrigerate for a lot of these fruits, right? And the refrigeration will slow down the production of ethylene, but it's still produced, right? It's a gas that produce that fresh produce produce for maturation. And there are recommendations on store only at room temperature, and that is mostly for quality purposes. I am one of those people that refrigerate all of their fresh produce mostly because I don't consume them quick enough. And I feel like I need to like I need to refrigerate them. So something that I typically do is wash and dry, right? Especially if it's something like berries, right? I will typically wash them and add them dry and then store. And of course, once you remove the moisture from the surface, you will extend the shelf life of that product. But I wanted to put this out there. You can you can find this this Purdue extension. If you look for storing fruits and vegetables, information from them. OK, so that is what I have for today. Let me go ahead and stop sharing and then I'm going to take a look at the chat. I did one that maybe you've had in your state. Is there a scenario where a family restaurant wants to produce and wholesale their special sauce? Do they need to be processing the recipe in a manufacturing plant in order to bypass the seven day requirement that FDA or local health departments would look for? Yeah, so this is a great question. Let's assume that the sauce has a pH of below 4.6. Let's talk about two scenarios. One with the product is acid or acidified and one where it's not. Right. So if the product is low pH 4.6 or below, OK, the pH is maybe effectively helping prevent the outgrowth of the stream. We put it in the fridge if necessary. But what typically happens is that we find a lot of these sauces at room temperature, right? And we don't really know if the microbial stability is kept at room temperature. Right. So we go to a lot of restaurants and that actually recently happened to me where I went to a restaurant, a Mexican restaurant.  They have three for different sauces that are made in-house. You bring them to the table and then you can be there for a couple of hours eating and drinking with your friends and you still have those sauces at room temperature, right? So I I'm a little I would be a little cautious about that. I would ask the health department or, you know, whatever jurisdiction you have to make sure that the pH is sufficient to make that product shelf stable and that if it is refrigerated, maybe what needs to happen is that you have smaller containers that people can take to the table that you don't return to refrigeration for other people to use. Does it have to be produced in a manufacturing plant? Not really. OK, let's see. What else do you refrigerate bananas? So I don't because I don't like to eat cold bananas. But bananas can be refrigerated, but it doesn't. They continue to respirate in refrigeration, right? And bananas are one of those fruits that produce a lot of ethylene. If you want to ripen something really quickly, put it next to a banana, right? So bananas will produce a lot of ethylene. I think, Julie, you should you know more than me on those things. So hopefully I'm not making this up. No, you're good. You're good. There's another new question. Eat by date site specifies length of time safe after eat by date, still tasty website does not specify. Wonder wonder if their time periods are shelf life from manufacturer. So I've never seen an eat by date. And I would also advise that this is not an expiration or a safety situation. This is, again, some descriptor that the company is using to say you should have this product, usually this product by this state, otherwise it's not going to taste great. And so that also helps the manufacturer say, you know, prevent a lot of complaints, right? Because if you taste a food product and it doesn't taste optimum or optimal, you might be willing to or tempted to call that manufacturer and say, hey, this doesn't taste great. Well, one of the first questions that they might have is did you consume it within, I don't know, seven days or whatever? So so I think that it's also a way for manufacturers and companies to push their peak quality parameters on the consumer. OK, National Center for Food Home Food Preservation's website for safe home canning is from the University of Georgia that is 100 percent correct. And they are they have a new person. Her name is Carla Schwann, and she's been trying really hard to update recipes and and lots of things. So there's a lot of information, hopefully, that's going to come out of the center again. OK, not date. Oh, yeah. Sorry. Go ahead. They have a new website. So if you've never been to the National Center for Home Food Preservation, they have a brand new website. So check that out. OK, very good. I haven't checked it out. So I'm going to have to. OK, let's see. Not date related, but can you address condiments such as ketchup that isn't refrigerated, but label says refrigerate after opening? Is it safe to say this is only quality related or are there cases where it's safety related? It's a great question. OK. It is mostly quality related. What is the issue with ketchup? Ketchup oxidizes really quickly. And so you put you put you leave the product out. Well, let me back up there. One of the quality issues with ketchup is oxidation. The oxidation changes the flavor really quickly. It doesn't necessarily have to with safety. In that case, it is more for a quality parameter. Now can refrigeration really stop oxidation? Not really, but there's other quality parameters that may be affected. I don't know the pH of ketchup. I would anticipate that it's not particular. I mean, it is acidic, but I don't know if it's below four point six. I would actually I don't think so. Julie, I don't know if you know the pH of ketchup. It's an acidified food. So it's probably three point so below four point four. Yeah. OK. So it is it is very shelf stable. That product has also been thermally treated. I would say the refrigeration is for quality. OK, once I have a leftover like chicken and I put it in the fridge, I put a date on it. Usually I throw it out after four days. Is that OK? Yeah, that is great, right? So I would say four days, sometimes seven days for cooked foods is OK. I would say for animal protein, four days is what you want. Yeah. OK, let's see if we have time for a couple more questions. Yeah. Yeah, there's a common common food canning in in fresher canning. Yeah, Nebraska came up with the four day throw away message. OK, that was probably before my time or maybe not. I don't know, maybe just haven't been paying attention. But the four day rule is a really good rule. The USDA, however, will tell you that cooked foods or leftovers should be thrown away away within seven days. I think it's what they said. OK, well, I feel like we've addressed Julie, most of these questions that you guys were excellent. Excellent. They all get an A. Yeah, yeah, certainly. You guys really put me to work in this in this presentation. So I appreciate that. So I have a question for you. I was in Costa Rica one time and had the most delicious coffee ever. So were you drinking coffee from Costa Rica today? At home, yes. So in December, when I went to Costa Rica for Christmas and New Year's, I brought back a lot of coffee. But the one that I use in my office, which I just drank a cup, it's just curic, normal curic. That's for the afternoon. When I don't care about the flavor of the coffee, I just care about caffeine. But in the mornings, definitely Costa Rican coffee. Yeah, very good, very good. Well, we'll let you get back with your day, but I certainly thank you for your time. And I loved your interactive activities. That was a lot of fun. So thanks to everyone for joining us again. We'll be back same time, same place next week. And be sure to fill out the survey and slide down, scroll down to the twenty twenty four dates so you're answering the right question. So again, thank you, Byron. Thanks, everybody, and hope to see a lot of you next week. Thanks, everybody. Bye.


Hi, everyone. Thanks for being here. Thanks for the intro. As mentioned, my name is James Starr-Urwini. And today, I'm going to be talking about downloading a billion files in Python. So really, this is a case study in trying to figure out whether we should be using multithreading, multiprocessing, async.io. And to help motivate this, let's say that we've been given a task. So say we come into work and we've been asked to do something. So our task is that there is this remote server that stores files. And these files can be accessed through a REST API. And so our task, what we want to do, is download all of the files from this remote server onto the client machine that's going to be running our code. So seems, at least conceptually, pretty straightforward, right? And if we ask for some clarifying details, we want to make sure we understand the problem entirely. So we might ask, OK, so what machine is this going to be run on? Do we have a cluster of machines? Is it just a single machine? And so let's say, for this task, we have one machine that we can use. It has 16 cores, 64 gigabytes of memory. So it's a machine that has a good amount of computing power here. And we want to know about the network. So what about the latency between the client and the server? Are we making calls across continents? Is it in the same network? What does that look like? And we'll say, for this purpose, that the client machine is on the same network as the service. And then we ask, how many files are on the remote server? And as the title suggests, there are a billion files that we need to get. But fortunately, the files are fairly small. And we're talking really small here, 100 bytes per file. So fairly small file size. And the last thing we should always ask is, OK, when do you need this done? And of course, we get a favorite answer. Please have it done as soon as possible. OK. So we start looking at this REST API. Before we start getting into the various approaches, we want to understand how this API works, to better understand this problem. So the API that we have, of course, there's an API to get a file. Just given a file name, you can download the contents of a file. There's also the ability to list files. So we need to know what files are available on the service. And the way that this API works is the way that a lot of APIs work that have a large result set. It's paginated. So instead of returning a billion files and says, here's all the files from this server, it instead will give you a fixed amount of files at a given time. So say, 1,000 files for a single page. And you'll have 1,000 file names and then a marker to get the next page, which is essentially, think of it as like a pointer to the next page of results. And so concretely, we'll have a JSON response, file names, which is a list of files, and then next marker, which is our pagination token. So then we can get the next page of results we can keep going serially through all of the files until we get no next marker, which means we've got all the files listed. OK, so that's the list API. If we look at the other one, as shown down below on that third column, or the third row there, there is a get slash file name that gives us the file contents. So this is the API we're working with. It's a little bit weird that there's like list and get in the URL. But that's what we have. And so I think now we have enough to understand the problem and start experimenting with a few different approaches here. Now before we get into that, there's a couple of caveats just to make sure that no one takes this or misreads like what I'm trying to show in this talk is that this is a simplified case study here. The results here don't necessarily generalize. So just because maybe async.io comes out ahead or multi-threading comes out ahead doesn't necessarily mean you should always be using that. It's really just showing a specific use case here, a specific case study that shows for this problem, with this client and this environment and this network, this is what ended up working for me. But I do think that sometimes having a concrete example is still really helpful. A lot of times, one of the things I struggle with is that you might understand how, say, multi-threading or multi-processing works, but then really understanding when you should use one over the other or what the trade-offs are is still a little bit challenging. And so hopefully by sharing some of this data here today, it can help people be more aware what some of those trade-offs are going to be. OK. So I will say, though, always profile, always test for yourself. There's really, you can't get around that. You still have to always do that. But let's get into the various approaches. So before we do any of this multi-processing or async.io stuff, maybe we can just use a synchronous version. It's the simplest thing that works. Maybe if that's good enough, then we don't have to do anything crazy. Wouldn't be that interesting of a talk. But we should at least make sure that it works and see what the timing looks like. And to be clear here by synchronous, what I mean is maybe we can just take a list, and then we can just go through each of these files here and then download them one at a time. So say, for each file on a page, download the file, download the file. And then once you're finished with that, get the next page and then go for all of those files. So the code for that is fairly straightforward. We're just using requests. And then we just have a for loop to go over all of these pages. And so the way that this code works is we have some constants at the top that match the API we saw in the previous slide. And then we have the first request that we make, which is a list request. So we get that first initial set of files. And then we just have a for loop here. So for each file in the file name, we're going to make a call to download file. And we'll look at that definition in a second. And then after that, we check, is there any more pages? So if there's this next marker, then we say, all right, requests.get this next URL, providing this next marker here in the query string. And we just loop over that until we download all our files. And so this download file here is also not that many lines of code here. We're just making a request.get call to this remote URL, checking to make sure that it's a 200 status code. And then we're going to open the local file name that we were passed in, and then write out the data to that file. So that's asynchronous results. If we look at how long it takes to do one request, it takes 0.003 seconds. So this is essentially the network latency for making a single request, amortized over the list call and the get call, and the overhead of processing it in Python and going through the request library. So three milliseconds. You think that's good, bad? Well, if we take three milliseconds and we multiply it by 1 billion, ends up being 3 million seconds, which is 833 hours or about 34.7 days. So I think if we were to say, OK, we have this version running, and we'll have this done in about a month. Check in in a month, and we'll have this done. That's probably not going to work, right? So we know that the synchronous version isn't good enough, and we need to find something that's going to be a little bit faster. OK. So now we can get to multithreading. This is the first thing we're going to look at. It's nice because it's been in Python for a long time. There's a lot of things that have been written about. It's been fairly well studied, so it's really easy to get started with multithreading code. What we're going to use here is a producer-consumer queue. It's a fairly standard pattern when you're using multithreaded code or even in multiprocessing, which we'll look at next. But what we're going to do is we're going to take one page of results, and then instead of downloading that, what we're going to do is we're going to put that in a queue. And then from that queue, we're going to have worker threads. They're going to be pulling off that queue and downloading those files concurrently. And then so in the main thread, we have this list object or this list files that's going to keep running, and it's just going to keep putting stuff on the queue, and you're just going to have these workers continually downloading these files. And then at the end, we're going to have those workers put their results in a result queue. That's so we can track status, we can track progress, look at errors, and that kind of thing. And one of the observations here is that we're trying to parallelize where we can. So if you notice with the list call, that's inherently serial, right? There's no way to run multiple list calls in parallel because in order to make the next list call, we have to have the next marker that was from that previous response. So we have to go in order. But once we have this thousand files, those can all be done in parallel. So we're trying to make sure that we have things that can be run in parallel or run in parallel, and then things that aren't are separated out. But we also have this nice decoupling as well. So we have the list files being completely independent of the downloading of the files. Now the code for this is, again, fairly straightforward. It's not too many lines of code here. It's similar to the beginning where we have our constant setup. But then we have the two queues we saw in the diagram. And then we set up our worker threads. So we have a number of threads here. And for each of those threads, we're going to start them up with this worker thread function. And we'll look at that definition shortly. And then we also start up a result thread that's going to be pulling on that result queue that's going to print out results. And so if we look at what the actual main loop here is, it's pretty similar to the synchronous version. So we go through each of our pages. This is that main thread doing that list call. And the only difference is that instead of calling the download file, it's now going to call this work queue dot put. And so the theory here is that it's quicker to put something on a queue in memory queue than it is to actually download that file in that thread. So now if we look at what the worker threads are doing, again, there's not a whole lot here. It's just pulling, waiting for something to come through on that work queue. And once it gets it, it's then going to call this download file, which is the same function that we saw in the synchronous version. It's just now running in a separate thread. Okay. So we run this with 10 threads. And how do we think we do? Better than the synchronous version? Not better? Well, it turns out it's actually slightly worse. So it's about 3.6 milliseconds, and which comes out to about 41.6 days. It's still not very good. And maybe we think, okay, that was with 10 threads. That's, things are happening concurrently. So, you know, let's just, let's just bump up the number of threads to 100, because you know that'll make things faster, right? So it turns out that actually makes things worse. It's about 4.2 milliseconds per request, coming out to about 48.6 days. So things aren't looking so good so far. Why is this happening? Well, we're not necessarily IO bound here. So the threading is really good when you have a lot of IO bound work. But in this case, due to the low latency of the network, so we're looking at single digit millisecond latency, and the small file size, there's not a whole lot of IO. It's very different writing 100 byte file versus say writing a two or three megabyte file. And so things like the GIL contention and the overhead of passing data from the main thread to the work queue, and then from the work queue, the worker threads running them, and passing that on to the result queue. There's just a lot of overhead. Each of those queues have locks and condition variables, and so there's just a lot of work going on. And because there's not a lot of IO, all of that overhead actually ends up making things slightly slower. Now there's a couple of other things to keep in mind when dealing with the multi-threaded approach. This is stuff that's based on my experience working with multi-threaded code and projects that the real code's actually much more complicated.  You can't really shut down threads explicitly. You have to request that they shut down and set some sort of invent or variable that they periodically check, and hopefully they see that and shut down. And then debugging is really hard as well. It's not deterministic. You typically don't use PDB. You'll have to use the PyCharm debugger or something that can handle multiple threads. And in my experience, the more that you deviate from the standard library and build stuff on top of it or not use things like Q.Q, the more likely you are to encounter race conditions. It just always seems to happen. The things that understand their library are fairly well tested and have a lot of people using them over the years. And so I usually try to stick to the stuff in the standard library when dealing with threads. Another note here, you might be wondering why not use concurrent.futures. Well, in this case, there is an executor.map, which will just give you, you give it an iterable and it gives you the results back and it runs them concurrently as well. In this case, it's creating a one billion element list. So if you try to create a one billion element list in Python, you will definitely crash and you will run out of memory. So we can't use that. But there is still a way that we can use concurrent.futures. And we'll look at that next when we look at processes, because hopefully processes are gonna, we're gonna have better luck with that. So if you remember from the details here, we have 16 cores to use. And the thing with threads is that we were only using a single core. So the thing is, maybe if we can use multiple cores, things might actually be a little bit better for us. So what we're gonna do here with multiprocessing is instead of using threads, we're gonna use processes and we're gonna try to use concurrent.futures.  So the way this is gonna work is we're gonna take one chunk at a time, so one page, and then we're gonna have all of these worker processes download them as quickly as possible. So 1,000 files at a time. Once that's done, we're then gonna move on to the next page and then have that queued up and then have those workers download 1,000 files at a time. And we're gonna keep going until we've got all the billing files. So the code for this, slightly different. So we have our constants at the top. We have our hostname, the list URL. Then we have this iterate all pages. So we're just starting to put some abstractions in place. We're starting to separate out the iteration from the actual logic and the orchestration of downloading files. And then we have this downloader class and I'll talk about that in a minute. But then the main logic here is in this process pool executor. So in this futures package in concurrent.futures, there's a process pool executor. It manages all of this for you. And by default, it'll create one worker for the number of cores. So in our case, we're gonna have 16 workers. And all we do here is it's a two-step process. So we're iterating through all of these pages and then we queue all of them up. So we start the parallel downloads. This is that first part in the previous slide where we queue them all up using this executor.submit and we get futures back. And then afterwards, we're gonna wait for them to complete. And so there's this futures.asCompleted. We're gonna get futures back as they finish. It can be in any order. And then we're gonna look at the results. Now the iterate all pages is sort of, it's kind of similar to before. The main difference is here instead of yielding one file at a time, we're iterating over files. Here, we're just iterating over pages. So we're hiding all of the iteration stuff from the main worker process. So it doesn't have to worry about that. And otherwise, it's the same thing. We make the list request and we keep looping while there's next markers and just keep giving more and more pages. So that's how the iter all pages works. And then the downloader is similar to what we saw in the synchronous version. So we're just using the request session.get and then we're opening the file and writing it out. So one of the things you have to be careful with with multi-processing is because it's passing data across multiple processes, you have to be able to pass whatever your task is, whether it's a string or an object, it has to be able to be serialized, sent to another process and then deserialized. And so it will use pickle for that. And so you have to also make sure some of your objects can handle that. So in this case, we're using request.session, which can be sent across process boundaries. Okay, so this is the multi-processing thing. We're using 16 processes. What do we think? Better than threading? Not better? Yeah, definitely better. So in this case, it's .00032 seconds. So that's an order of magnitude better. So before we were at three milliseconds, now we're at .3 milliseconds, which if we do the math comes out to about 88 hours or about 3.7 days. So that's definitely better than a month, but it's still not great. We're hoping maybe we could do this a little bit quicker. And some of the things that keep in mind with this multi-processing is that speed improvements are because we're truly running in parallel now, because we're leveraging all of these 16 cores, we're actually making requests in parallel. But just like multi-threading, debugging is much harder. PDB typically doesn't work out of the box. And there is overhead. So what we were doing with the multi-processing is sending each file name to a worker process to then download. So there's a lot of IPC overhead there to be able to say, for the main process, I want this worker process to download this file and I want this other worker process to download this other file. There's a lot of overhead involved in that. So maybe we can improve things. But we looked at, there's a trade-off between doing things entirely in parallel, which we saw with a multi-threaded approach, where we just had this completely decoupled list files from the SCET file. And then here where we're doing things in parallel but in chunks. And even though we're doing things not completely parallel, it still ended up being faster, we got to leverage a standard library. And it was less code for us, which is great. Okay, let's move on to async.io. Hopefully things get better, right? So with async.io, what we're gonna do is something slightly different. It's a little bit hard to visualize what's happening here because it's mostly the way that it's run in the event loop that's different. But at least conceptually what we're gonna do is we're gonna make our list call, like we've been doing before. And what we're gonna do is start a new async.io task for this. And so the lighter color here would indicate that the task is running but not done. So we're gonna spawn all of these tasks, move on to the next page and start queuing up more of these tasks. And they're actually running now. And so maybe as we're starting to create more of these tasks, maybe one finishes and so then we'll keep creating more tasks and some will finish and we'll just keep going on and these can be in somewhat arbitrary order these tasks will finish. So that's what we're doing with async.io. Some of the benefits here that are mostly in how this is run. So because we're creating a new task for each one of these requests, that's something that we really couldn't do with multi-threading, right? We don't necessarily wanna create a new thread for every file that we wanna download. But here the task object is much more lightweight and some of the benefits here with using async.io is that it's all in a single process, all in a single core. You switch tasks when you're waiting for IO or really anytime something yields or something's waiting for something. So in theory, this should keep the CPU busy, right? We should have much more efficient use of the core. Now the code for this is a little more involved compared to some of the other approaches. We're using some different libraries now. We're using async.io from the standard library and then two other libraries we're using here are AIO, HTTP, which is our substitute instead of using requests. And then we're also using UV loop. In my experience, UV loop is just a way to, like it's basically free speed ups. I haven't had any problems with it and it just always makes the event loop run faster so I always just use it. And what we're using or how we're gonna set this up here is we're gonna still have the same hostname in the list URLs here, but we're gonna have a sem afford to control how many tasks are running and then we're gonna have a task queue to make sure that we only have a certain amount of tasks that we run at any given time and then we have that results worker that's kind of similar to the results thread from the multi-threading approach. So one of the things that I found with async.io and really any of the approaches is that you can list objects or you can list these files much, much quicker than you can actually download the files. And so we have to be really careful that because there's a billion files, if we don't cap that, you will eventually run out of memory because you're queuing way too fast. So the way that we use this AIOH TTPs, we use async with, we create a session and then from that session, we're going to iterate over all the files and we'll look at this definition in a bit, but now we're going through each file and like we saw in the diagram here, we're just creating this remote, or we're creating this task that's gonna download the file. It's responding a new task for each of these files and then we're putting it on the task queue. Now the iterall files is similar to the iterall pages. The only difference, aside from using async and using AIOH TTP, is that we're yielding this file name. So again, we're just hiding all the details from having to do this pagination and we're just giving this file name each time. Okay, and then the download file is again fairly similar. We're just using async with session.get in order to download a file. We're using the semaphore here to control how many download tasks are going at a given time. So you can actually do this also with AIOH TTP. You can set a limit for how many connections you want to a host. In this case, using async with the semaphore lets us control that in our own code and it's just a little more explicit so it's easier to follow how many actual download file tasks are going at a given time. But one thing I wanna point out here, and this I'm not entirely sure about, is that there is an AIO files library that I initially used and that's to do all of the IO using an async, basically gives an async interface to the file IO. And I believe it does it behind the scenes using threads, but in my experience here with this specific problem it ended up being a lot slower, not a lot slower, but ended up being a little bit slower. And I guess I suspect it's for the same reason that the threads didn't have a whole lot of benefit. It's not really like a lot of IO. And so in this case, normally, writing to a file in an event loop is a really bad idea. You're stopping the whole event loop. But when I compare the numbers, this was actually faster, so this is what we went with. But again, I'm not entirely sure if there's a better way to do that type of IO. Okay, so the async IO results. Faster than multiprocessing, not faster. Well, turns out, faster than the threads and the synchronous, but not quite as fast as multiprocessing. So with that in mind, it comes out to about six and a half days if we were doing this, so which was a little unexpected. When I was doing this, I guess perhaps I was biased, but I was thinking that async IO would be the clear winner in this case. And so now we're looking at, here's the summary of the results that we went from the 34 or 40 days using synchronous code, using multi-threading code, all the way down to about 3.7 days with multiprocessing. So we would say, all right, out of all those three approaches, we should go with multiprocessing, right? That's the clear winner. That's a takeaway. But for me, I was thinking, that still doesn't seem like that's the best we can do. And normally I might stop here, but I was just curious and I wanted to see if we could do even better. And my thinking was that, okay, async IO is the best way to take advantage of a single core and a single thread, but multiprocessing is the best way to ensure that we take advantage of all our cores. Maybe there's a crazy way that we can combine all of them together to get even better performance. And it turns out you also have to use threading as well, which I'll explain in a bit, but like, so we're throwing everything at this. We're gonna see how fast we can get this. So I won't be able to show the code for this due to the fact that it's really complicated and we're running a little bit short on time, but I'll work through how this actually works. So the way that this works is that we have worker process, right? Within the worker process, there are two threads. And with one of the threads, it's running an event loop. It's crazy, but stay with me. Okay, so what we're gonna do is we're gonna have 16 of these processes going, 16 event loops, right? And the way that this is gonna work is that we're gonna try to have each core being maximized as best as it possibly can. And we need to have a thread in each process because we need something to interface with multi-processing cues, which is using like pickle and it's using sockets or writing to pipes and it isn't async aware. And then we need something to bridge that into the async land and back and forth. So here's sort of how the workflow goes. This was just something that I was playing around with and I'll try to walk through this. It's a little bit involved. So what we're gonna do is we're gonna take a pagination token. So this is just some opaque thing like foo or whatever. Take that from the main process. We're gonna send that over to the worker process through a multi-processing cue, right? That is then gonna send it into the event loop for the async.io.q. And so this main thread is then gonna do the pagination. So it's similar to what we saw in the async approach, right? So it's gonna get a thousand files and it's gonna get a next marker. Let's say that it's bar. That's the next page of results. Now, before it starts downloading, what it's gonna do is take that bar token, immediately send it back to the thread and then that thread is eventually then gonna make its, or that value is then eventually gonna make its way back up to the main process. Skipping a few steps here, it actually goes to the output cue, which then gets fed back into the input cue. That then goes to another process, right? That then does the pagination. And so while that's going, you have this worker process that then is gonna download its thousand files, while another process is doing its own list files call and then gonna eventually download all its files in the event loop. So the reason I like this approach is that, number one, you get to leverage all the cores. And we clearly saw that when we do that, we get a nice performance boost. Okay. Number two, we download all the files efficiently, right? So we saw that async.io is the best way to take advantage of a single core and a single thread. So that's great. And then the third thing is that we've minimized the inter-process communication overhead, right? And that's because instead, before where we were sending a file for every time we wanted a process to download, now we're just sending a pagination token, which is a thousand times less, right? So there's one, each page has a thousand files. So we're only, we essentially cut the IPC overhead by a thousand. So this is a really nice way to get, sort of take advantage of all the things we've seen so far throughout this talk. Okay, combo results, better, worse. Well, it turns out it is actually awesome. It's, what is this, a 0.03 milliseconds, about 30 microseconds per request, which comes out to about 30,300 seconds, or 8.42 hours, which is cool. So the real summary is that by combining all of these, we went from something that was about a month, all the way down to something that was about eight hours or about a third of a day, using asyncio multiprocessing and all that stuff. So what I actually recommend that, I mean, probably not, it's really involved to write. And so some of the lessons learned here is that it's multiple orders of magnitude based on a different approach, but it is really a trade-off between simplicity and speed and depending on, maybe the multiprocessing is enough and four days is enough for you, or maybe you really need to get this done in a day. And so another lesson, last one is just, you should have max bounds because one of the things I ran into all the time was getting memory errors because I wasn't properly bounding things correctly. But yeah, I hope you got something out of this. And I hope that, you know, this gives you more context about multi-threading, multiprocessing and asyncio. That's all I have. Thanks, everyone. Whoa, thank you, James, for getting us so many files and actually less than one day when you took more than one month in the first version. Thank you a lot. We are a bit short for a question. I get a hang out outside if anyone has questions. So yeah, we can go outside. All right. And we have, so thank you, thank you a lot. Thank you a lot.