Hello and welcome my map, and if you've ever changed your mind on what army you're building halfway through building a different army Just go ahead and like the video If you're new here, this is why I quickly unbox and show you everything you're gonna get in various products If you're new here, this is just the no fuss unboxing of what you can expect to find in the new Terrain crate line Today, I'll be unboxing one of the amazing new sets from Terrain crate the abandoned factory To give you a speedy rundown if you're new Terrain crate are a line of incredibly detailed pre-assembled scenery so no glue required All of our terrain is colored plastic so you can either crack on or paint to your heart's content Terrain crate features a huge range so no matter the scenario or game you're playing you'll be able to find something to suit you and Lastly, these are designed to outlast and resist even your most sourced of losers now enough of me Let's crack it open So the abandoned factory is one of my favorite sets just because it's got a good mix of like cover but also some big items that you can use as Objectives or main centerpieces in your in your gaming board. So once again, we'll start at the top. We have a Loading door sort of factory door there. So that can be put against the wall as your entry point or your exit we have a Furnace or incinerator or smelter or anything of the sort a big engine block or machinery of some sort Over here, we have a gun or tool cabinet and also the doors can be removed So if you're wanting that to be locked up and have your players try and Break their way in or lockpocket or something like that. That's an option you can do Over here, we have a trolley or girder and mechanics trolley two pallets and a wooden box or crate to go on top of that And over here, we have just a collection of boxes Some more just like junk To create a piece of cover We have a ventilation shaft or floor grate so that can be used for walkers to get into your into your map a Pipe and pallet just to create another solid piece of cover just to break up line of sight We have a workbench to go up against a wall a little medic bag or medical kit to go with the girder and a barrel and Collection of boxes with some tools on top So some really hefty big pieces of terrain and then some of those ones that you're always going to try and grab So your your pieces of cover your big ones to break up entire profiles and nice objective ones like the gun locker There's an absolute great mix of stuff just to fill your board and make it really interesting If you found this quick unboxing super helpful, make sure to hit the like button It helps people find the way to the channel and if you can't get enough of terrain or wargaming in general Make sure you subscribe to keep up to date on all things mantic games. Thanks, and I'll see you next time You


In this study, ground glass powder and crushed waste glass were used to replace coarse and fine aggregates. The results showed that the use of waste glass powder provided better bonding with cement due to its pozzolanic effect, resulting in increased compressive strength. However, the presence of large glass pieces resulted in voids being formed in the samples, which had a negative impact on the strength. Simple equations were developed based on existing data in the literature and the results obtained from the current study to predict mechanical properties of the concrete with recycled glass for practical applications. This article was authored by Ali Aizen Selik, Yasin Aniralpa's Kilik, Azair Zeebek and others.


The Coalition of Kano Support Groups for Vice President Oshimbajo has shown their excitement following his decision to run for the presidency in 2023. The support has hit the major roads of Kano singing and dancing. The group expressed confidence in Oshimbajo, noting that he was the right candidate to lead the nation. Today marks a greater milestone in the fortunes and prosperity of our nation, Nigeria. We the National Coalition Group, NCG, having placed a cry on call on Professor Oshimbajo to contest for the converted seat of the presidency of the Federal Republic of Nigeria. We wish to use this opportunity to say thank you, Mr. Vice President, for answering the cry on call to run for the presidency. It is convinced in your cerebral embodiments as the Senior Advocate of Nigeria, as the Professor of Law of Evidence, with years of experience, with seven years of experience as Vice President. Nigerians are tired of experimentation in the presidency of this country. And we are suing for continuity, experience, courage, and winning qualities. Our Progressive Congress APC, Awadia Party, has the handsome opportunity and best choice before it, as President Oshimbajo will form a major standard bearer in the 2023 presidential election.


The study of yttrium ion garnets magnon spectra at finite temperature using a tide-binding model with nearest neighbor exchange interaction reveals different spin reduction due to thermal excitation at two sets of ion atoms, resulting in good agreement with experiment and only two relevant magnon modes for the ferromagnetic resonance. This article was authored by Kation.


Alright, thanks for staying with us now. The increasing number of attacks on farmers in Nigeria is leading to people being forced to leave their homes, disruptions to market and people losing their livelihoods. Now, according to Nigeria's security tracker, armed groups killed more than 128 farmers and kidnapped 37 others across Nigeria between January and June 2023. In June alone, 19 farmers were killed by non-state armed groups in Borno State in Nigeria, Northern Nigeria. Now, the food insecurity in Nigeria has been severely affected or the food availability has been severely affected by violent conflicts in recent years, including the Boko Haram insurgency in the northeast, armed banditry in the northwest, where there have been several attacks and raiding of villages, stealing livestock and food and killing and displacing farmers. There are farmer heather clashes in the northwest and other parts of the country and separatist agitation in the southeast. Now, these conflicts have made it difficult for farmers to produce food and for people to access food. As a result, millions of Nigerians are facing food insecurity and this is a huge problem, especially with the current economic situation in Nigeria. So tonight we're asking how can we curb food insecurity? How can we make food to become more available in Nigeria? Please let's hear what you have to say. Remember to join the conversation, send us an SMS or WhatsApp to<PHONE_NUMBER>3. Why so ladies? I mean, this is an interesting conversation. First, because we are farmers. I kind of like have an idea what is going on. Do you understand when it comes to the farming area of like growing of the crop? So if you're not dealing with locals, right, you're dealing with what's it called, herders coming into your farm. If you're not dealing with that, you're dealing with, would I call it zero or little access to your farmland based on, you know, bad, very, very bad roads that you have to them now deal with what's it called construction of that road. You're not dealing with that. You're dealing with breaking down the constant breaking down of your equipment. So every day you're finding yourself going again to your technical people to help repair either your tractors or even your machines and whatever. If that is not the case, you're having issues with workers, right? You have people that are not willing to work. You have, you now have to import workers from maybe nearby Benna Republic and all of those things. Like literally, right? If you want to look at farming, like in terms of like growing the crop, there is a plethora of challenges that you're not talking about seedlings that are so expensive, right? Then by the time you're done with the seedlings, you now have to buy very expensive fertilizers, very expensive insecticide to keep the pests out of those crops so that they can grow well and you can get good yields. Then when you're done with that, harvesting it, you know, you are not dealing with, oh, the crop is there. Nobody's buying it. Then that is like a major loss and all of that. So like, they are just a myriad of problems, right? So I don't know how if we say we want to curb food insecurity in Nigeria, you know. First of all, why are we having insecurity in Nigeria? I've just said this now from the farmer's perspective. Of course, what we're talking about, when I was giving the intro, there's the insecurity part, which is very major.  And, you know, there's also the problem of our, the inflation and the devaluation of our Naira and all of that that is also impacting because the seedling I would have bought for, say, 20,000 Naira. I'm having to buy that same seedling, that same quantity, for almost like 100,000 Naira. Does it make any sense? And it's the same yield. It's not like the yield is going to be any different. So let me hear your thoughts. Okay, so sorry, I'm sorry. So for me, first of all, food is life. Food is, I like this mantra that says that leave no one behind when it comes to food. I think the world is too blessed with resources and especially a country like Nigeria that hunger really should not be a major problem.  And unfortunately, most of the food that we eat and even exports are from the northern part of Nigeria.  Again, I would also want to speak about the investments we put in agriculture in Nigeria for a country with a kind of population that we have. I think we need to invest more in agriculture. Agriculture is not just about farming. It's the whole value chain. We need to critically look at the value chain from financing to education to seedlings to harvesting to fair trade practices. Because that's very, very important. If you go to Benway State, it's crazy because I was talking to someone from Benway State. Benway State is supposed to be like one of the richest states in this country because they're the food baskets of the nation. But Benway State is unfortunately one of the poorest in this country. Now, why is Benway State that poor? It is because, I mean, they grow the food, they plant the food, but they have zero knowledge about the practices of how to, that is a business side to agriculture. And we need, we can't leave it to people who are not very educated if we want to grow as a nation. We need to bring in a lot of, a lot of expertise. You know, it can just be for the selected few. We can't, we can't say agriculture needs to be as popular as entertainment. As tech. Exactly. As banking and finance. Exactly. It needs to be that. It has to have more corporate players. Exactly. Exactly. Exactly. So if we find the farmers who are doing all the work, who are not very well versed in modern ways to increase yield, to have a better harvest, you know, to do more, to diversify, then food will always, we would always have food insecurity in the country.  Because again, they will now be at the mercy of terrorism, of not enough workforce to work in that sector. They would be at the mercy of, I mean, economic inflation and all that. And again, it will come back to the consumers, you and I. Because again, I mean, the market forces would also determine the price of food. If food is not available, it is only going to be for people who can regularly afford it. And that means that people are at the very low end. And of course, when there is no food, there's poverty. When there's no poverty, there's violence. And it goes on and on, you know, so. I'm hearing your thoughts. Well, Bill Gates made a prediction in what is he called in a couple of years, maybe 10 years or 15 years from now, there's going to be like food scarcity. And so I was checking globally, what is the major cause of food insecurity? And one of the top three on the list is climate change and global warming, scarcity of land for farming, and then inadequate supply of water for irrigation. There are other things, poverty, technological barriers, but the point I'm getting to is, among these top lists, climate change, scarcity of land, inadequate supply of water for irrigation. Nigeria doesn't have an issue with that. Nigeria is so blessed with a predictable weather such that, you know, within this month, so at least within March till September, we have rainfall. Within November to maybe March, there is like dry season, right? And so I would say that from my perspective, or from what I've studied and observed, that one of the reasons for the looming of persistent food insecurity in Nigeria is the fact that we don't have a good value system or appreciation system, right? And here is what I'm saying here. Like you rightly said, food is so important, like more important than tech, even though that tech helps us to like maximize food production, right? But food is so important that we don't we don't reward people who put in that effort to provide because if people, if we Nigerians actually invested way more value in food and agriculture than we did in politics or entertainment, then we would know that if there is insecurity, or if there is terrorist attacks or kidnappings going on in those major places that we know that our food is coming from, we should prioritize it. If in Benua, they're having flooding, they're having bad roads, or they're having a low cost economic crisis, we should know. Okay, food, no matter what is going on, food and shelter, food clothing, shelter, those are like the top three basic human needs, right? And I think topping the chart is food because it can wear one cloak for like 50 days, but you can't not eat for 10 days. That's literally killing yourself, right? And so if we would learn, we as individuals and the government as well would learn to prioritize and value where this food is coming from and say, okay, on our list of the budget for this year, what are the major food producers in this country? What are their problems? Let's fix it. And then by the time you fix it, like there are other states in Nigeria, like there are some eastern states that they don't prioritize farming, they just do subsistence farming just for me, myself, and I, but by the time they look and see, oh, there's development coming in, like these people have been prioritized, then all that massive land that they have that they're waiting to build 50 acre of their father's house, so they can have events in it. They can say, okay, no, seriously, have you seen both people build houses? I saw one house when I went to Mama's village. It is the whole of Italy, right? From the beginning to the end. I said, one human being. You don't need all that. You just leave one house, a compound to have your events, and then the remaining one, you can turn it into a farmer. If you don't want to be the farmer, you can lease it out, right? So when people see that there is actually a reward coming from agriculture, then it would encourage people, and it would even encourage foreign investors. So I like what you said, Sansi. So let me try to attempt, because I know that, you see, there has been a lot of serious talk about, what's it called, a Greek, and I know that setting state governments are actually taking it seriously. But you see, our major problem, first of all, we have to deal with the mind part of this issue, which is, even when I was watching Idris Elba's gold new documentary that he has on YouTube, a fantastic documentary by the way, I recommend for everyone to watch it. How they urge South Africans to drop their hose and their whatever, and go and start doing mining. It's the same thing in Nigeria. We were known for our granite pyramids. We were known for agriculture. So cutting, we were known for agriculture, but we took up all of those things, dumped them on the site for oil. So if we're told that we need to solve this problem, we need to now do reverse engineering. Go back to the basics. But you see, for us people to do that, we must now find it. I like the what you said about being attractive, right? We must find the attractiveness in it to be able to now encourage more people to come into that sector. And that's why when Joella was talking about corporate engagement, we need those kinds of engagement. We can't have just petty people just saying they are farmers. No, people that have like real time business sense, you know, come into the sector. Because now, like you rightly said, those top three problems, they are not our problem. Can we change it literally because our weather is changing, right? Things are changing. The time you're expecting rain, you do not see rain, it's coming through. But at least it's still a bit manageable compared to parts of the world that the issues with flood. Lagos State Government did something at some point where they partnered with Kirby State Government. That was where the Lake Rice came. So Lagos and Kirby made Lake, LA and KE made Lake Rice. And for me, I felt that that was a beautiful partnership. Because guess what? They have to farm that way to produce en masse. Lagos has the capacity to commercialize it. So imagine if we start to have those kinds of interstate partnership. You see, it is this problem of I go every month, I walk home and I don't work at a federal location. You will never think outside of the box. So government, state governments, even with their arable lands that they have, they are not thinking agriculture. Even for people that have farm lands, like for us now, we have hectares of land, right? To get Sea of O. Jealous and you, as a farmer, if you are farming and it is confirmed that you're farming, your Sea of O is supposed to be free. You know what I say so? If they give you free Sea of O, then they are serious about you taking that to your farm business to the next level. But that Sea of O might be tied to a condition that you must mention that you maintain farming. It is only with a Sea of O and all of those things that you can now use those properties as collateral to be able to access financing. For us to be able to expand the land and expand the produce and expand all of those production, you don't need to have external funding. You cannot do small, small funding. So there is a lot that goes into food security, right? And right now in our country, it's just like, let's picture it now. We are in the days of when Joseph was inside DJ, when they told him that farming is coming. That's what Bill Gates wanted.  So what are we doing as a government? We're not even trying. Now I'm hearing, you are telling us now that he's mopping up seedlings and all of those things, right? All of these things are happening for people that are thinking, for people that are looking at it and seeing the future, that if we do not position ourselves, do you understand? But our government was still sleeping. Look at Lagos Ibardo Expressway. Look at Ibardo. Let me tell you, when I went to Amsterdam for a conference, I then decided to take a bus from Amsterdam to Paris before now taking a train to south of France. Do you know that between Amsterdam and Paris, San Si, it's just land upon like farm lands, stretch. See gigantic windmills.  Gigantic windmills. You will see the one that are doing the crop farming, the one that is on all of that there. This is our lake. You, they talk about kidnappers. Is it not inside those bushes that kidnappers are hiding? If we clear up all of those lands and people actually start to, you know, farm in all our forests that we have, we will not be. So when the world is complaining, we will be selling food to the world. Sorry to get you, but on the other hand, I think I would like to, like the same example you gave about, what was the place you went to? Amsterdam. Amsterdam. Yes. I would say that when I travel, which you are not, you've been, you lived in the north. When I'm driving through the north, maybe like from Kaduna to Zare, they are farm lands. Farm lands. I know it's not enough, but I have to commend the northerners. You at least get to something. You'll see like this massive lands and you see, or whether you're taking the train or whatever, you'll see they are putting that land to good use. You see people cultivating their lands and just hectares of cornfields, right? But I wish Lagos would imitate that. Because in the South West, South West, South South, there's a lot of vacant lands and bushes. We don't need well conserved nature. Even if you are farming, the tree is still nature conservation. So we need to like maximize those lands. Let's take a break, right? When we come back from that break, we'll continue the conversation and take your comments. Stay with us over right back. All right. Thanks for staying with us now. If you just tuned in, we're having a conversation around food insecurity. We're asking how do we curb it? Now, please let's hear what you have to say. Remember, you can join the conversation, send us an SMS or WhatsApp to the 1-803-4663. Okay. So, Dee, you wanted to say something? Yes. So, I like, I mean, the thoughts around the Northern people, you know, doing the best that they can. But you see, that's one step. So, for it to be successful and sustainable, then we must, as not just individuals, corporate organizations, business-oriented people, and then the government, we must look to the other value chain, the other, you know, parts of it. So, again, you talked about land. I would expect that we need to start looking at land reforms, you understand, you know, even in terms of 10-year land, land tenure reforms. But when the farmer farms, they've invested so much in farming, they know they're going to harvest. Where does it go? How does the farmer transport? So, we need to look at infrastructure. Infrastructure plays a huge role. You can't put food, perishable items, to move from the North to the South, three days. I mean, you talk about insurance, you talk about a whole lot. So, the farmer bears the loss.  So, they don't have the power to even say, okay, you know what, for my next planting cycle, I have the funds to do this, unless people step in. So, I mean, why are we not looking at train systems? Why are we not looking at infrastructure in such a way that we're able to distribute food across the nation? And no, we is, exactly, because by then, the farmer knows I'm not concerned about selling my produce. Let me worry myself about planting. That's the one. See, let me even tell you something here, right, just to add to what you're saying. If the government even took it upon themselves, you know, so you can, you can decide. And I think at some point, Yamio Shibajon, they were talking about it. I'm not just 100 percent sure, but they were talking about building those storage facilities for farmers to be able to, at least if they are done farming, they can, you know, you are sure of the things being preserved and all of that, because the wastage, have you been to K2 markets? I saw it as a high level of, when they were bringing in caravans, caravans, as you see, the wastage. Well, that's even just recent. Like, yes, I saw it, wasn't two weeks ago, it's even last week. We're not even touching the wastage that happened. Then you touched on something very critical, which was the fact that the fair trade, the fair trade is very important that somebody has paid their dues, they don't do hard work, they've watched the plants grow. Then you now come and you are buying those farm produce or peanuts. It is not a known, there should be a standard. Yeah, the priceability. So, no, part of what I learned when I did that online course for emerging markets, right, with Harvard, they touched on the community in India. That community, they were known for milk production, right? The milk, they were, they milked their cows and guess what? In the end, it's only maybe like two buckets. So the two buckets, but the time they will travel, by the time they get to who will buy it and all of that, it has, it's sour. So this man that was just thrown there, he said, what will he do? He now looked and studied. So there were too many, too many things he considered. This distance to go and be, he said, you want to go and be distributing your make, he said, no, no. That one, okay, already they had to put this, a pasteurization center or whatever it is to be able to collect those collection centers from them, from the farmers very close to the farmers. Then they were paying them a standardized wage. Guess what now happened? The farmers were now able to buy better feeds. So the cow that would produce normally maybe two buckets. The cows could produce four, some cows could produce six in a day. So that was more money for the farmers. The farmers were happy. They were able to feed the cow's fat and they were also supplying them. Now look at it, many years down the line, that company is still standing in India. They're not only supplying milk in the whole of India, they're supplying to other parts of the world. I just don't want to mention the brand. Now even the other value chain by products of milk, like the butters and all of that, they have a long list of items. But guess what, their farmers are extremely happy. You see the farmers, the ones that could not even, they bought motorcycles, some of them have bought, the farmers are happy. They're able to educate their children. They even provide them healthcare, schools. It's an entire value chain. So I like the idea that you're thinking holistically. But then for me right now, training might be a stretch because our governments, I don't see them understanding that this rail system is a big deal and they must really prioritize it. Even if they don't solve the road problem, right, without even the trade, D, look at it now. If you normally would take, I mean we used to drive from Kerala to Benin, route three, five hours you're in Benin. But now it would take you forever to get to to where you're going. Now they are kidnapping on the road. So there are so many layers of why this is a big problem. And as long as it continues in length, as our population continues to increase, I mean, they're saying that by 2030, our population will be the second largest population in the world. I just, they say, make whenever is day. It's just so sad because we have the workforce. We have everything. We have everything. And I, the sad thing is we're very, we were reactionary rather than being proactive. Nigeria can comfortably, comfortably feed Africa. What is Africa? I mean, comfortably. Nigeria has like massive. What is Africa? You see, for me, one of the things that I always say, and I'll go back to, I'll go back to politics for a little bit. I like the fact that you always talk about governance. And I think that, let me just say this, for the next election cycle, it is important that we are looking for people that are not just coming to talk about manifestos. If you want to be the governor, let's stop this focus on presidency. Start with your states.  First of all, to increase general revenue from your state. Because imagine if all Northern, I don't even know why we're not doing major agriculture in the South here, because there is land. Have you ever gone to Ogun states? Have you seen the amount of land in Ogun states? He shows there. He shows there. You look at whole states. And people are saying they're so poor in this state, but it's dependent on federal government allocation. I can't understand it. Well, this is a question I have. Is there like, I'm not aware of it, and I didn't research it before the show. Is there like a land loan system at work in Nigeria? Yeah. You can't do this land. My friend Barbara, she has an a father farm. So she had started off first of all with plantain and plantains did not really do so well. Okay. And she then moved on to, every time she sees us, she says, why are we buying land? Why are we buying land? You know, but she is actually operating on the lease structure. So they lease the land, and it is, you know, after the harvest and everything, the shares, there's a formula for it. But those options are very important. That's what I'm saying. That's what I'm saying to you that for even the people that have taken, that they painstakingly bought those lands, why don't you give them CO4s? Because you say the CO4s is not because we just want to own the land. It is possible to have the capacity. Yes, it's the capacity to be able to hand it over to BOI, to say, you know what, these are my needs. We want to be able to mount a, let's say maybe we could produce like for us now, we are palm producers. We farm the palm, we also process it all the way to the bottle. Imagine if we could do maybe like a 500 litre or whatever, 1000 litre per day. If we increase their capacity based on the equipment and all of that, we can move that production to like 10,000 litres, right? Sure. We're hearing that they are importing palm oil. It's an insult. Sure. The kind of things that you hear that they import to Nigeria, it's an insult because it's not like we do not have the capacity to be able to do it. So if we really want to get this food and secure it, right, we must first of all understand that certain policies when it comes to our Greek, right, must be scrapped.  So that you give some level of leverage. I heard, I mean, I'm not like I heard. I know the government gave out seedlings at some point. They were giving out fertilizers at some point.  I watched the documentary on cloths TV. It was the documentary. I think somebody had done that documentary. I stumbled on it and I was watching it and the lady I think she's a fish farmer. And she was saying that yes, they came, they collected their data, but since then they didn't hear anything from them. So do you understand what I'm saying? When the corruption and like literally if you want to do your corruption, carry on with another sector. That's what I'm saying because you don't understand the looming danger that we have.  So she said they came, they took their numbers and this is what they do.  But in turn they get that data, the reason justify, okay, we have 10,000 farmers here in this community, 100,000 farmers in this community, collect, you collect allocations to be able to hand over to those farmers. But it doesn't get to them, you know, in the end it is yourself who are suffering because if this food scarcity comes, because it will come, it's not a function of it, it is when it comes, it will hit everybody.  So that's another thing we need to work on, storage facilities. It's the same thing. So there's a lot that can happen, so even your tomatoes, it doesn't have to go bad. We can convert them, we can add, so this is where technology now comes in, you understand? We can convert all of those things, we don't have to waste them. The amount of waste stages is like literally, when you see food wastage in this country, you almost want to cry because we are talking about the fact that there's no enough food. The homeless see the ones that go to waste. And it's also causing a price hike because if I produce like 100 cucumbers or 100 carrots, and by the time they transport it from the farm to the market place, I realize that 45 is spot. Because I have to cover my costs. So the remaining 55 days, if I was meant to sell it for 10, 10 there, I would end up selling it for 14 there or something. Because that's 40 days, it makes sense. Food security is as important as the security of the nation because it's at the core of the nation.  See that they all die from Israel. Because they find food, they can't go eat it. But because at that time, they were in favor with government, they can't carry their food. Years later, when government, we don't know them. Because literally, people will migrate to where there is food. So if we do not start to pay attention to this looming danger that will happen, if we don't pay attention to it, what eventually it will mean, it will be that we will go cap in hand begging other nations of the world for food. You don't want to be in that position. And any country that is feeding you. Let's take comments. Do you think that part of the issue is because Nigerian youths are not as interested in... Because they have not made it attractive. It has to be attractive. The narrative now is probably for people who are uneducated people. So the very few people that are in that sector as educated people, young people, they are not... Agriculture is a game of, it's an industry of patience. You need to be patient. But I mean, if we keep selling the idea that... Exactly. So they are looking to us, where in the blink of an eye, let me make my turn. And it means that it must be incentivized to actually get people... And so you must also find patient capital. Yeah.  Very minimal. Very minimal. So that's what I'm saying. There are so many things. And this is not actually going for too much. There are so many things that the government can put today as policies that you get better players in the industry.  Incentivize it like Joella said. Remove all those bottlenecks, remove all those bureaucracies.  Yeah. Okay. Let me quickly read this. No country can function optimally without proper organization and structures. Carbine-feeding security in Nigeria can work if a government follow each state or region according to what it can naturally do. Everywhere in the USA, it is not a fan place. The government always encourage each state according to what it is naturally endowed with. In the US, the state of Wisconsin is highest producer of cheese and alcohol. And it is known for that. California area is very good in producing wine and other food crops. And it is known for that. In Nigeria, being made as the food basket of the nation, should be encouraged by the government to produce enough food to feed the nation. This is from Sanctus. Thank you Sanctus. It is known for yam. Yeah, they have really good yam. The best yam.  I say there's nothing you are looking for in Nigeria. There's no kind of crop you're looking for that we cannot grow. We have no business having this food is not in season. Even in jobs, strawberries. If you see the kinds of strawberries that come, it's very juicy and sweet. So if they pay attention long enough, even the apples, they'll be able to grow it in just because we just ask the weather for it. The strawberries in jobs are so big and cheap. I was like, what? This is not like that. I shopped right for like $1,000. I'm like, no way. We are in 14. Oh, is it the potatoes? I beg. These are not the right.  We are just begging, please. This food matter, it will concern all of us also. So if there's any way that we can start to think, let's start to, you know, put heads together. And this is real, real problem solving. Not to be seen to be solving the problem. We need to solve the cultural problem that we have in this country. Ah, multi talk finish. Thank you, Sanzi. Thank you dancing to you. Now before we go, please ensure you follow us across all social media platforms that wish you Africa, you can interact with us further, drop a comment. And more importantly, follow all our engagements, like and share, invite your families and friends to watch and follow. If you missed our quotes from Bill Gates, he says, innovations that are guided by small holder farmers adapted to local circumstances and sustainable for the economy and environment will be necessary to ensure, according to Buhari, that food security is the future. We get done, they talk about a Greek. He started with mosquito. He started with mosquito, went to health. No, he started with computer. Went to mosquito, went to health, nice in a Greek. Okay, follow the money. We'll see you guys tomorrow. Get here.


Hello. Hi, welcome to the presentation on Netflix and FreeBSD. My name is Jonathan Looney. I am the manager of a software development group at Netflix. My group maintains and develops the operating system that runs on the content caches that serve Netflix's streaming video. And I'm going to talk to you a little bit about how we develop that software, and specifically how we use some open source software, and particularly FreeBSD, to deliver streaming video to happy consumers around the world. So to start with, I need to tell you about OpenConnect. OpenConnect is Netflix's CDN. OpenConnect is a global, efficient, and purpose-built CDN for distributing Netflix's content. It is global. It is large. It is very fun to work on the OpenConnect network. At our peak, we deliver over 100 terabits per second around the world. And that makes the engineering challenges very interesting. So 1% differences make a big deal. They're a big deal when you're dealing with these large scales. As I was preparing for the talk, and I was thinking about this, I was realizing that if we had a 1% change in the number of TCP segments, we retransmit. So for instance, if we were retransmitting 1% more of TCP segments on our network, that's an extra terabit per second at peak that we're sending out to the internet. And that's a lot of traffic. And that's a big change. And so we work in an environment where small changes are a big deal. And we also value small improvements.  And so it's a really fun environment to work and do engineering work. The heart of our CDN is our content caches. You could almost say that the backbone of the Open Connect network, they are what delivers more than 100 terabits per second at peak. You see here a picture of one of our storage appliances from a few years ago. It's the 40 gigabit per second attached storage appliance. And it has 248 terabytes of storage and a 2RU form factor. We also have other varieties of caches. Our highest end production, one at the moment, delivers 100 gigabits per second attached. And it can deliver essentially full line rate in a 1RU form factor. The OCA runs almost exclusively open source software. And two of the more important things we run are the operating system is based on FreeBSD. It's a lightly modified version of that. And our web server is a slightly modified version of Nginx, or a slightly customized version of Nginx. This is our typical traffic profile. You'll see that we have lots of Netflix clients that are pulling data from our CDN. We have a group that manages the steering for that, so we can steer the Netflix clients to go to the right caches to serve their content. There are redundancies built into the system. And controlling the clients is very helpful, because it gives us the chance to tell them exactly how they should behave in various failure scenarios. So we can build redundancies into the system, such that if a single OCA fails, a single one of our content caches fails, the clients can seamlessly switch over to other content caches. And that means that if we are testing new software and that new software has some sort of fatal error, our clients can seamlessly switch over to something running more stable software and continue to be served, hopefully without even noticing any difference in the quality of their video. We also get a lot of data back from these clients. And so again, controlling the clients is very helpful, because the clients can tell us, or controlling these client applications is very helpful, because the client applications can tell us exactly what they think about the quality of the cache that they're talking to. They can tell us how fast they were able to get the video, whether they got it fast enough to satisfy the streaming needs of the client, whether it was very choppy or smooth, things like that. So we can get a lot of good information from our video applications. And that helps us to make informed decisions about whether we've made things better or worse in the software that we're developing. Our typical workload on one of our content caches looks like a typical web server. You've got a bunch of storage. You need to get data off the storage into memory. We then encrypt it. We often encrypt it for transmission over TLS session. And then we send it out. We send it out some NICs. Now this is not rocket science in a sense. This is something that many people have done before. And there are even people who have probably other people who have done it at the high speeds we are. What the secret sauce, if you will, or makes this special, is that we are trying to get high speeds, high efficiency, or high speeds out of our systems with high efficiency. So we want to not only deliver a great experience to our end users, we also want to control the costs. And so we actually work very hard to optimize our workload so that it can fit into relatively inexpensive parts that you could go buy off the street. There's almost nothing that we use that you couldn't go buy from someone if you wanted to. We don't have custom ASICs and things like that floating around. And so the secret sauce, if you will, is simply being able to get things through the system as fast as possible, as cheaply as possible. And you'll see that there's a couple of resource constraints we have. The first one is the disk I.O. and the network I.O. All has to go over PCIe. And so we're constrained to the number of PCI lanes we have. We're also constrained in the bandwidth those PCI lanes have. And then all the data coming in and out of memory. Again, we're constrained by the number of memory controllers we have, the number of memory channels, as well as the speed of the memory. So those are the two main resource constraints that we fight against. And you can either buy your way out of that by buying bigger CPUs, or you can try to make things more efficient by utilizing the resources you have more efficiently, more intelligently. And we try to focus on the latter. So using FreeBSD and Commodity Parts, we get 90 gigabits per second steady state out of our latest 100 gig storage appliance. Actually, I think all of our 100 gig storage appliances at this point. The latest one, we're using 55% of the CPU on a 16 core 2.6 gigahertz CPU. For those that are interested, it's an Intel Xeon 6122 CPU. There's 16 terabytes of NVMe in this appliance and 96 gigabytes of RAM in a 1RU form factor. The reason I'm quoting 90 gigabits per second number is because that's what we shoot for in production. We aim to use 90% of the attachments. The reason we aim for 90% is we do have a steering system that tries the appliance to the correct caches. And if you target 90%, you can do that pretty reliably. If you go above that, it gets a lot harder to do the control loop correctly. On top of that, we need some headroom in case there are failures in the system. So if a content cache fails, the clients need someplace else to go to. And so we need a little bit of spare capacity in the system. And the 10% gives us just a little bit to a little bit of headroom to absorb clients that are switching away. So we would consider 90 gigabits per second to be our target for this platform. And we can do that with roughly 55% CPU usage. And here's some graphs that show that. You'll see that the bandwidth is hovering right around 90 gigabits per second. And the CPU utilization is right around 55. And this is measured in an actual system serving production traffic. This is not a synthetic benchmark. This is what we get in production. So let's talk about the operating system that runs on these caches. It's a, as I said, it's a lightly modified version of FreeBSD. For those of you that are not familiar with FreeBSD, here's the 32nd introduction. AT&T had a UNIX distribution. They, the University of California, Berkeley got a license to use that distribution. They created what they called the Berkeley software distribution. And there's a lot that I'm skipping over. But essentially FreeBSD and NetBSD are both descendants of that code base. And the other, there's lots of other BSD distributions. I'm not trying to cover them exhaustively. But this is where FreeBSD came from. It's not a Linux distribution. It's not really related to Linux, other than the fact that Linux and UNIX look similar in a lot of ways, more or less, depending on who you ask, probably. The other thing is that FreeBSD is a complete software system as opposed to Linux, which is a kernel, and then the software distributions turn it into a full operating system. FreeBSD is a full operating system. So the FreeBSD release cycle looks probably like your typical software release cycle for an open source project. There is a head branch, which is the development branch. That's where new features and bug fixes go first. And then there are these release branches that come off of that every once in a while. And the release branch is called the stable branch. So for instance, there was a stable 11 branch about two years ago, I believe, which is the branch that all of the 11.x releases come off of. All of the releases that are part of the 11 major release, all the minor releases come off of that release branch. Likewise, there recently was a stable 12 branch that was created. Over time, of course, the code diverges, as you'd expect, with a software development model like this, and the new features go into head, and not all of them are backported, depending on just how stable they are. So a while ago, when I first joined Netflix, it was right around the time that we were running stable 10, which was the major release before stable 11. So our operating system was based on stable 10, and we had a choice when stable 11 was released. We could stay on stable 10 and get farther behind, or we could switch to stable 11 or head, and we decided to rebase our code onto stable 11 or head. And then we had the choice. Well, it basically stable 11 and head are the same thing at this point, because it's just branched. So which do we wanna do? Do we wanna go to the next stable release? Or do we want to track head? And we decided to track the development branch, and so we rebased onto the development branch, and we began to track that. And we did that for about two years, and then stable 12 came along, and we had a choice. Follow stable 12 or stay on head. And there was essentially no discussion or debate. Everyone loved tracking head so much that we've continued to do that. So it's worked really well for us. This is sort of what the code inheritance looks like for those of you that need this mental model or appreciate the mental model. So FreeBSD heads code bases at the top. We have our internal master branch, and about every five weeks we take a sync from head to our internal master branch. We develop some more code and then we branch off our release branches. And we shoot to have a release just about once every five weeks. Our typical release cycle is five weeks of development, followed by five weeks of testing. The five weeks of, so there's two releases in flight at any given time. One in development, one in testing and deployment. We start off with a free, emerge from FreeBSD head close to the start of the development process. Then we also, throughout the five weeks we're doing feature development and integrating our features into the master branch and with the other features that have been developed. And of course we're doing continual testing throughout the five weeks. At right at the five week mark we make our release branch and then we begin the process of doing further testing, further intensive testing on that release before we deploy it more widely. So at this stage we would be running this release on approximately, what we'd be using approximately 45 caches to test this release with a mixture of representative hardware from the different kinds of hardware that we support. Half of those caches would be running the old operating system, half the new operating system and we would do a very close comparison of the performance of the old operating system versus the new one. And so we look at obvious things like panics and crashes and so forth. But we also look at the metrics of how the system is performing. It has the CPU utilization gone up or down. Are we able to push out the end width as fast? Are we able to push out bits as fast? Have TCP retransmissions changed noticeably? Has power usage changed noticeably? So we look at all of the metrics you would probably think that we look at and maybe even a few more regarding the system itself. We also look at the metrics the clients give us. So remember the clients can tell us whether or not they like this new software. And if the clients tell us that their experience getting data from the cache running the new software is as good as the experience using it from the cache running the old software, that's a pretty good indication that we haven't messed anything up. And if their experience is even better, that's great. After we do our dev testing and we fix whatever bugs we find, it moves on to a canary testing. So we've deployed at this point to a much larger set of caches. So many more clients are hitting this and we can again do the comparison to see if things have gotten better or worse or stayed about the same and make sure that matches our expectations. And then finally we go through about two weeks of deployment. In an emergency, which does happen periodically. So for example, a security vulnerability or something like that, we can speed up this process very greatly.  But if we need to, we can rush out fixes in a hurry. So one of the questions you might be asking is, well just what kind of features do you need to develop? And the features that we develop are basically aimed at one of two things. The first one is making our customer experience better or making the OCAs run more efficiently thereby lowering our costs. And we want to do one of those two things. Ideally we want to do both at the same time. But we try very, very, very hard not to sacrifice customer performance in order to save money. So we try to do both at the same time. And there are things we can do as software developers to make the customer experience better. So for example, getting bytes off the disk faster is something that can help the customer experience because their video starts playing that much faster. And there are things we can do as software developers that will make that better or worse. But here's some examples of some features that either we've developed internally or sponsored in the community. The first one is NUMA enhancements. The previous NUMAs had NUMA support for many years but it hasn't worked for our particular workflow very well when we've tried it. So, or at least it did several years ago. So we've explored adding, or we've been adding some enhancements to it to make it work better for everyone but also particularly for our workflow. The next two asynchronous send file and kernel TLS are things that enable us to get higher speed, higher efficiency through the box by avoiding a bunch of context which is in copying data which in kernel and user space. I think other people are probably, some other people are probably using DPDK in this space. We aim to keep as much in the kernel as we can. And so kernel TLS and asynchronous send file are ways that you can, the user space can tell the kernel, please do this for me. And then the kernel takes care of it for the user space. A string for the send file, you tell the kernel, please send these bytes from this file over the socket and it will just do that. And it'll only really tell you if there's a problem. The kernel TLS, you give the kernel the right encryption keys then the kernel will go do the TLS encryption for you. And so you combine the two together and with one system call you can send a large chunk of a file to a user without the user space having to make further system calls or do more copies. The next two were Pbuff allocation enhancements and unmapped mBuffs. Those are both ways that we gain efficiency by taking bottlenecks out of the system. The Pbuff allocations, Pbuffs are the physical buffers that handle data or store data when it's coming off of the disk. So they would be buffers for disk operations. There was contention for a lock which was making the performance very bottlenecked. And so we did some work to rewrite that and be able to get more parallelism during the Pbuff allocation process. Unmapped mBuffs, again it was about taking bottlenecks out of the system. Someone on my team managed to find a way that we could take the same amount of data and instead of representing it in I think it's like 64 network buffers we can now represent it in one network buffer. IO scheduling, again we can really make a significant impact on users if we get data off the disk at the right time. It actually really matters when you read and write data. You don't want to read it too early, you don't want to write it too early, you don't want to read it or write it too late either. So it really matters and we've done some work to make sure that we are doing IO at the right time and doing the right amount of IO and so forth. And likewise the same thing applies to trims and we've also been doing some work to make sure we send down the right trims at the right time and also at the right pace because sometimes disk drives don't behave well when you send down a whole bunch of trims all at once and so we've done some work on scheduling those as well. And finally we do work in TCP regularly so there's a whole group that works just on TCP algorithms and so forth. We've done a lot of work in both research and development in algorithms to get TCP to transmit data better where better would mean again sending the right bits at the right time for that session. And we've also added some logging and debugging infrastructure so we can tell just what's going on when things don't work the way we think they should. And that's been very useful for us in developing these things and we've shared all of this with the, I think we've shared all of this with the community at this point. So I think that just about all of this is either upstream or is in the process of being upstreamed in some way. Tracking head makes it, lets us stay forward looking and stay focused on innovation. The problem that you can run into when you are tracking the stable branches is that you continually have to deal with the problem of my code doesn't match what everyone else is developing on upstream. And so when there are bug fixes that occur or new features that you want you have to figure out how to get them back into the code that you're actually running. Likewise, when you get the new merges from upstream sometimes you have to redevelop your features because the APIs have changed or data structures have changed and all of a sudden the code you had doesn't work anymore. But by tracking head we're able to stay very close to what's going on in the upstream development community and instead of continually redeveloping the stuff we've already done we can just work on developing new things that will benefit us and hopefully benefit other free VST users as well. My hypothesis is that downstream users of open source projects can get stuck in one of two cycles which I've labeled vicious and virtuous. And this is not entirely original thought but I think you can easily get stuck in this vicious cycle, what I call the vicious cycle which starts with infrequent merges. And so if you're a downstream user and I don't mean end user in this case, I mean a company like us that's trying to write an operating system or either write an operating system and use an upstream operating system and customize it for uses or use an upstream application and customize it for uses, that's the kind of user I'm talking about. And I think this cycle starts with infrequent merges. So when you start with infrequent merges and this is infrequent merges of the development code. So this could mean that you are tracking a stable branch and or a release branch and then every year or two you jump to the next release branch. That's in essence, that's an infrequent merge of the development code. And when you start off with that, what happens over time is you get farther and farther away from the upstream project. So when you do actually try to do a merge, you have a lot of conflicts. You have regressions, things don't work right. And it becomes very painful. And so every time you do one of those merges, it's very painful and disruptive to the organization. The developers who were developing code suddenly have to stop developing new features and go back and make all the old stuff work again with the new code. And that's not very fun to do as a developer, but it also means that's a whole bunch of time you're not writing new features.  And when you do them less frequently, you have even more merge conflicts. And you have more regressions, which means that you have to spend even more time fixing them all, which means your feature velocity gets even slower. And the whole thing is so painful that you want to do the merges even less. And at some point, maybe you even stop taking merges from upstream, but instead you just simply try to cherry pick back the bug fixes that you think you need. So you can compare that to what I call the virtuous cycle, which starts with frequent merges of the development code. So when you're frequently merging the development code, you're diverging from it less over time. And that means that each merge is much easier. There'll probably still be conflicts, there'll probably still be regressions, but there should be fewer of them. And each merge should be much easier to handle. Furthermore, because you're so close to the development branch, it's easier to get things upstreamed. It takes time, upstreaming things is not just about pushing your code into the upstream project. Rather to upstream things, first of all, you need to generalize the feature and make it generally applicable to all the users of the project. On top of that, you probably need to do some code cleanup because there's probably some rough edges. And then if you are, and that's just if you're running the development branch, if your code is based off of a stable branch, you need to do the extra step then rebasing your code onto the development branch. By running the development branch internally, we're able to skip that last step. So we still need to clean up our code, we still need to generalize it. And there's some that costs extra money for us, but we're willing to do it because we think it's the right thing to do. And all of that means that we can get our code upstreamed that much easier because we don't have to rebase it regularly onto the development branch to get it upstreamed. And having that code upstreamed means that our diff against the development branch of the upstream project is relatively small. That lets us focus more on getting our developing new features. So we're spending less and less time dealing with old features, we're spending more time dealing with new features. And on top of that, it lets us collaborate effectively with other people in the upstream community who are using this project. Because we're, most FreeBSD developers run FreeBSD head and because we also use head internally, we are able to directly share patches with them and they can apply the patches, they apply cleanly, they get the NUST patches, they apply cleanly, and so we can collaborate together when we're doing our work. And all that means is that it makes it easy for us to do more frequent merges. In fact, we've gotten to the point where my developers will sometimes just commit stuff upstream to the upstream project rather than to our repository and we'll just pull it back in the next upstream sync that we do. Now having said that, we do keep local diffs for a few reasons. The first one should be fairly obvious, it's information covered by NDA. We do occasionally get information from vendors, which helps us to do something we need to do. But it's covered under an NDA and they will not let us upstream the code based on that. I think we have something like 100 lines of code that fall in this category. So it's a very, very, very small diff. But this does happen occasionally. We also have features which are still in development or testing. We will sometimes commit features to our tree for testing purposes and we don't upstream them usually until we're sure they actually work. And so we'll keep that local diff until we're sure the feature works and we've had the chance to generalize it. And that's the third reason we keep local diffs. There are features we have which are too specific for us. They're things that help us with our workload but they're not general enough. We have a very specific workload. As I showed you, our whole business is getting data off of disks into memory, encrypting it in memory, sending it out on a NIC. That's what we do. And there are other people that use computers for other things. And what is good for us might not be good for them. On top of that, we have a rather homogenous environment. All of our caches are Intel 64-bit. And there are people that run other architectures. And again, what's good for us in Intel 64-bit might not be good for ARM 64-bit or ARM 32-bit or RISC processors and so forth. So there are features that need to be generalized and we will hang on to those internally until we've had the chance to generalize them. And once we have generalized them and made sure that they work for everyone or at least don't make things worse for anyone else, then we will go through the process of upstreaming them. But it's our intention to upstream any code that we can. I think that this is one of the great things about working at Netflix. I've worked at other companies that used open source software as the basis for their product. And they kept everything inside. So they had lots of local patches that it was very hard to share outside of the company. Netflix does allow and in fact encourage the use of open source software or it makes sense, but they also let us freely contribute our software back. And so the approval process to upstream a feature is essentially, it's automatically approved. And that's really nice. So we can get our software upstreamed rather easily from the Netflix perspective. And we're willing to pay our developers or give them some amount of time to go through the process of upstreaming those, cleaning up the code, generalizing it, making it work for the broader community because we think there's value in upstreaming what we've done. So we get a couple of benefits from tracking FreeBSD head. The first one is we get quicker feature iteration. As I mentioned, we're not, our developers aren't constantly stuck looking backwards, but rather we can, they can develop looking forwards and they can rapidly develop new features knowing that they're going to be able to get them, they're going to develop them once for us in our code base. And then that's the same code base they'll be upstreaming them into. And so they don't have to think about what's going to work in stable 10 and then what's going to work in head and sort of develop the features twice. They can just develop them once and just worry about the one environment. We also get quicker access to new FreeBSD features. And this is probably the right point to mention that we're very happy running FreeBSD. We get great performance out of FreeBSD. It's very stable. And we get access to lots of really cool features that people in the FreeBSD community are working on. And so it's a great benefit to us that we run, we track heads so that we can get access to the new features as soon as they're committed. And so someone can commit a new feature today and it can be running in our network somewhere between five and 15 weeks from now, typically. So we like the fact that we get quick access to new FreeBSD features and we can do cool things by getting access to those features. We also get quicker bug fixes. And this isn't just because the code gets committed first to head, that's true. So FreeBSD developers, they're fixing a bug, they'll put it in a FreeBSD head. And then once they're sure it works and it's had some soak time, they'll back port it to older releases. So we get access to the bug fixes a couple of weeks earlier, generally running FreeBSD head. But on top of that, we also get quicker bug fixes because our bug reports come much closer to when the code's committed. So let's say you're an upstream developer in the FreeBSD project and you commit code today. It may be two years before that ends up in a FreeBSD release used by the general community. Once that happens, two years from now, someone may find a bug and report it to you. Are you gonna remember why you wrote that code two years ago? Contrast that with what happens when we find bugs. You commit the code today. In the next one to five weeks, we get the code in a sync. We find a bug and we report it. And we report a bug to you on code you just wrote a few weeks ago. You're much more likely to have the context in your head and it's likely that you'll remember why you wrote the code the way you did and you can quickly help us troubleshoot and fix the bugs. On top of that, they're easier to fix generally because there's not two years of other stuff built up that depends on that bug being there. That sometimes happens over time. People, they rely on a bug being there because it's been there for two years. They think that's just the way things are supposed to work. And you try to fix a bug and you end up having to untangle this rat's nest of dependencies that it built up by finding the bugs sooner and reporting the bugs sooner. We're able to generally get fairly quick fixes and before lots of other dependent stuff has built up that makes it harder to fix. It also enables collaboration. As I mentioned, most freebies-d developers run freebies-d head and so we're able to directly share patches with them and that's good. We really like the fact we can collaborate with other people in the community. And I wanna be clear, it's not a one-way collaboration where we're trying to figure out what we can get from other people. We really do like collaborating with people and being able to share the patches we have, the things we're thinking about and also find out what they're thinking about in ways that we can work on projects together, perhaps. It also minimizes our merge conflicts. As I mentioned, sometimes our developers will even just commit code directly to the freebies-d project and we'll pull it back as part of the DexSync. In other cases, it may take us a little bit of time to upstream things, but we can upstream them much more easily because we don't have to go through that step of rebasing onto freebies-d head first. And so that really helps us keep our local def low and minimizes the merge conflicts that we have. It also amortizes the merge cost. So if we did, once every two years, we switched, rebased our code on to the latest release, that would be a very painful, long, difficult merge. Instead, we spread it out and we do it once every five weeks or so, we do a little bit of that merge. And so it's in much more manageable chunks. The diff is much smaller, it's much easier to spot bugs. If we absolutely needed to, it's even possible to go do a binary search through the code that's been committed to figure out where a regression was introduced. So we've got much more manageable diffs, it's much easier to deal with. But overall, I would say we probably spent about as much time doing our, in aggregate, we spent probably spent about as much time doing our once every five week merge, as we would on the big merge, at the end of the two years. But I think that spreading it out is much more pleasant for everyone. Freebies-d also, I think, gets some benefits from what we do. The first one is that they get wide deployment of their head branch very early. So their head branch is, usually a commit on the head branch is deployed in our network somewhere between five and 15 weeks after it's committed to the Freebies-d head. And I think it's a testament to how stable the Freebies-d head code is that we can do this. It's a very stable, very usable release. And we're able to successfully deploy that code very quickly on a large number of caches. And they get the benefit of having that early deployment and that early wide deployment of their code so that we can find bugs. Now, obviously we try to find them before it's deployed that widely, but just in case they slip through, their code is still seeing use on a large number of content caches around the world. They also get early intensive testing of that code. We test the code thoroughly before we deploy it and, again, it gets tested in production. And we also have an incentive to upstream our code. And so we, because we wanna minimize our diffs, we have a great incentive to upstream our code and it's easier for us to do that. And so I think more of our features end up in the upstream code base than might otherwise. And I think that's good for the FreeBSD community as a whole. Again, we're not trying to just get our code into upstream FreeBSD, but we want to benefit the upstream project by giving them access to the features we've developed internally. And we think that that's good for everyone. So you might think of some objections to running development code. So the first one is, it isn't stable, right? Well, actually it really is. The FreeBSD community is fanatical about keeping head stable. And in fact, it may even border on unhealthy. It's, they're that fanatical about keeping head stable. Which is great, I think is great. It means that you can download and use FreeBSD head and it generally just works really well. Why should you pay to find the bugs that others are gonna find while they're testing head? Well, the obvious problem here is if everyone has this mindset, who's actually gonna test head? Who's actually gonna test the code? And someone has to do that testing and I think that we're well positioned to do that testing for, at least for our use case. On top of that, we would probably still find some of these bugs anyway, two years down the road when we get the code. But it would be much harder for people to fix because they would have lost all that context about why they committed that code. Aren't there more security bugs in development code? Well, actually no, not in our experience. I think almost all, if not all of the security bugs that we've run across in the past two years have, most of them, have been in the stable branches as well. And so on the whole, we probably have the same security bugs in general that the stable branches have. But the people developing the fixes for those security bugs are developing them on the head branch. And we get access to those security fixes that they've developed for the head branch. No one does this. Well, that's not completely true. There are other companies that do similar things. There's one storage vendor that announced a couple of years ago that they were going to try to base their storage appliance operating system on FreeBSD head. You pay a monthly cost to do merges. That's true, we do, but as I said, in aggregate, I think that it's no more than if we waited and did this once every two years. And I think each one is actually much easier and much more pleasant for everyone involved. And you get new bugs each month. Yes, that's true. We do get new bugs each month. We also get new features. And the new features are really cool. And guess what? If we were running the stable branches and we did emerge once every two years, we'd get new bugs then, too. So, but we wouldn't be getting new features during that two-year time. So I think that on the whole, it's true, you get new bugs each month, but it's manageable. It's at a manageable pace. They're a little bit easier to debug and fix. And it's counterbalanced by the new features we get, as well. So in conclusion, running FreeBSD Head has been really, really good for Netflix. It lets us deliver very large amounts of data throughout the internet very efficiently while maintaining a high degree of feature velocity. We've found FreeBSD Head to be very stable. It's stable enough that we deploy their commits in our network, usually, as I said, about five to 15 weeks after they're committed. It lets us deliver a large amount of traffic very efficiently, and it lets us maintain our development agenda. So we're very happy using FreeBSD and, in fact, using their head branch as the basis for our operating system. It looks like I have about five minutes if anyone has questions they want to ask. Are there any questions people have? Raise your hands high. All right. I see one right there. Is that it again? Hello. How much of the code is not ported back to FreeBSD? How much is not? Like which features, for example. What's that? Which features, for example. Which features are not ported back? As I said, our aim is to port back all the features, except for the ones covered under an NDA. At this point, there are a few features, particularly in the TCP area, which are not yet upstreamed. But we're in the process of working through our backlog and upstreaming them as we can. And we're also usually happy to share diffs, if we can. And so when people write us and ask for diffs, we're happy to share them. And we give them the caveats of, this isn't quite ready yet. Any other questions? Why did you choose FreeBSD kernel instead of Linux kernel? Can you repeat the question, please? Why did you choose FreeBSD instead of Linux? Why FreeBSD instead of Linux? OK. Originally, from what I understood, I wasn't there when the decision was made. Originally, from what I understand, it was all based on the license. So it was based on the BSD license versus GPL. At this point, we stick with it because it has a combination of a very positive, collaborative development community, a good set of features that are at a nice, stable velocity, and good performance for us. We have tested Linux in the past several years, and it didn't give us as good performance as FreeBSD. But of course, it's not exactly a fair comparison because we didn't do all the engineering work on Linux that we've done in FreeBSD. So it's not exactly a fair comparison. But we clearly would have to do a little bit of engineering work to be able to switch to Linux at this point if we wanted to. But as I understand it, the original reason was the license. But we're very happy with FreeBSD. And I think we get a lot of benefits from using it at this point. Any other questions? Yeah, go ahead. How many developers are focused on this in Netflix? How many developers? How many in Netflix for this topic? How many developers is in Netflix for this topic? How many developers work on the operating system? So my team has six or eight people, depending on how you count. And the team that works on the TCP protocols in that, I think, has four people on it. And there's a couple of other people that also assist us. So all told, I would say probably 16 or so. So it's a nice, small, manageable team. I wanted to ask, which file system of FreeBSD do you use? Is it UFS or ZFS? We use UFS. And are you using SSDs or just hard drives? We use NVMe, SSD, and hard drives. We use them in different systems for different reasons. We have some systems that are meant to store large amounts of content, but serve it less efficiently. And those have hard drives in them. We also have our 100 gig appliances at the other end of that. We have 100 gig appliances, which are meant to serve a much smaller selection of data, but serve it very fast. And they would be NVMe or SSD. For your development testing and your canary deployments, what is the percentage of your user base you deploy to? For the canary testing, you mean? Yeah, so for the, we do several stages of testing. The first one is to about 45 caches or so. The next stage is to a larger number. I'm not sure exactly what it is. I'm sorry. But it's, yeah, I'm not sure what the number is. I'm sorry. It's another group that does that. At that point, we've handed it off to another group, and they do that testing. So I'm not sure. I don't know that answer. I'm sorry. OK. Thanks again, Jonathan. No problem.