Good afternoon. Thanks for coming. This session is about availability and storage auto-scaling of stateful workloads running on Kubernetes. My name is Layla. I work at Shopify as an infrastructure engineer. I'm on a team of 10 engineers called the search platform who develop and maintain the infrastructure that powers search at Shopify. This is the agenda for this talk. In the next 20 minutes or so, we'll learn how the search infrastructure team at Shopify hosts a stateful workload, which is search on top of Kubernetes. We'll talk about the obstacles of providing high availability and storage auto-scaling and scaling in general for stateful workloads that run on Kubernetes and how my team address those challenges. I also have a pre-recorded demo at the end and we'll conclude the presentation with a Q&A. A little bit about Shopify for those who don't know about us. Shopify is a cloud-based commerce platform that lets you start and manage a business by allowing you to create and customize an online store and manage inventory, payments, and etc. Currently, we have more than 3 million merchants that sell a lot of products with us and they collectively have sold over 700 billion dollars in gross merchandise volume. Just to give a better sense of the scale, only during the Black Friday Cyber Monday weekend of 2022, which is a major high volume commerce event that kicks off the holiday shopping season, our merchants sold over 7 billion dollars in GMB. Search is a fundamental part of any commerce platform that allows buyers to search and filter for the products that they're looking for and also allows merchants to fulfill the orders that they have received. When you go to any online store and search for a certain product, your request goes to a search engine that's backed by a secondary data store, which is different from traditional databases. We call this secondary data store the search infrastructure. At Shopify, we use elastic search to provide search and filtering services to Shopify Core, which powers all of our merchant shops as well as their storefronts, as well as the development teams who need search services. We run Shopify's search on top of Kubernetes on Google Cloud Platform and deploy and maintain these elastic search instances by using a custom Kubernetes controller that we have built. So let's catch up on some definitions. Stateful services unlike stateless ones rely on persistent data and respond to the same inputs in different ways depending on their history. According to this definition, elastic search is considered a stateful application that provides a stateful service and therefore we deploy it using Kubernetes stateful sets. Now let's see what stateful sets are. A stateful set is basically a controller that manages pods that are based on an identical container spec and maintains a sticky identity for all of its pods. These pods are created from the same spec but are not interchangeable. Each has a persistent identifier and more importantly has a persistent disk that it claims across any rescheduling. To have a, again, a better sense of the scale, an elastic search stateful set that stores search data for Shopify Core is composed of 120 pods, each of them having a four terabyte persistent disk. For high availability reasons, we have multiple instances of elastic searches with the size deployed across the globe. Speaking of availability, with search being a critical service, we need to make sure that it's always available, meaning that failures of pods, systems or even natural disasters that lead to regional failures should have minimal impact on the service. We also need to make sure that elastic search stateful sets always have enough storage space and they're able to adapt to sudden changes of storage requirements. The first question we want to answer here is whether deploying a stateful service with Kubernetes automatically guarantees fault tolerance and high availability. We know that the main step towards availability of fault tolerance is redundancy. One might think that since Kubernetes stateful sets manage multiple pods based on an identical container spec, running a stateful set will automatically provide high availability by adding redundancy. However, this assumption is not true at all. Consider this stateful set with three pods. Fault tolerance means that the application is able to provide the same service even if one of these pods fail. But can we say that it's true for a stateful set? Depending on the application that the stateful set is running, each persistent disk has a different set of data. So, for example, if pod zero fails, although pod one and pod two are still running, they will not have access to pod zero's data set. And therefore, they cannot provide the same service that pod zero was providing before its failure. So, the service will not be available until pod zero is recovered by Kubernetes. In other words, although we have redundancy for the pods, we don't have any redundancy for the data that's stored on the disks. So, the lesson learned here is that Kubernetes does not provide data redundancy out of the box, and it's up to the application to replicate the data. Taking elastic search as an example, the data stored is sharded by elastic search, and each primary chart can have multiple replicas. Elastic search has a mechanism to distribute the primary and replica charts across the disks in a way that if one of the disks fail, there is at least one disk that has a copy of the data and is able to provide the same service. Applications replicating the stored data improves availability, but we still need to be ready to survive regional failures. For that reason, we have implemented a pipeline using the Kafka streaming service that replicates an entire elastic search state full set between two geographical regions. We currently have more than 100 elastic search instances living across the globe, collectively storing almost two petabytes of data. As mentioned earlier, one of the reasons that a stateful service becomes unavailable is a lack of enough storage space. For example, during high load events, such as the BFCM weekend, the Black Friday Cyber Monday, the required storage space increases, and the disks can get full rapidly. We could pre provision extra storage to avoid running out of storage, but we will have to scale back down after the high traffic event, otherwise we will be paying for the resources that we're not actually using. And we will see later that scaling down stateful systems is not a trivial task. It is also possible that as time passes, your requirements change, and you no longer need the amount of storage that you provided in the beginning, and you need to scale back down. For both situations, we will need to make our storage scalable to adapt to the changes of requirements. This way, we will only pay for the amount of storage that we actually need. And we also won't run out of storage. This brings us to the next topic, which is storage auto scaling and its implementation. Now let's take a look at how my team implemented storage auto scaling. As I mentioned before, our elastic search stateful sets are deployed and maintained by a custom Kubernetes controller that we have built. This custom controller monitors elastic search custom resources, and their corresponding stateful sets and other objects that in general form an elastic search instance. One of the tasks of one of the many tasks of this controller is to make API calls to its managed elastic search instances. To get the current free storage space based on a defined heuristic, the controller decides if the disks need to be expanded or become smaller. Excuse me. Yeah. But how can you expand or shrink the disks of a stateful set that already has data? To update the disks of a stateful set, the first approach that comes to mind is by simply running cube control edit stateful set and updating the storage field to a different amount that you want. And you will see that Kubernetes does not allow you to do this. Here in this recorded video, I this is my attempt to edit the already existing stateful set. I am editing the storage field in this example from two to three typivites. And you see here that Kubernetes does not allow you to do this. The error shows that you're only allowed to update certain fields in a stateful set spec, replicas, templates, and et cetera. And storage is not one of them. So how can we make this work? Let's dive deep into details of handling storage scaling. I'll start by by the case. Sorry, I have lost my mouse. Okay, I'll start by the case where we no longer need the storage we have. And we realize we have over provisioned the storage handling the scenario is challenging because Kubernetes does not have a feature to automatically support scaling disks down. The reason this feature does not exist is that because shrinking disks by mistake will lead to data loss. And it should be implemented with care and handled in a case by case basis. For our use case, the custom controller, as we mentioned before, calls the elastic search API and has the knowledge of whether like this disk scale down is allowed. And based on the actual storage usage, it triggers a disk scale down if if the numbers allow it to scale down the desk. Remember, we cannot update the stateful set. So instead, the custom controller deletes the stateful set object with the cascade flag set to false. This is important. Setting this flag to false allows keeping the pods and disks after deleting the stateful set object. So after the stateful set is deleted, the custom controller recreates it with the new smaller size. All the other specs remain the same, but only storage changes from eight, for example, to four tabby bytes. Since here we are scaling down storage. But replacing the stateful set object with a new one does not automatically shrink the disks. There are other steps that the controller needs to take in order for the storage and the desk to reflect the changes you made to the stateful set, the custom controller should first delete the volume claim object. Deleting it will put the claim with the will put the volume claim object into a terminating phase, but it will not be actually deleted because there's still a pod that's bound to it. So we want to force this termination. To force this, the controller will also need to delete the pod that's bound to the terminating claim, and that will delete the pod and the disks for real. The reason we don't delete the pod first and then the volume claim is that if we do so, the stateful set is faster than us and will recreate the pod before we're able to delete the claim and we won't be able to proceed. Now that the disk and the pods are and the pod are deleted, the stateful set will do its job and will recreate the pod and the disk now with the new smaller size. The controller will have to do this one by one for every disk. As you can see, this scale downs don't come ready out of the box at all, and should be handled carefully to make sure that there are no data loss happens. In our case, the custom controller drains every single disk before terminating the disk to make sure that no data will be wiped from the disk. You can imagine that doing this for the large clusters that we have is very time consuming and also error prone because there are many steps that this controller needs to take to make this work. For example, one of the custom controller calls to delete a pod might fail and the whole scale down process might get stuck and you would need an engineer to intervene and solve the problem. In the beginning, we used to follow the same process for disk scale ups as well, meaning that we would replace and delete the stateful set object, delete the claims, the pods and all that steps and in order for Kubernetes to create a larger disk for us, fortunately we found a better solution based on a feature that Kubernetes provides. Kubernetes has a feature called volume expansion. In order to make use of this feature, the disk should belong to a storage class that allows volume expansion. Let's take a look at a stateful set spec and the volume claim template. In the part that's marked red, we see our templates, our volume claims are created based on a storage class called the ESPD SSD and looking at the spec of that storage class, we see that there is a field called allow volume expansion that is set to true. So any disk created based on this storage class has the ability to be expanded. Setting this field to true is a requirement, however, you need to take some additional steps to make this actually work. Now, let's take a look at this case. We are trying to scale up the disks of a stateful set. The custom controller to do this, the custom controller again needs to delete a stateful set object. Remember we cannot update stateful sets with the cascade flag of course set to false. Once the stateful set object is deleted, the custom controller will recreate the stateful set with a new size. Now, there is a difference here. We no longer need to delete the volume claims. The controller just needs to update the volume claim objects. The storage from like four to eight tebibytes in this case for example. And the rest is taken care of by Kubernetes. Kubernetes will automatically expand every disk without downtime and this will be as you can see this is a lot easier compared to the case where you had to delete every disk and drain and like be extra careful. Oh okay. I have a demo in here. It's a pre-recorded one. We have a stateful set with three replicas and we see here that each pod has a volume claim of four tebibytes on the right hand side. I'm just triggering a disk scale up by editing a custom resource that we have internally built called the elastic search instance. I'm updating a field of data volume size that represents the size of the disks of a stateful set elastic stateful set. Doing this is basically triggering a scale up. And here I'm watching the disks as they get expanded by Kubernetes. We already see that the first one is expanded to five tebibytes. I think I have to skip over a little bit because at the same time yeah here we see that the second one is expanded and after a while the third one is also expanded. So you should expect each pod each disk to be expanded in about a minute. But I wanted to just skip over. Okay. Just like for our case of a 120 disk cluster it takes about five or six minutes to scale up an entire large cluster. The reason is that we do this in batches and some of the scale-ups have it like most of the scale-ups happen in parallel. There are two ways to increase storage. You can either expand the disks you already have which is called scaling up which we just covered and you can also add more disks which is called scaling out. Adding more disks is a rather straightforward approach. No need to delete the stateful set or update the disks. You can simply update the replicas of the stateful set which we now know that it is allowed and more pods and disks will be added. But this approach is not always cost efficient. The reason is that in addition to increasing the number of disks you will also add compute resources which most of the time are not really needed and we're only interested in adding storage. To make the best decision you should compare the cost of scaling out and scaling up and go with the approach that is the most cost effective one. To summarize we learned today that Kubernetes does not automatically provide availability and fault tolerance for stateful services and it's up to the application to implement data application and provide availability. Also, storage auto scaling can be automated by using custom controllers. We saw that scaling down storage is not a trivial task and Kubernetes does not support it and it should be implemented with care to avoid data loss. And lastly we saw that scaling up storage is more straightforward and there's a Kubernetes feature that allows for volume expansion. Thank you for coming and if you have any questions or feedback I'll be here. Thank you. I see a question, two questions I see. I think there are mics over there if you want to ask yours. Hi, thank you very much for your talk. I was wondering if the custom controller it's available open source and if you are using the official custom resource definition for the elastic search or you build your own as well. It's an internal custom controller that we have built. It's not available externally and the same is for the custom resource. Thank you for your presentation and my question is how do you handle collisions? So for instance you're in the middle of downsizing the volumes for instance at node number 20 and then there is another event that says that you need to expand the volume. So what's the logic behind that? This logic like we have some sort of locks that we have built in the the custom controller. So this like since it's the controller that decides whether the disc should be expanded or become smaller this collision is basically not going to happen because it's the controller that is deciding about this. So if controller thinks that this should be expanded but it will continue to contract those right? So the custom controller first of all like it sees the actual usage if so we're talking about the case that it needs more storage. So it comes up with a number that like this would be a good number for the discs, the new number for the disc and it triggers the scale up and like we saw the process that happens. The number that the controller picks is a number that it will not make the discs over provision so that in another loop the same controller will will decide to scale down the discs. So these two like competitions will not happen. Thank you. I also have a question. So you said that there's a replica shot that basically makes sure that you don't lose the data. So is there any kind of mechanic where you first downscale the replica shots or first the original because you have to make sure that it stays the same right and the size have to stay the same and you have to save the data so is there any technique or structure that you do with it? That's a really good question. This is actually implemented by Elasticsearch. Elasticsearch allows you to deploy or like spread or distribute your shards in different availability zones. So the primary shards and the replica shards of primary and the replica of the same shard they don't end up in the same availability zone. So we built our custom controller in a way that it only touches one availability zone at a time. So it's never the case that you are expanding or shrinking the discs of that has the both primary and replica shards of the same shard at the same time. Is there a difference if you first touch the primary shot or the replica shard? No, there is no difference. Okay, thank you. I might also have a question. Thank you. Thanks for the talk at the first point and we are also using Elasticsearch and I'm wondering what we have noticed is if you update for example an Elasticsearch instance and you have a rolling upgrade for example. Elasticsearch needs a moment to redeploy their replica shards and the the state is in a yellow state for the time being. How did you take care of that? Well, you delete the stateful set of course and then this says the same then you build a new one but yeah have you ever had any problems in time-consumting for example if the replica shards are being replicated and rebuilt and stuff like that? If it takes too long, well as I mentioned in the earlier slides we have the same Elasticsearch dataset in different regions. So we have this mechanism of failing over traffic. If we see that this is taking too long the Elasticsearch cluster has been stuck in the yellow phase. We just fail over traffic to a region that is not under this maintenance yet. So just to be extra careful but it has never led to data loss like that I can say for sure but how fast it happens it really depends on like if you're in different availability zones that's fine because you can simply switch over to another availability zone and be done with it. We can't do that unfortunately because of data protection and stuff like that but yeah it's an interesting idea. Thank you very much. Thank you. All right so you mentioned for the scale out that it's relatively easy to just add replicas to the stateful set. When you scale back in do you have to do any additional steps or then do you just rely on elastic to redistribute everything when you take parts away from it basically? I don't think you can rely on Elasticsearch. When you basically scale down on a stateful set like Kubernetes has no idea what's going on inside the pods so it just starts cutting down pods from the the pods that have the higher index number from the the ones that were created later so unless you have set your volume claim policy to retain you will lose data if you just decide to scale down. We allow scaling in by using our custom controller and the way that we do that is when such updates happen like when such things that are requested for some pods to be removed it is really slow. The custom controller one by one again drains every disk and like makes sure that the disk that is about to be deleted is actually empty sometimes it gets stuck because the person because these are not automated like we don't allow this to be done automatically so if a team requests their elastic search stateful set to be scaled in by mistake like they think that for example two pods are extra but at the but they're actually wrong and it's only one that's extra so the custom controller will not allow this it does some calculations it sees like how many what's the storage size free storage and if it's really allowed so we have this protection otherwise you will lose the data of the disks oh sorry that is deleted. Okay so just to be clear you still go through the custom controller even when you're scaling out even though you could just edit the stateful set that part wasn't clear. Yes yeah we also go through with the custom controller. Thank you. Hi thanks for the talk. A question about the pvs you're deleting them right? The pvs not the claims. Yeah they are storage yeah right and so how do you re-sync the data do you rely on elastic search or do you have any other mechanism to achieve this? Re-sync the data after uh can you yeah you delete the pv and then you bring one back in which has a lower disk size and so what you do with this how do you re-sync the data is it some custom I suppose you rely on the elastic search for this? Yeah that's true we rely on elastic search like the elastic search has a mechanism of rebalancing shards so that almost every disk has enough has the same amount of storage obviously like it's not always possible to have really close disks but yeah like once an empty one comes in elastic search will trigger a what we call shard relocating and it moves back like some of the stuff to the newly created disk. Okay and what does it take in time is it hours or on hour? Yeah it's in the scale of hours. Okay thank you. Yeah thank you. Okay I guess that's it thank you.


Over the last 30 years, the Australian dairy industry has faced a number of challenges, including changing product markets, variable world prices and prolonged periods of unfavourable seasonal conditions. Faced with these challenges, the industry has remained resilient and undergone significant structural change. Milk production is now more efficient and uses fewer resources per litre of milk produced, which is evident in the productivity performance of the dairy industry. Productivity is a measure of how well producers combine inputs to produce milk. Broadly speaking, it is a ratio of output to inputs. Over the last 35 years, productivity in the Australian dairy industry grew at an average annual rate of 1.6%. This compares favourably with productivity growth in broad acre agriculture, which grew at an average annual rate of 1.1% over the same period. Significant factors in the productivity performance of the dairy industry have been widespread adoption of new technologies such as larger and more automated milking sheds and many smaller, less efficient producers leaving the industry. Additionally, dairy farmers have used extension services and other learning services to gain a better understanding of dairy production systems to lift average milk yields per cow. Going forward, there is scope for further productivity growth through the wider adoption of cutting-edge technology and further industry consolidation. Productivity growth provides a buffer to producers against rising input costs, but it does not guarantee rising incomes. So what has happened to farm cash incomes during this period of restructure? When I say farm cash income, I am not just talking about revenue. Farm cash income is a measure used by A-Bears of the cash funds generated by a farm for investment and personal consumption, after paying all production costs. The costs for production in this measure include interest payments, but exclude capital payments and payments to family labour. The first thing to notice is that average farm cash income has become a lot more variable since deregulation, as producers have become more directly affected by price variations on world markets. However, over the last 20 years, average farm cash income of dairy farms has had an average annual trend rate of growth of 1.2%. Two years ago, average farm cash income was at its highest in real terms for the last 20 years, but is projected to fall this year for the second successive year. This year it is forecast to fall by 28% to 113,000 per farm, which will put it very close to the trend growth path and slightly below the 10-year average to 2014-15. Last year, farm cash incomes fell because of falling farm gate prices in most regions and higher expenditure on feed grains. However, the effect of the falling prices on farm cash income was reduced by a higher production in most states, particularly Tasmania. This year, farm cash income comes are projected to fall because of falling farm gate prices in most southern regions and also lower production in most regions. There has also been higher expenditure on fodder, particularly in areas where there have been unfavourable seasonal conditions. However, while expenditure on fodder is expected to increase, expenditure on other inputs such as fertiliser and fuel are expected to fall. Farm cash incomes are projected to fall on average in Victoria, South Australia, New South Wales and Queensland, but increase marginally in Western Australia. But there will be regional variations in the falls because of regional differences in milk prices. For example, the projected fall in New South Wales is expected to be smaller than for Victoria and South Australia because dairy farms in northern New South Wales have received higher prices than those in the south and are projected to perform better financially. Going forward, Nirvana is not expected to be around the corner, but there are signs that give some confidence there may be some improvements over the next couple of years. First, I would like to discuss our outlook for world dairy prices. Next year, we expect to see a modest recovery in import demand from China and continuing demand growth in Southeast Asia, the Middle East and North Africa. This demand growth is forecast to lift world prices of dairy products over the next couple of years. However, higher production in major exporting countries is expected to lift world supply of dairy products and limit the extent of the forecast price increases. Over the medium term, we're expecting demand growth to continue in Asia, the Middle East and North Africa. In these regions, rising incomes, higher populations and more consumption of Western-style foods are all expected to contribute to the higher demand. Additionally, there is expected to be limited capacity for production to increase in many of these regions, which is expected to increase their import demand. While demand is projected to grow over the outlook period, we think production will also grow. And as a result, prices are projected to fall modestly in real terms over the last couple of years of the outlook period. This will see prices at the end of the outlook period average between 10 and 20 per cent above what we think these will average this year in real terms. Next, I will discuss import demand from China and the Russian Federation in a little bit more detail and a risk factor to these price forecasts. Last year, there was a significant drop-off in Chinese demand for dairy imports following a build-up of Chinese stocks over the previous couple of years. This fall in demand was a significant factor in the fall in world prices that occurred. Chinese stocks are now falling, but import demand from China is expected to remain subdued for the first part of this year. In the second part of this year, import demand from China is forecast to strengthen with low farm gate prices expected to constrain its domestic production and as its stocks continue to fall. Over the medium term, Chinese import demand is projected to grow, but at a slower rate than over the five years to 2014. This is because milk production in China is projected to increase as a result of improved herd genetics and pasture management and productivity enhancing investment in larger dairy farms. A significant fall in import demand from the Russian Federation was also a significant driver in the fall in world prices last year. This was due to a poorly performing Russian economy that resulted in falling incomes and a lower ruble. On top of this, the government of the Russian Federation banned the import of dairy products from some of the world's largest dairy exporters. Import demand from the Russian Federation is not expected to pick up significantly over the next year, largely because the Russian economy is assumed to remain weak. Over the medium term, the Russian Federation is projected to remain a significant importer of cheese and butter with domestic consumption projected to grow faster than its production. The present trade embargo has been extended officially until August this year, beyond which it is uncertain whether or not it will end. We have assumed that it will end in August, but if it does not, such an outcome could put at risk our price forecasts over the outlet period. What does this mean for Australian dairy? I mentioned earlier how the average farm gate price of milk has fallen this year. Next year, the prices forecast to rise by 3%, which is expected to result from high world prices for dairy products. Over the medium term, we think the price will increase moderately in real terms for a couple of years before easing in the latter part of the outlet period, when world supply is projected to increase and world prices are expected to fall in real terms. Nevertheless, milk prices are still projected to average about 6% higher at the end of the outlet period than this year in real terms. Over the next five years, Australian milk production is projected to grow steadily to around 10 billion litres at the end of the outlet period. The projected increase in milk production is expected to come from an increasing average milk yield per cow and a modest increase in cow numbers. There has been a long-term trend in rising milk yields, albeit with some short-term deviations from this trend. This has resulted from improved genetics and technology, better pasture management and increased use of forage crops and feed grains. We are expecting these influences to continue and the average milk yield per cow to increase over the outlet period, albeit less quickly than during the two decades to 2010-11. Measured by a value around 45% of our milk production is exported in the form of dairy products. So what is expected to happen to dairy products is important to Australian dairy. With higher production and growing world demand, the volume of dairy product exports is projected to increase over the outlet period for all the major dairy product groups. The value of exports is forecast to rise over the next couple of years because of higher export volumes and higher world prices. But over the last two years of the outlet period, the value of exports is projected to fall when world prices are expected to fall modestly. Most of our dairy exports are projected to continue going to Asia. With the projected demand growth in this region and trade agreements signed with Asian countries, it is expected Asia will continue to provide opportunities for Australian dairy to prosper into the future. In conclusion, I would like to summarise the main messages in my presentation. First, two years ago, average farm cash income from dairy farms was at the highest had been for the last 20 years. And although it is projected to fall this year, it and dairy farm productivity have been trending up over the last couple of decades as the industry has undergone major restructuring. Second, world dairy prices, the Australian farm gate milk price and the value of dairy exports are expected to increase over the next couple of years and finish the outlet period higher than this year. Additionally, Australian milk production is projected to rise modestly in each year. Third, growing demand in Asia and the recently signed trade agreements are expected to offer opportunities to Australian dairy to prosper in the years ahead. Thank you.


So hi Let me start this with a question. So how many of you are know know about docker or kubernetes raise your hands many and How many of you are using it in production make a spoke? spoke So So let me first introduce Quint Lane We are technological company We are trading stocks on different stock exchanges around Europe We are based in Prague. We have a relatively small team of developers. There is 14 of us As I said, we are developing a trading platform, which we are also using. We are not selling it outside And we use mostly open source projects to run our infrastructure and Even our most of our code uses open source project projects Outreading platform is written in Python obviously it uses React.js for front-end In our applications even in our trading platforms, we heavily used async.io because There's a lot of data and we want to process it in parallel Concurrently excuse me And we are storing data in Redis and timescale DB Also, we are using third-party libraries Integrating it with integrating them with site and Those libraries are written in C C++ Java or even Scala So site and right So There are lots. There are a lot of data. So we have to split our application into multiple processes and also Applications around which help us Like there's reporting some graph graphs and Other tools we use Also, we are using messaging to integrate all these applications together and pass data from one to another. When I first started working for Quantlane That was kind of a chaos All the applications were deployed on physical servers Managed by Circus, CircusD. It is process management system similar to SupervisorD, if you know Packages were installed in virtual end So each application had a separate virtual end with installed pre-installed packages Probably the standard and Everything run on the single user Most of this Also this error that it was simple It was simple in a manner that when a new user came to the project all he had to do was Clone project, create virtual end, install packages and he could run the application. It was the same for the deployment There were some disadvantages like there was some package version in hell, which means that We had different packages and whenever we updated one of our own packages It may use some newer third-party libraries and we had to somehow Migrate those changes into our other applications And there was no failover because when the server died everything on it died and the record was not automatically migrated somewhere else So what happened next was docker when I started we are already looking into the use of docker as it was Yes, it has a promising features like it was able to unify in our environment So that we could run the local development with the same kind of package. Let's say image As it was in staging CI and the any production The deployment was also simple because all you had to do was build any build the image just run one command and you had probably running application migrations were also simplified As The image contain everything you don't you don't have to Install anything else just pull the image and you had it It's speed up our CI No matter that image When we build them build and the image is built it contains everything so we can use this image through all the Through all the other stages of CI so and we had atomic releases as the built image has some tag and Even some hash so we can we have image with We have image with a hash which is atomic and unique in our registry So there are some there were some challenges we had to overcome first to introduce docker to our infrastructure and Those were like how do we store the images for this case? We decided to use github registry as we already had github instance in our infrastructure and it had this feature Next thing was image caching because It's kind of sad, but we have a pretty slow uplink to the internet like our internal network is fast It is it has a gigabit but uplink is around 20 megabits. Maybe so so as I said, we are using github and github has CI which has steps defined in the gith repository so anybody who has access to get a good repository can update the pipeline definition and Motivated to whatever he wants As we wanted to have a built stage in CI Like the simple CI build is this you just want docker build and it should build the image and Prepare everything but the user has access to CI definition he can update this and modify it maybe to something like this And by this he can effectively get access to the server on which The docker demon is running this means that You should have dedicated building environment which you can just Take and throw out replace it with a new one clean So whenever Somebody get access to this environment and does something harmful to it you can just clean it and Go without any problem for that Next thing was CI pipeline design Because you want to have fast CI you don't want to spend 20 minutes on Building and then testing and then maybe integration tests publishing Whatever and the last thing was cleaning cleaning of all the images The standard docker registry which you can download from docker hub Does not have Automatic cleanup of all the images and because our images have around 50 500 megabytes and we have like hundreds of them We have we had to implement some kind of cleanup of these images We are not we are not running in AVS in we are not only running in cloud So we have we don't have infinite storage so What docker brought us? Was as I said unified and stable environment by means that we had the same image for local development CI staging and then production and Everything was baked in so developer When he built them the image he could be sure that the packages which are in it and All the images I don't mean know just those which are specified in requirements but also the third party requirements and The full chain will have the specified version and this will be same in CI and staging and production Be because of the docker nature when you build an image it has everything packed in so Basic of this is to create one image bake everything in all the requirements all the development requirements application some environment specifics you need and You can run everything in this image as I will show you on the next slide It also brought us isolated environments by means that When the application runs it cannot access other applications running on the system It cannot take control over the other processes which are running there That's probably as some kind of security feature so fastest CI and This is our spice. This is our pipeline So first we built them the image as I said it contains everything and it contains all the packages application and some environment definitions Then next we run tests co-quality unit tests packaging Those are run inside this image in parallel So each of those tests may run like maybe two minutes and this speeds up entire process because in those state in this that stage none of the jobs have to install the packages which was the bottleneck of our CI so next we optionally deploy Next we optionally release a bleeding edge version and deploy to staging and then we run integration tests and publish the documentation know that bleeding edge release and state deployment or staging are optional So we can run integration test Immediately after the unit tests are complete so a docker has also some kind of Looker has also these advantages like there are known bugs And every day you can find a new one For example, there are memory leaks There are some race conditions which lead to deadlocks It has no fail over if you don't use docker swarm. I don't know what kind of state docker swarm right now is but Also when the looker demon dies you cannot Manipulate the running containers your containers may still be running, but you cannot stop them or restart them Or create new ones There are few Gotchas We found out when we started using docker and for example, there's a PID one pitfall who knows about PID one pitfall Yeah Something you don't know So PID one pitfall is basically a problem Or maybe a feature let's say So when when you run an application in docker, it is started with PID one PID one has a special meaning in linux because it is an edit process which starts everything like SSH your UI Whatever PID one doesn't inherit default default signal handlers which means that you have to implement them by by yourself Who knows about signal handlers in linux? Okay, so So you have to implement those signal handlers because You you usually want a graceful graceful shutdown of your application So when you run docker command the process is started with PID one and When you run docker stop on it, which should Terminate the process The first what docker do is it sends sick term to it If the application doesn't shut down in 10 seconds after this it will send sick kill Effectively killing everything you may lose your state with this So it's good idea to implement the signal handlers. This also applies for a process which doesn't run outside of Which runs outside of python outside of docker? but Also, it has one other meaning that you have to Like the when a process runs in docker you have to take care of the sub processes you run So if you are using sub process Module and running other processes such our process says you have to terminate them and Clean up after them because if you don't they will remain there and The crew will probably somehow take care of them or cannot will kill them at at the end so also we have to take in mind the user within and within the container because When somebody get access to your container and Runs a shell in it Let's say some kind of attacker so The one can get the root access if you run your applications your application on the root Which means that he can modify the file system within it even run other applications And you should avoid this you should avoid this because you don't want it to modify the Modified the entire container he can even run some kind of spambles You don't want that you After we migrated to docker it took around maybe a month We started looking where to move next and we found Kubernetes It has some kind of Maybe I didn't like docker has a whale Kubernetes has a wheel. I Don't know so What Kubernetes is is basically a cluster orchestration. This means that You have a bunch of servers you install Kubernetes on them Kubernetes is somehow manages all of them and you just tell Kubernetes these to run The application somewhere in the class that you don't care what where you just want to have it up running and maybe Accessible on some kind of address What Kubernetes was interesting in what Kubernetes Kubernetes was is interesting for us because It's sold some kind of fail over when the server failed So when a server in Kubernetes dies it Migrates the workload from the that server somewhere else So you don't have to care about it and you have you can sleep at 3 a.m.. It won't wake you up the configuration can be stored in namespaces namespaces are Logical dividers you can have namespace for production for staging given for different applications maybe Namespace for monitoring blogging even your applications and For each of these namespaces you can have stored a global configuration Which can be then propagated? From this namespace to those services running in in this namespace Also if suppers some kind of basic service discovery you can access other services based on DNS Like my service my namespace service cluster local is the standard address Just have to fill in this my service my service and my namespace It has an ingress controller It supports an ingress controller ingress It's a way to expose the services applications running in Kubernetes to outside world so the outside world can make requests and For example retrieve some kind of website which is running in Kubernetes The other way around like the services between the Kubernetes can still access the outside world but for that outside world to access the Kubernetes services you have to have an ingress controller also Kubernetes has one fancy feature and that's deployment history which means that When you deploy some kind of new service and it doesn't behave you want to Revert it to previous version. So you can call kubectl rollout undo and this will Deploy the previous version. So it's a really handsome utility when you want to simply Revert revert something and you don't know which version version it was running before Right now we are in the Process of migration to Kubernetes our main trading platform was already migrated a week ago There are still other services which are running running still in docker on other other hosts But we are planning to migrate them in maybe two weeks and then we can Join all of the other servers into the Kubernetes cluster we have in the environment Configured by namespace variables So we have production namespace in which we have configuration For Which specifies where you can find services like the messaging some kind of data storage access to databases and so on and We are deploying our services using Plane almost with ginger to Like ginger to files containing gammas with our variables this allows us to have some conditionals in templates and have a single deployment file which is adaptable for different Processes or maybe profiles configurations and so in this example you can see that we have some profile specified and If the profile is C2 loo the other there are other environment variables added to the deployment So no some notable features about Kubernetes are probes imagine that you have web application and Deploy it to Kubernetes. You want to check that it is a running and if it starts like maybe 30 seconds is You just don't want to care about it. You just deploy it and want to see it running as fast as possible so probes what probes does do is that they check if application is running by Accessing the some kind of port you specify or running Some internal command in within the container This allows cubanities to check if the service is running and if it's healthy if it's not cubanities We are automatically restarted And then there are updates strategies There are two major one one is rolling up update What this strategy does is that when you deploy a new version the old version is still running and new new version new deployment with new version is started and unless the deployment with new version is running available and stable the Previous version is the previous the previous deployment with old version is still running So when the second version the new one is available the first one gets shut terminated it's shut down and all the Traffic is forward forwarded to the new deployment Another update strategy is to recreate What it does is that it first shadows the previous version then starts the new and You can have some non-zero that non-zero the whole time this update strategy is good when you have Some kind of resource which has a unique log maybe You can have some data stored in files and you don't want two applications to access the file simultaneously so those were the fancy features or interesting features of cubanities and as I mentioned doctor has whale cubanities has Wheel so beyond that There are many wheels. So we are looking into the cubanities Federation next and that's beyond So thank you Thank you, Peter, please raise your hand if you have any questions Artis ginga templates integrated somehow with cubanities conflicts and secrets for deployment Like you can send only pure yaml files to cubanities and jason files But you can you can't send the yaml ginga do files. So you first have to fill in the variables then you can send it and The way our deployment is designed that we take the Ginger file and fill and fill in everything we have so we have also secrets and config maps within the deployment Can I have one more? Which deployment strategy should I use if I have database migrations between versions I I should use this recreate Strategy or there are some others better solutions if you don't have any Any way to fix Like to take care of the difference between those two versions you have you should use the recreate but Maybe there's a way. I don't know. How do you monitor all this? That's an excellent question. We are monitoring internal bots. We're using hipster and we are monitoring entire cluster by Prometheus So previously you mentioned the fact that the problem of the P1 within Docker. How that's Kubernetes helped you Kubernetes doesn't solve that you have to solve this by yourself if within your application Okay, I've been the assault and if so, how? We are solving this by registering signal handler, which As I mentioned, we are using gacing IO So what do we basically do we just stop the loop which? terminates all the all the running futures I think it's called futures and Then we just have a finally block where we just closed all the handles states states save the state and Clean up everything we need But also you can add the signal handler to loop directly hi Do you have some persistent data on hard drive and how do you do you manage this data between all the service? Yeah, we had we had a persistence on file system We are right now migrating it to Redis, but Until now what we had to do when we migrated between Hosts was that we had to shut down the service migrate the data and start the service again. We have no shared storage We have time for last question At the beginning of the talk you said that you were running the problems with github runner How did you solve the problem with for the clean environment for each pipeline? We have a dedicated Dint like docker-in-docker so service running which which handles the Which handles the pipeline but so we wait we have we have shared we have shared shared dint but When we are shutting down it sometimes so we clean it So basically when that gets to the end it doesn't affect the entire host on which everything else is running so Actually, we have the same problem and the problem with deans is that you need to have a privileged container to run the docker-in-docker and Essentially, you have a route if you're if you do that So I wondered if you Had another solution Maybe we didn't solve this the right way Thank you. Yeah, we are women. We are planning to migrate it to different hosts to separate hosts. So maybe this was so this Okay, fantastic. Thank you Peter. Let's thanks


🗗 ṏ ṏ Ṏ ṏ 🐓 Ṏ ṏ ṏ ṏ ṯ 🐝 ṓ Ṋ 🐚origa the Vedic My god did not do all these things as I told you guys do because your religion changed enclosure hunger斐 police ?! ohh wo


Steen station officer Greg Chapman from Helen fire station received a call the night to initially to a rubbish fire Arrival we found the house well alight all occupants were accounted for we've had a number of Exposures including bees Asbestos a large number of gas cylinders as well as the high winds which were pushing the fire towards the property next door We've been able to save the exposures Most of the house 80% of the house in the original fire has been destroyed The occupant required attention by emerging medical responders from the fire brigade and ambulance Victoria He started to show signs of a heart attack, but has now been cleared by ambulance Victoria


Soul and Patreon was made. It was dedicated to a guest DJ right here. DJ Mimi, since we DJ Khalifa knows nothing about soccer and football, we had to bring a specialist DJ Mimi herself because we are going to be talking about Afghan and we are talking about publicity as well. So talk to your friends. Good morning to everybody. My name is DJ Mimi. I can be found on all social media platforms on at DJ Mimi 254. The Melanin Queen. All right, are you going to play Melanin for us today? My chance? Hopefully. All right. What do you have lined up for us? Everything nice, good and easy. Everything nice, good and easy. What about the weekend plans? Do you have a weekend plan for us? I'm not playing anywhere this weekend. I'm just going to be supporting friends. You're going to be supporting friends? Yeah. Friends are you supporting? Name one best friend that you want to be. DJ friends. Just know that they're DJ friends. I don't want to put names out there so that they can't hold it against me. One will be like, why didn't you say me? All right. So do you think Kenya is going to beat Algeria? Okay. Soccer is not my comfort zone. Like me DJ Khalifa, but DJ, they only care about the music. So chances are tonight you're going to be watching the game with the music in the background. Commentary as you talk about. Yes. So I go with the name of Baymo. So it's Baymo and we are back with this edition of publicity. I don't know how to put it, publicity. It's a problem, Joy, for real.  So you guys were back with our guests. Let's say we got me right here and don't want to say me, I don't mean me. We've got a guest called me and another one by her name. Her name is Bilha. And so what we discussed before was kind of what the industry is all about. And we want to touch a little bit of what the Kenyan industry is about. But before I do that, I wanted to kind of ask me quickly to talk about something that she pointed out that was quite important, which was the cinema work that you're publicizing for as well. Yes. Yes. I'm currently working with Anga IMAX Cinemas. Anga IMAX Cinemas. I believe we all know IMAX are 20th century. It's I think the oldest cinema in Kenya, but we have new branches as well. And what you're trying to do right now, as of now, is bringing back the cinema culture, because it kind of went down Kedogo. And we went and the youth went out to my drinks, and that kind of thing. It sees being like movie dates, movie dates, and after work, movie, catching a movie with colleagues. We kind of forgot that, but that is what we are. So nice and harmless. So nice. Yeah. So that's what we are trying to do, like bringing that positive entertainment back. Like you don't have to go to some other places that are not very good. Yeah. You can still have fun and still be positive. I see. Yeah. Do you think these streaming sites are to blame? The ones we stream on and chill, you know, the, the movie streaming sites that we stream on and chill, that thing and chill in. Okay. The thing that we do while chilling. Yeah. Do you think there are to blame for killing the movie, the theater culture, right? Well, what I'll say is a movie, a cinema, someone who has experienced the cinema cannot be swayed because it's totally different. The experience is different. Yeah. The experience is different. Like, I mean, walking in, you know, getting your popcorn, your, your hot dog, so that then going into a totally dark room, sitting down with other people that you might not be knowing and catching a movie like the latest, you know, what's happening in 3D, because that thing and chilling, I don't think it chimes in 3D, does it? That thing, that thing. That thing. Okay, okay. It doesn't. Yeah. Well, thank you so much for sharing what you've been doing with the IMAX cinema. Could we maybe just touch a little bit about the Kenyan market or what we're calling the hashtag KE, you know, music where there was a problem recently where Kenyan artists were complaining, ah, you guys ain't playing us on air. Man, you guys always playing Nigerians in Tanzanian, bongo music and whatever kind of stuff. Even more regular than you do play all kind of music. And you know, as publicists and as PR, how do you feel about such statements? And where can we change, according to your view, where can we change that kind of sentiment? Well, in my perspective, I think it involves everyone, you know, from the artists to the presenters, to the DJs, like everyone has to do, like DJ Khalifa said, DJs, if the music is good, they'll play it, right? So it's the artists to ensure the music is good quality, it's playable, you know. And then after that, don't just produce music and just have it on YouTube and chill. No, that's where we come in. That's where you come in? Yeah, that's where we come in. Your job is to produce the music, to be creative. And then from there, you give us the job and we reach out now to the radios, to the TVs, to everywhere that we would want. Yes, and then now the reception from the DJs and the artists should be positive. Like they should also understand that Kenyans don't just want to listen to that music.  Yeah, so not to be biased, but then again, not to play music just because it's Kenyan. Yeah, we don't want sympathy. Yes, so there's no, right, Lumis Khevo, it's not about playing, we're not going to play music just to be sympathetic. Your stuff has to be good. Yes, so you're saying that what needs to happen is that people need to do all sorts of things. First of all, artists need to step up their game and then PR, they need to reach out to PR to come in and kind of push their music. Is that how you feel, Bilha, about the Kenyan market and music? I mean, I know a lot of Kenyan artists that are really putting in work. I know that one of the biggest hurdles for Kenyan young Kenyan artists is obviously knowing where to put it and stuff like that, and I agree with how that's where we come in, but I also very much strongly encourage artists that do not have representation, do not know how to access us to really do their best. I mean, the one thing I like about gospel artists is that they do it. They can work with their CDs in their bag, station to station. You know, sometimes they see the receptionist and it's called CD Aki Chupa, but Kesha Nakuja, Naotena, and that's one of the beauties. That's why gospel music is heavily local. They've really pushed for local in that section. So I wish that secular artists would pick up the aggressiveness that gospel artists have on that, but I do agree with how we don't want pity. At the same time, we want people to look at content and be like, that's content I want to bang for anybody. Like if Jay-Z came here and said B, which Kenyan artists are you rocking to? And I'd be like, is this kid called Kevin Grant? You know, play here. Or like Zero Sufuri, something, you know, something I believe in. I don't want to just push it because it's Kenyan. And that's kind of the same scenario we have with AFKON. We're not sure like to support AFKON, Kenyan just because it's Kenya or like it's good football. I know where the problem comes when it comes to AFKON. People are going to be placing money somewhere and it's a different bargain. But when it comes to Kenyan music, I was talking to different people because sometimes I travel across the country a lot and they tell me over here we listen to Kenyan music, their local music. If you go to Kisumu, they listen to their local artists. If you go to Kisi, they listen to their local Kisi artists. If you go to Mombasa, they listen to their local Mombasa artists. The problem, I think, maybe was with the urban music, you think? For sure. 100% the people that are complaining are urban artists. If you go to River Road, they're making money, they're building mansions, buying, you know, cars, left, right and center for rent. The number one Kenyan artist is Paul Moi. He's the only Kenyan artist who sold 100,000 CDs, physical copies. It went gold. It never crossed my mind. Precisely. Ben Githai makes a lot of money. Gospel comes second, but when it comes to urban gospel, it's trailing way behind secular. Urban artists are obviously the ones complaining because, again, it comes to design thinking when you're making music, what you're making your music for. You have to think about your consumer ahead. What are their numbers? They know who they're making the music for and those kids are behind them 100%. And there's so many. And there's so many. And they're coming up from nowhere and even cookies are filling. Oh my God, I'm the one who doesn't know this reggae stuff. I need to get on top of it. I need to learn the newest slang because there's so many you feel like you're the one outnumbered. The problem with urban music is that it's very much a niche product. So when you have a niche product, the problem is you're marketing it also in the wrong place. If you're a hip-hop artist, you're not supposed to just market it in Kenya because your consumer marketing Kenya is probably like 10%. You're supposed to market it in the continent. So you take 10% of here, 10% of Tanzania, 10%. The artists that understand this are like the Jack Roosters and the Blinky Bills of Kenya. They know that their music is not popular here, but they have some market here. So they play here, but they are constantly traveling where their market is. So that's one of the problems. Obviously, Tony Nhaduna is making a lot of money and he's not just making a lot of money. MJ, our very own MJ. And he's traveling the world as well, selling that because it's also an exportable product. When you're rapping and you're competing with Chris Brown and Jay-Z, like a Kenyan rapper because that's what we are competing with. What are you offering that's different? With reckless, I can say what is offering that's different is his slang. It's Kenyan swagger. What are you offering? You're coming with Jay-Z pants, but you're not a Jay-Z background. But I think that you have to have an X factor. Yes, X factor is the rate.  Tim is not on our side, but before we end this, I'd like to talk about what are some of the publicity campaigns or that you saw in the recent past that really stood out for you? For me, so I can mention local and international. This year, I like the Wakadinali Kovu challenge. If you go to Skawakadinali's IG page, you'll see lots of Kenyan rappers. I didn't even know there were so many female rappers. There are so many. I mean, there's so many on his IG. So that was a very good campaign from the Wakadinali team. Something to learn from him. For sure. And the fact that they were doing it on low budget, zero budget, they just released a song and put it on YouTube and said, yo, who wants to rap? And again, because the Wakadinali nobody thought they would come on board and then Calligraph joined and then Kashi joined and then Kevin and then everybody's on top of that. Then PlayKE this year was dope, obviously, but he needs to continue. Last year, I made Kenya Challenge is the one that took home for me and obviously internationally. It's the Thanksgiving campaign that happened in America, in conjunction with YouTube to the Puerto Rico country, where basically they donated what they did is like, Puerto Rican artists decided to donate money for their YouTube views that day to charities of their choice. So if you went to look for Jay Balvin that day, it was support domestic work. I'll definitely look into that. Let's get some from me maybe as we proceed. Yes, for me, what stood out was the Chukua selfie. I really liked it because it had like an educate. It was an entertainment. It's powerful. Yeah, yeah, but I don't know where it went. The one that I learned from is the Hooduma number because I learned like what not to do, like don't force something, don't force a campaign on people. You should let people like what a publicist should do is put out a good image for what they're pushing, and then people will definitely jump on it. So publicity was lacking in the Hooduma number campaign. Yes, yes, yes. And it was like Chukua ama, you know, like it's a threat. Yes, that was lacking and from that I learned what not to do. So at least we got positive ones and we got negative ones that we can learn from. Let's listen to the DJ. Do you have a publicist? Yes, I do have one. He works for MediaMax. He's more of a publicist, I think he's more of a PR. So he's like, yeah, like promotion and everything. Is it working for you so far? Do you feel like it has given your career a push? Yes, it has, because this is somebody who believed in me when I was starting a big gym for like six years right now. Then I took a break in 2017 and anybody who is in hobbies knows what a break can do to somebody's career.  It's not that easy and it takes a lot of efforts from everybody who is supporting you. So this person has been with me since I started and even when I decided to come back, he was still willing to be here with me. So traditionally publicity works. Yes, it's time for the moment of truth now and it's time for the game. So Joy, should we do this? We should do this, we need to do this. We need to do this. Yeah, we got to do this. Thank God it's ready. This is what social pride is all about you guys. The reason why we do call it social, we don't just sit here and socialize, we actually socialize. We play music, we talk, we play games, only a couple other things missing. And most importantly, we bring social media to the television and yes, today we have the Afghan challenge and a reminder we are going to be a TRM today. Just activating the fact that we are going to be live or Afghan is going to be live on channel one KBC channel one and the hashtag is Afghan ECO KBC. Don't miss a match and please tell us who you're supporting Kenya versus Algeria and where your money is at. So the game is quite simple. So we are going to be putting pictures on the screen, jerseys of teams that are going to be at the Afghan and your job is simple. Just tell us which jersey belongs to which team. Is you ready? We'll try. We'll try. All right. And you better miss the Kenyan jersey, please. You better miss. All right. So this is the first one. I'm already lost. I'll try, send it off. Pass. All right. That was simple. It's Angola. All right. So zero, zero. We have zero, zero. Let's get to the next one. This one is tough for sure. But just try. We have 50 something countries in Africa. Just pick one. I wish they could draw a flag on the jersey somewhere. This is a hard game for ladies, man. No, I just I'm looking for a clue in the paper. Just pick one country from out of the 54. It's West Africa for real. For sure. Algeria. West Africa. West Africa. West Africa is a country. No, a country in West Africa. I've learned one word for real. You know. Oh, you good trial? Yes, my name is. All right. Is that because they have raster colors on the jersey? All right. So one more. We are. We are. All right. Let's move on to the next. Ah, this is one, one, one, one, one. What's your answer? All right. All right. Let's move on to the next. DRC. Oh, sorry. Am I supposed to play? Sorry. Yes, dear. All right. That is a spoiled vote. Let's move on to the next one. I removed myself from the game. It's you, too. Yeah, yeah, yeah.  All right. Just sold the bikes to you. You're just sold the bikes to you. Like this.  Uh-huh. But we are looking for the country name. All right. It's simple. This is also. This is Gini. So it's, it's, it's doing from here. You can see the flags. Yes. That's why we had to hide it. All right. Hmm. This is the big, the type breaker now. Kenya. I wish you resisted. What about what you want to answer? Kenya. What is the name of the player? This is a type breaker now. Here it is. So, what is the first name? What? Yes. So three, one is, three, one is the score. Like, do we have another jersey to show? This is a simple one. Just look at the faces and the size of the bodies. Algeria. You, you're really insistent on Algeria. Algerians are like Arabs, by the way. Arabs who come from Asia. I'm an Egypt. No. But you said look at the faces. So it's probably with African Congo. I'm just throwing out names now. Mali. It starts with an M. It's Mauritania. You never imagined. So how much is the score right now? It's three one. Who's three? She's a three. But at least you got Wanyama. At least you got Wanyama. Do we have another one? All right. Can do this. Hell no. It's a southern African country. Hint. Me? What's your answer? Wow.  No. Maybe I'm not going to give you that one. The score line is three two. We're not going to give you that one. Do we have another one? Hey, he moved. Hey, look at those thighs. Hey, my friend. If you're hit with those thighs. Let's see. Let's see. I know this color. Agana. What's your answer? The DJ got it. It's Uganda. Yeah, it's true. Actually the colors. Yes. So Uganda. So it's three two one. But she hasn't been played. Producevating. So do we have another one? This is our last one. So if you get this one, it's a draw. If she gets it, it's a... Yeah, so you guys make it good. Make it good. There's a time these guys were carrying money in Kuala Lumpur. She said... She was not going to say it, but she'll say Zimbabwe. So I think it's a three three. So maybe you can shake hands. Just not I said Kenya. Me is the winner. Loki. Loki means the winner. But it's a three three. So we've come to the end of this. All right. So it's about that time. What is that one advice?  Use digital to your advantage. Use digital to your advantage. Distribute on digital for free. Put it on YouTube. Promote on Facebook. It's like 2000 a week for a budget. Like don't just sit at home and say I don't have money to pay Bilha. I don't know where to find... Mimi or DJ Mimi or me. Yeah, I'll get one, two, five, four. Promote it and then eventually they'll get two minutes time. Then they'll be like, okay, let this all send me the three days. All right. So promote on Facebook and IG. Promote. Use digital platforms to your advantage. Knowledge. Okay, I'll give a quote that I'll leave with. Doing business or building a brand with no publicity is like a man winking at a lady in the dark. You're the only one who knows what you're doing. Nobody else. So, yeah, always have that at the back of your mind. My example. Promote to my face. What she said. Oh, hey. Hey. In the south, here too. All right. Have you enjoyed any other... For me. As we wrap this up.  All I can say is the best thing to do is always listen to the pros in your industry. If you've got people in your industry that are good at what they're doing, always learn from them. And like she said, don't try and do things by yourself. You're the one who's going to know what you're doing. As guys, you're just going to be like, so who's that again? And you'll be there like, I've been toiling for years. I've been out here. I've been doing music for 10 years. Right. So that's what I'm going to say. It's been a blast actually playing the game for AFKON Challenge. You guys, yeah, you saw how AppoMotion machined what Kiddler was about. Football is not exactly my strength, not exactly my forte, but you know, it's still an amazing game. We still stand to learn a little bit from Kenabari who are shoving it down our throats, my forte. But it's all right. You have to support your country. Are you guys for short, by the way? Support Kenya national team, not only KE music. Me, I support KE music and our national team. All right. Yes. I have to check the lineup. But mine is a sympathy one. Stand and guide me. No, because I'm here for patriotism and with light. Because it's a good defense, but who am I lying to? All right. So there's a question coming in. My director tells me, let me just find it. So the question is to our guests, why are our artists not like participating in these things like the AFKON making anthems for our teams? We have comedians like Eric O'Mondi coming out to do the job of the musicians. Well, to be honest, we're very, we're not united necessarily. I was thinking about this the other day when Agor Maya were playing, I think there's a kafa, and then they were like in the airport somewhere sleeping and no one in the creative industry was talking about this. We're not necessarily united. We almost feel like everyone's plight is everyone's plight. Like we have our own problems as artists, players of their own problems, film industry has their own problems. I mean, we need to be more unified. And secondly, obviously, it's because the teams have neglected to do their job. People are not being creative. People are not looking around to be relevant, which is a huge problem. Yeah. And it's obviously trying to be relevant. All right. Me, what do you say about this? I think I'll agree with what she's saying. And also, like maybe they already had their strategies in place. Like, yeah, they knew what they were doing and AFKON was not one of them. Yeah. So they're sticking to what they plan to do. All right. So what you're not saying is instead of somebody looking for a wife, I'm going to invite you to AFKON. That's your video, yeah? We can put it like that as well. All right. Anyway, thank you very much for the insight. We appreciate it so much. And a quick reminder, we are going to be at TRM. The whole Y-to-Y-4 team is going to be at TRM, activating AFKON ECO KBC Channel 1, our mother channel. The alternative games are going to be on Y-to-Y-4. AFKON ECO KBC is a hashtag. That's right, you guys. So do stay tuned in. And I think it's Val, who's going to be with us over there. Yes, Val is on the ground right now. Right, Val is on the ground right now. So you guys, do stay tuned in. All your football heads and all you people who want to learn like me, do stay tuned in. My name is German Shacha once again. I think we have come to an end of Y in the morning. No, no, an end of this particular segment. There's still that Kalei football. Val is asked to tell us something from the ground as well. She has to tell us something. And then we shall maybe come back and say our goodbyes to you guys. But until then, my name is German Shacha. You can find me on join.com. Shacha and all social media platforms. All right. And the fans are saying, how can they get a hold of you as well? You didn't give them your social media handles. So maybe me, you'll go first since you have the microphone. All social media handles, me, Mwangi. By the way, you guys, me is M.I.H. Yeah. Send a message. See you later. See you. Yeah. Please. Please. On IG, I'm Miss Billy254. On Facebook, I'm Billher W. Ngaruya. Billher W. W. Ngaruya, yes. All right. Thank you very much for sharing that with us. All right. What about you, my assistant, before you get to talking with your fingers? DJ Mimi to fight. DJ Mimi to fight. Mimi is in like, where? Mimi is in where? All right.  Then we'll be back with tomorrow morning. It's very more for me on every social media platform.