2-5 for my name is Max Oliver C.K. This time round we're going to speak about the state of boxing in the country with a man who was a team manager for the Heaps Code in Tokyo Olympics. Of course they just concluded championship and he's also the public relations officer for Boxing Federation of Kenya. Shugare himself, an international boxer, good to see you Shugare. Thank you ma. You were telling me that you were talking about side shows. Instead of talking about your role. Not really. And of course these are up air shinani guns. Of course when you are up air we are talking with Shugare about a lot of things. You know side shows that happened in Tokyo, Japan. They are staying at the village. How is it like considering it's COVID-19 pandemic no mingling, no interaction? I don't know when you come, when you hear those who have been to Olympics before speak and now you compare to what we experience in Tokyo. It's totally different because basically it's like we were somehow prisoners because we could not interact with the Japanese public. So we were in the village where we were about 11,000 of us from different nationalities. When you come out of the village you get into a bus taking you to a competition venue or practice venue and then after the competition or after the training session you board the bus back into the village and they are very serious security screening being done getting into the village and out. No especially getting in. So the accreditation card has got some backcodes whereby once if you get out of the village and maybe you go somewhere else not to the competition venue where you are supposed to go then you are going to be tracked using the accreditation card and when you are coming into the village you have to be screened so that you go and place the accreditation card on some machine and then it reflects your photo in the computer so that they are able to confirm whether you are rightfully supposed to be the one getting into the village using that particular accreditation. So they were very thorough in terms of security. There was not a lot of mingling. Every day we had to do kofi tests and I think there were very few positive cases within the village during the time you had the games. I think it was tough considering that especially when you go for the competition they are no fans it's mainly the boxers who are participating who are coming maybe to watch how the opponents are playing so that they are able to strategize for them going forward and it was not very lively in terms of lack of fans in the arena but I believe it was a wonderful experience for those who are able to get there and compete. All in all how was our performance? Our performance was not the best. I believe each and every one of the boxers gave their best. One case we were not very happy about was the one involving Niko Koth. Oh first match. And that was the first bout. Again a sub boxer from Mongolia. Yes I know that Mongolia is also very experienced considering somebody who was in Rio and got to the third stage at the Rio Olympic Games his three times Asian champion and so we knew it was somebody who was a bit experienced. I had done research on him and we gave a game plan to Nick which we were able to implement very effectively and Nick was able to neutralize the opponent very effectively such that during the first round the guy was not coming forward normally he comes forward attacking to the body but we told Nick he had to stay far wait for him to start and then he counters. First round the guy was not doing what he normally does so we had to change strategy in the second round which was again very effective. For first round Nick lost, second round he won, third round was the decider and we thought Nick dominated that round but to our surprise the decision was 3-2. The judges were split 3-2 in favor of Mongolia and that's how we lost that bout. Again unfortunately the organizers of the Olympic Game Boxing Tournament who is Sot Aiba who is an International Federation in Charge of Boxing it was IOC Boxing Task Force they had removed appeal mechanism so once that decision is entered you cannot appeal against it so it was very unfortunate and very frustrating for us because we thought that Nick had won that fight but everybody saw what happened and it really got into his mind and I remember I saw some interview he did and some video tape went viral him talking and apologizing to Kenyans how contrary to their expectations he didn't perform as they anticipated of him and I think people shored him with a lot of praise for being a great patriot and just for honorably accepting the defeat even it was not deserving. Ya he's our commander, he's the captain and he leads very well normally I don't know somehow it's a coincidence that when you go for a major tournament it's normally the first one to get into the ring and he opens for us very well we also did it very well unfortunately it also disappointed the other boxers who look up to him especially the next to get into the ring was Christine Ongare the following day and her she was affected so much because she was even speaking about it that if this can happen to Nick what will happen to me Yes but again even her she was competing again it's a very experienced boxer from Asia Philippines in particular I would say the person she was fighting again is normally competes at bantamuit level which is about 54 kgs and they were competing at flyweight which is for women is 51 kgs and for her she weighs about 48 kgs so in terms of weight she was disadvantaged in terms of physique she was disadvantaged so in terms of power again she was also disadvantaged but she did put up a good fight and again also she also she will say she got pressure in that our sports years came to watch her fight and she will say now she had the pressure of fighting this opponent and she knew she had biased officials and now she had to please our minister Yes but I thought it was a very good guest here for the CS to come and watch the fight where I was I waved to her she said hi it was good I will say we competed again in this very formidable opponents the draw was not favorable to us we got some very very tough fight because the next after Elizabeth after after Christine it was Elizabeth no it was Elia Joey Yes Hibwait and he was fighting again is Cuban four times world champion and Cubans are extremely talented in boxing Yes they are the super house they are the main drill deal in boxing do we need to do some benchmarking from them actually we really need to is there a partnership a collaboration system between Kenya and Cuba I understand I understand there was an MOU which was entered into an former CS HSA was a CS I think they went to Cuba and together with the president and I believe there was and there's a deal that was done but unfortunately has not been implemented I hope that one can be implemented because for Cuba Cubans are being so well they are the ones who won the tournament in in Tokyo they won four gold medals and in the first bout Hibwait category Julia De La Cruz from Cuba was meeting Elia Joey from Kenya it was a tough fight for us we knew it was tough because Julia De La Cruz we all know about him we know his tactics and I would say amongst the people that fought Julia De La Cruz in Tokyo Elia Joey could have fought him the best because he gave the strategy that we gave Elia Joey was not to go forward looking for De La Cruz because De La Cruz boxes with his hands down and he's very flexible and his legs are very fast so you will be cheated that you are seeing a target on the face and the time you are going to hit the face you know the punches are coming from when you are not seeing them because normally you expect the guard to be up so you are seeing these punches coming him the hands are down so you just here you just meet with punches Hajiwe Adi 412 and I believe he lost to Abeta Boxer generally what's your placement and overview about the competition is there a need for Kenyan Boxing Stakeholders to go to the drawing board so that in the next championship we perform better I believe we really need to go back to the drawing board because I can say in terms of officiating there's something missing there's a link we really need to try and get our officials to be at that level so that they can be also telling us what is it that these judges are looking for because sometimes we go for these major championships you don't agree with the kind of judging that's happening you wonder what is it that they are looking for because sometimes Aboxer is aggressive he's landing more punches yet when the decision comes out it's not what you expected again I've got one question to ask not only Kenya but Africa because you realize that the only medal we got I think was Ivory Coast or Ghana no Ghana Ghana we got a medal in boxing and that was the only medal we had in Africa what was our major undoing our major undoing I will say without linking my eyes COVID yes most of the Boxers who represented Africa at the Tokyo Olympic Games competed in the Olympic qualifiers which were in Senegal in the month of February 2020 yes and they never got into the ring until Tokyo Olympic Games there were no tournaments being held because of the COVID pandemic yes for us Kenya we were a bit lucky that we were able to participate in two international tournaments which was zone three boxing championship that was in the month of March in Kinshasa and then the next bout we were fortunate to participate in the Kurokov in Russia in the month of May those are two tournaments so by the time you are going for for the Olympic Games our Boxers the one who had most bouts from the time he qualified which was a almost a period of about two years was Niko Koth having participated in five bouts which is a bit inadequate because normally you know even our local league is not running so there are no local tournaments and there are no international tournaments to guide yourself so I would say in terms of preparation that's where most African countries missed a lot of opportunities to prepare their team adequately for participation at the Olympic Games and it showed in terms of the performance because you see Cubans Europeans performed a bit well when it came to the medal table I would say Covid really affected Africa so much because there were no tournaments for us to participate in and guide ourselves as part of preparation and just going to the gym and doing your normal training sparing it's not the same as competing you know in a tournament environment it's very different in a few maybe in the past when Kenya Kenya was a powerhouse in boxing we used to have so many international tournaments you could hear people talking about Bangkok Thailand it used to be that official name was Kings Cup in Thailand it was a big tournament we had Inter Cup in Germany we had Eastern Central we had Fescaba Africa Championship nowadays those tournaments are not there so we require to have a lot of exposure for our boxers to compete against some of these Europeans so that when you meet during a major tournament you have that experience of having competed against some of them and you are able now you improve and you are able now to gain confidence so that when you meet them maybe maybe now next major tournament to expect we have World Championship this year but I believe where our performance will be gauged very well will be in the Commonwealth Games in Birmingham because the Cubans will not be there Americans will not be there so we are going to maybe or even the Russians so we are going to be competing mostly African countries and the British Colonies and I believe there we are going to do much much better if we are able to get good preparation going forward and I believe we are more than capable of going out there and winning this medals we just need to correct a few things here and there and then now we also need to invest to invest so that boxers such as Niko Kout, Elia Joy who are aging yes and are just about to retire we need to have young guys coming up to replace them so that these young guys we can also give them exposure and when it comes to this major tournament they can be able to go out there and compete effectively to replace the ones who are aging considering that boxing was one of the discipline that is not usually on the traditional Kenya's medal table or Olympic discipline which we usually go for because we go for athletics majorly and boxing is one of those that makes to the Olympics which is not our traditional mainstay and we have had that drought for I think 33 years without a medal in the boxing category which was the major lesson you took from Tokyo going forward for you now I want to correct you first boxing is the only other sport other than athletics that has won Olympic medal in Kenya and we have won a total of seven medals including Robert Wangila's gold medal in 1988 and we have won best boxer at the Olympic level that is 1968 Philip Huarwinga he won a bronze medal in Mexico 1968 and won Valbeca trophy for best boxer so and he also won silver medal in 1972 Munich Olympic Games so we have a number of medals we have won so it's not right to say that we are not in that basket we are more hopeful when we are going to compete at the Olympic level than some of the teams that represent the country but I believe it's a high time we also try and win medals because like we rightly said it's been a while from 1988 that's when we won our last Olympic medal and in that games we won gold for Wangila and bronze for Chris Sunday so I believe it's a high time also we try and prepare and see if we can get to that level where we can produce young boxers who can compete at that level and be able to win medals we noted that most of our opens are much much younger we had two boxers who are about 38 years and they are competing with boxers who are about early twenties and it's tough when you are boxing again and those young guys they have got their strength it's not easy and probably just like what we see also in football it's not easy for somebody who is 38 years to be getting a big contract unless you are Ronaldo or Messi so when it comes to age factor we need to invest in our young guys so that we have guys who can replace the ones who are getting older effectively and you know it's not just replacing so that you're able to go and maybe represent the country in Olympic Games is being able to represent and maybe be able to challenge for medals and it looks like it's been a smooth sailing for boxing federation of Kenya since Jamal Otieno rose to presidency of the association and the preparations for the boxers the eight squad representatives in Tokyo Olympics was fairly well because even the team was training in Lavington at the VV fitness launch and even everything was catered for them so I think we can exonerate BFK when it comes to you know complaints of poor preparations for the teams that represented Kenya at the Tokyo Olympics because going by our social media reports people have been condemning you know Kenyan federation honchos for failing to prepare the athletes properly can you exempt BFK from that blame? Yes, I will without a doubt uh for anybody who has been following boxing you you will understand the kind of arrangement that we had as box refederation of Kenya to prepare the boxers and we provided them a very conducive environment where we have state-of-the-art boxing gym at AV fitness in Lavington and it's an environment that even the our National Olympic Committee was not able to sustain for us to use as we prepare for the Olympic Games because we moved from AV to Kasarani where in terms of facilities is not the same and even when you ask the boxers themselves they'll tell you the difference that they encountered changing that environment from Lavington to Kasarani and when we go to Kasarani it was supposed to be a bubble bubble training camp and again it's when we go to Kasarani that we got our first case of COVID in the team which was not easy and which most people don't know about and I can tell you when you encounter COVID case during training camp it's hell because boxing it's a contact sport you don't know how much whether the case you have has spread to a few members of the team because you operate very closely so once we encounter a case in the camp it meant losing about five days until the next test are done because I think you are doing test of gravity 96 hours it meant losing that much time in terms of doing sparring awaiting the next test and confirmation of who is clean and so that you know what can be done because again I will not to train you following the government measures of social distancing and you see being a contact sport you know somebody hits you on the face and you are very close so it was a bit it was a big challenge the kind of preparation that we had at Kasarani COVID presented a very serious challenge to the preparations and it's totally I've also had almost similar cases with athletics I think I saw something on TV yesterday about the guy who under 20 120 championship Wanyoni gave a story and I felt for him Wanyoni of fresh work or 800 meters because most Wanyoni is dominated the competition Ya the Wanyoni did Wanda this time and he was very impressive so he was saying how he had tested positive then a few days to the race he tested negative and that's how he was able to participate and I could relate to that because we experienced the same during our camp and I know what it is and even psychologically it really gets to them and he rightly say that most people are so fearful of getting COVID than the race How is this bubble camping for the first time getting introduced due to COVID-19 pandemic I think it's a global practice where athletes are confined together so that they don't get subjected to infections related to the disease How is it like? That's when you talk about it that's ideally how it's supposed to be whereby once you actually you are tested before you get into the camp and once you get into the camp you are not supposed to go out and whoever is out is not supposed to come in but I think there was a problem somewhere I don't know in terms of organization because the guys who were serving us there were commuting from home so I thought that was a weak link which needed to be addressed because if it's a bubble it's supposed to be 100% bubble even for medical officers even for the staff who are operating in the hotel because they have people who are cooking for us and if them they are going outside and coming back home of course we don't know whom they are interacting with and you know COVID does not choose where you work and your status in life if anybody can get it and anywhere so it was a bit of a challenge that's how ideally it was suppose we get in you are not able to come out and when you are out you are not supposed to come in but we had a weak link in terms of some staff who are operating there were going out and coming in but I believe the last few days when we were just about to go out to Tokyo that's when it was corrected but it's very tough having to stay in that control environment but again I think they were also preparing us for what was likely to happen in Tokyo because in Tokyo again we encountered and most chic this time because if you violated any of those there were penalties and the worst of it could be refocation of your accreditation status which it was very okay we were like nobody encountered that but it will have been very unfortunate if you are having to return home because of not observing the COVID the COVID measures that have been put in place and we had also signed some code of conduct which you are supposed to follow we had been sensitized enough so I think we are good in terms of preparing us for what was to happen to us when we go to Tokyo Generally Shuguras we wind up tell us about the world international day for boxing yesterday what did you do with it yeah yesterday we celebrated and I didn't know that world international boxing they normally come from 28th of august I know you thought it was supposed to be 27th 27th no it's because you've been misinterpreting the ones in December 26th boxing day considering where I come from unjustifies to misinterpret yeah so yesterday was I think it's the first ever yes international boxing day and we were lucky enough to be chosen as one of the centers main centers actually in Africa where we were to host the celebrations for the day we organized an event at Nyaya National Stadium we tried to call about five people from all counties that have boxing and we facilitated them to come to Nairobi for the event and we had a successful day we were joined by some officials from the minister of sports from the national Olympic committee I was particularly impressed by Secretary General Mr Francis Motuko who really yeah he trained with us and I think he had fun he enjoyed he's a good dancer so we started with Zumba as part of the warm up for the sessions so we had a a three hour session where we were training and hosting some some bouts exhibition bouts so that we celebrate the day for boxing and we called boxers who were young or as young as nine and we also had veterans some of the big names that brought so much glory to this country the likes of Steven Moshoki was there George Foreman Joseph Acasamba those guys who did as proud during those days and we also had hit squad they were there all of them we gave certificates to the veterans just to appreciate them for their efforts and service that they've given to the sport of boxing we also gave out certificate that the coaches who did some IBA star one coach which we were able to conduct here in Hilton it was virtual so the officials who qualified and the exam is very strict you have to get 80% and above for you to qualify to be a star one coach so we had nine people whom we got their certificates and we handed them to them yesterday during the world international boxing day and it was big more so in Belgrade Sabia because that was where the main Sabia was the main center because that is where we are going to have the world championship this year sometimes in October we will be able to send some boxes there to represent that during the world championship finally what's happening to our lady and someone who brought glory to this particular country I've seen her trending for the better part of yesterday Konjestina Ching because government had promised that it would put her into rehab so that she can get reformed and she can regain her normal status like she was before because she is someone who did exemplary achievements for this particular country during her heydays what is happening under BFK are you seeking to intervene because it looks like situation is getting out of proportion for Konjestina's case it's very unfortunate what she is going through first I want you to understand the roles because Konjestina was not in how much a boxing how much a boxing or nowadays it's called open boxing it's what we do now as we are able to go represent the country in the Olympic Games Commonwealth Games Africa Africa Championship Konjestina was in pro was in professional which is technically under KPBC but as a as a federation we can always give support and actually I know our president was involved in getting her to rehab those are I think most credit went to former governor for Sia Sonko for Nairobi yeah when he went into rehab so I but they were working together with our president Mr Jamal and we took Konjestina and Bilali to a rehab but sometimes I think her problems are beyond rehab yes actually and she has gone to rehab now twice I believe there was another another time when Radul Karol Radul organized for her to go to a rehab which was in Thika and she stayed there for about three months and she completed that very well I know Radul had a bigger plan of seeing how she can be helped in terms of getting making her to be active and maybe even also facilitating maybe the caregivers who are taking care of her to be able to get amines of running a livelihood so that they are able to support the efforts of trying to now take care of Konjestina ah I'm not sure if she's getting any any support at the moment from the government though promises have been given before but sometimes these promises are given and they are forgotten immediately they are given so I will think it's a high time we also try to appreciate her one of our heroines because she did she did the country proud when she was fighting she used to tell us to come early because she is going to finish the job quite early so I believe it's a high time that also maybe the government takes a serious initiative of trying to assist her sometimes her problem looks more psychological I don't know it's it's leave it to the doctor it's something much technical than us just talking about it here because I have I have seen some of the initiative that have been held to try to to assist her but somehow it keeps recurring how did speak about it authoritatively because I think from medical perspective we need serious psychologists to look into it and tell us exactly what is she suffering from I think what I saw trending was something like she burnt her house or something I'm not sure I don't have evidence if for real she did that but I think her problems are much bigger than just asking a federation to assist let's hope that we will get to the bottom of what she's going through kundistina of course someone who did this country proud bringing glory back home but right now a lot of predicaments medically and she did psychologically she might be facing a lot of problems but we very optimistic that it will be solved soon the government will intervene and all the relevant boxing state called us locally sugar it's been an honor coming through your parting shot as we wind up from the boxing perspective I think as a federation we are going to try our level best to see how we can continue keeping our boxers active we've been very disadvantage from the pandemic point of view whereby our sport was not reopened I think from the time when we got our first case in the country we've not been having boxing actively going on across the country luckily for national team we've been able to go to some international assignment but we think we really need to do something so that we are able to reopen boxing even if it means having to vaccinate all the boxers and seeing how we can put up measures to be able to hold a few tournaments before the end of the year so we want to assure our fans that we are going to do everything possible to bring back the sport of boxing we might not do the league at the moment but we can organize one or two tournaments to be able to keep our boxers busy and we want to invest more in to in our youth so that we are able to get young boxers coming up to replace those who are aging out we also need to put a lot of emphasis in women boxing because come 2024 olympic games in paris we are going to have equal number of boxers for men and women in the spirit of gender parity which is being spearheaded by the international olympic committee so there is a very big opportunity for young girls to come join boxing and within the next three years they'll be able to represent the country at the olympic games joining some of the pioneers who have done that and that is Christine Ongare who was in Tokyo 2020 and Elizabeth and Diego who represent the Kenya in the London olympic games 2012 Thank you Dan Kankuru also known as Sugarai International Boxer formerly of course he played the sport and he is currently PRO for Boxing Federation of Kenya and he was the team manager for HIT Squad and they just concluded the Tokyo Olympics Championship joining us this particular afternoon to share AAC insights with regards to the state of the sport in the country and he says the future is very bright we are looking forward to see how that goes of course the line continues until three o'clock in the next few minutes Arsenal will be up again Manchester City in what is expected to be I don't know when I would call it a mouth watering clash because some people saying that you know it's sort of a one-sided affair but you never know in football anything can happen of course that's coming up next but the fun zone is due to happen in the next few minutes don't go away stay tuned


And Don's spacecraft has been spirally closer and closer to this mysterious world. And in just a few days, literally this week, we will fire our ion engine and set course for our nest destination, the dwarf planet Ceres. But even further after, Cassini is darting between Saturn and its moons, returning a ceaseless stream of astonishing discoveries. In Cassini's eyes, we have seen the sun glinting on the methane base of Titan. We've flown with Cassini at over seven kilometers per second through a 50-kilometer geyser erupting from the surface of the moon in Celadus. And in just a few more years, we're going to reach this spacecraft into one more daredevil maneuver and thread the needle between Saturn and its rings. This is happening right now. And there are still so many more places that we need to go even in our own neighborhood. Near Jupiter, Europa awaits. Beneath its frozen crust is more water than on the entire planet Earth. We must attempt a landing here. And when we do, when our robotic avatar arrives, you will stand on that desolate planet state. You will look up and see Jupiter overhead and then descend with us through the frozen crust to the ocean underneath. You will look with us onto a world that even our most powerful telescopes have never seen. But we're not going to stop there. You will swim in those waters, search for life in a world we saw. When we found in the Hubble Space Telescope of what looked like, every dot, every smudge you see is an entire galaxy.


Hello, welcome to our really, really specific open telemetry collector talk. My name is Tyler Helmuth. I am a maintainer of the open telemetry collector trip repository and the telemetry Helmthrop repository. I started off as a user of open telemetry about two years ago, and then I started to start contributing. My name is Evan Bradley. I also help maintain the collector, and along with Tyler, I'm one of the primary contributors to OTTL. I didn't have much experience with open telemetry prior to joining the project about a year ago, but it's been really great to be a part of the community. So first I'd like to quickly cover what we're going to talk about today. First we're going to talk about why and how you could use the hotel collector to process data in your telemetry pipelines and how the open telemetry transformation language can help you with that. Then we'll take a look at OTTL in action using a real world use case, and finally we'll tell you about the current state of OTTL and where it's headed next. So before I start, for anyone who isn't familiar with it, the open telemetry collector is an observability pipeline middleware that can receive, process, and export data at scale. The collector comes with a lot of functionality, but we're really just going to be focusing on its processing capabilities. So why might you want to use the collector? Well, there's a lot of reasons on the slide, but the first big one is that you can process at the edge. Processing at the edge allows you to split this work across multiple machines, which can help increase data throughput of your pipeline. It also gives you a chance to control what data enters your pipeline to begin with and how it looks coming in. For example, you may want to remove unnecessary data early on in your pipeline to avoid racking up bandwidth costs. Your data may also contain PII that isn't ever allowed to leave its origin, so it needs to be redacted or removed as soon as possible. You can run the collector at the edge or anywhere else in your pipeline because it can be deployed anywhere. It can be deployed in containerized, virtualized, or even functions of service environments. And you can process data close to its origins or further away, such as at critical points of your pipeline, such as at the point of ingress at the boundary of a secure network. You can trust the collector to adapt well to these situations because it's fast and versatile. It has been written with high throughput and low latency in mind, so it won't slow down your pipeline, and it has low CPU memory and disk space requirements. Custom collector builds can also be made to fit the situation at hand by selecting only the components you need. And for the cases where you can't find anything that's already out there, all collector components are written using the same core APIs, which means you can leverage these to add your own code to accomplish a task. The flow of data through the collector is organized into pipelines, which are composed of individual components that each handle a particular task. The collector has five classes of components, but the three that we're going to focus on today are receivers, processors, and exporters. This diagram shows an example pipeline, where data comes into the collector at one of the points on the left, proceeds through the pipeline until it's emitted on the right. Starting on the left with receivers, receivers take data from an external source and translate it into P-data, the collector's internal format that's based on the OpenSlamtree protocol, or OTLP. Processors then take this data and perform some kind of filtering, editing, or additions onto it before forwarding it on. They both consume and emit P-data, so they can be chained together within a pipeline. Once processing is complete, an exporter will emit the data to its destination in a desired format. Most transformations of data happen inside processors, so we're going to be focusing on those today. So let's take a look at a slightly modified version of a real situation seen by collector users. Suppose that we want to monitor our Kubernetes cluster to quickly detect field deployments that happen because our pods aren't starting. We can use the K8 object receiver to monitor the cluster and collect the events, then forward these to our backend that takes logs over OTLP. The logs have all the data we want, but we're seeing a few problems. First, our Kubernetes cluster is very active, and it turns out we're collecting way too many events, and it's costing us way too much. Additionally, the log payloads are missing metadata, like the associated pod, and this is making it difficult to efficiently query for them in our backend. We know the collector should have some processors that could help us out, so let's figure out which ones we should use. So first, let's take a look at how this processing might look prior to OTL. Well, we know that the collector has processors that support each of the hotel signals. Processors are designed to work with at least one signal between traces, metrics, and logs. Our events are coming in as logs, so we're just looking for processors that can work with logs. Using the data that we have in our logs, we're looking for one or more processors that can fulfill the following requirements. We want to filter out all of the access logs using data from the body first, and then with the logs that we keep, we want to be able to use their bodies to extract their log levels in an associated pod onto the OTLP payload as attributes. So when we look closer, while there's a lot the collector can do, the situation isn't so simple. We've used the collector to process our metrics and traces before, but we're finding that a lot of the functionality offered for logs is different. The configs for the processors that we're looking at here are different than the ones we've used before, despite doing pretty much similar things, and so we have to learn a whole new config structure. And once we learn that config structure, the real kicker is that there's no way for us to even access the log body in the way that we want. First, filtering requires running a regular expression on a stringified version of the entire log body, and this is going to be really slow. There's also no way to lift data from the body onto the attributes. Taking a step back for a minute, what features would we like to see in a more generalized solution to our problems? Well, first, we'd like access to the full OTLP payload in one place, which is going to be necessary for us to access the specific parts of the body for filtering and metadata extraction. We'd also like to be able to form operations at any part of the payload so that we can get things looking exactly how they need to. And finally, we'd like it if our prior experience of metrics and traces would carry over when we're working with logs. So this is where OTL comes into play. OTL is a domain-specific language that specifically targets processing data in the collector. It runs inside the collector's process and is designed to directly work with Pdata, which cuts out the overhead of specialized execution environments like language VMs. A key benefit of OTL is that it works very well with Pdata's hierarchical structure. You can see an example of this data hierarchy in the log on the right. Just like OTLP, Pdata provides structure for common information found in telemetry, such as the resource or application that produced the telemetry or the instrumentation that collected it. Both of these are included in addition to the data itself. An obvious question to ask is, if the collector is configured with YAML, why use a DSL in this case? Well, using a DSL for processing allows users to easily express their desired transformation. Compared to a YAML configuration, it's substantially easier to write, edit, and read. The example transformation on this slide showcases the fact that the OTL statement is way less verbose than the equivalent YAML config. And we also hope that OTL syntax is going to feel familiar to many collector users. It's pretty simple and it's very similar to popular programming languages like Python, JavaScript, or Go. So as a language, OTL has been carefully designed to be just sophisticated enough to be sufficiently expressive without having a high learning curve or being difficult to read. To enable the user to do any sort of transformation you would like, OTL comes with lots of built-in functions that work on a variety of data types that might be seen in any signal. Using functions to do these operations also allows them to be composed. The output of one function can be used as input to another. OTL also includes enumerated constants that have been taken directly from OTLP that allow the user to name a value instead of having to remember its integer representation. This is really helpful for things like metric type or log severity. OTL also has ware clauses that allow you to find under which conditions a transformation should occur, and you can combine conditions using its Boolean operators. For cases where you need to work with numeric values, there's a standard set of math operators. So when we take a look at the prior landscape of overlapping or missing functionality and mismatched configs and processors, this becomes cleaned up. All signals are now processed the same way, config formats match, and the user can easily express their desired transformations. Coming back to our use case, let's take a look at how the data that we do have can solve our issues. Well, first, we see a reason in a log body, so if we can determine what sorts of events we don't want, we can filter them out using those fields. We also see information regarding the pod, so we can look at extracting those details into attributes on the OTLP payload. Our backend is gonna later use these attributes for querying when we go to find the logs. When we use OTL to solve this problem, we find that we can fill our requirements with only two components. Using the filter processor, we can drop the data we don't want, and then with the data that we keep, we can use the transform processor to transform it. The log on this slide demonstrates the final result that we're hoping for. It has metadata that shows the pod name, event type, and severity for the log. All of these will keep in the log intact. I'm now gonna hand it over to Tyler, who's gonna show how OTL can do this. All right. Let's take a look at how OTL enables our goals. We've decided that we can reduce our ingest volume if we drop events that have a reason of completed, because we've decided we don't want them, and we intend to use the filter processor to make this decision. This is an OTL statement. Sorry, this is an OTL condition. OTL conditions work with the underlying telemetry, but they never change it. The filter processor can use OTL conditions to choose which data is dropped. When the condition is met, the processor drops the data. In this scenario, we expect the K8s objects receiver that we mentioned earlier to be emitting the K8s events as logs, and those events are nested maps inside the body of the log. If we see a body that is not structured like we expect, we should drop it because it's not a K8s event. You can see in the first example that the body is a map, and it contains a nested map in that object key. Since the conditions are not met, the data is kept. You can see in the second example that the body is a string. A string is not a map, so the condition is met and the data is dropped. Now that we've ensured the body looks like how we expect it to look like, we can start checking the values of the event itself. Remember, we want to drop events that have a reason of completed. OTL allows nested indexing, so it's really easy to reach into the event and grab a value, and compare it to the static string completed. You can see in the body of an example that we dropped the completed event, but we keep the top event. So now that we've filtered out the data that we don't want, we can start safely transforming the telemetry we want to keep. This is an OTL statement. Unlike conditions, OTL statements transform the underlying telemetry by executing a single function, in this case, a function called merge maps. The target for transformation in this function is the first parameter, cache. Cache is a super specific hotel empty map that we can use as a place to store information in between statements. The second parameter for merge maps is another map. In this case, it's that nested map inside body that we checked on earlier. The third parameter is a static string, and in this example, it's the string absurd, and it tells the function how to perform the merge. So in this statement, we merge the object map into the cache map for later use. And while this statement isn't strictly necessary to achieve the goals that we want with the end result, it is gonna make the future statements we look at simpler. When you're using OTL, this will be one of the most common operations you perform, setting. The set function allows you to set a telemetry field using another value. The target of the set function is the first parameter. In fact, you'll see that pattern often. We have a standard that the first parameter of a lowercase function be the thing being transformed. So our target is an attribute on the log called reason. If the attribute doesn't exist, the function will create it. The second parameter is the new value. Our team has decided that we need this attribute to be lowercase. So we're first going to invoke the convert case function to change that text to lowercase without modifying the value in the cache itself. The output of convert case is then passed on to use as the new value and set. So this statement sets the reason attribute on the log with the new value of the event's reason. And it does that by retrieving the event's value or the event's reason from the cache and passing it to the convert case converter to get the lowercase version. You can see here in the example that we've added a new attribute on the log called reason with the value of backoff. One of the great things about open telemetry is its semantic invention. Our backends know how to take advantage of these standards, so we wanna set them whenever we can. In this instance, we set the case pod name semantic invention for any event that's associated to a pod. The first parameter I've set this time is a resource attribute called case pod name and the value is the name of the pod and we extract that from the cache. This is a really, really powerful concept in OTTL. Even though we're working with the log directly and we're grabbing values off of the log, we're able to reach up into the resource the log is associated to and change its values also. Now, what if this event isn't actually for a pod? That was the case. We wouldn't want to execute this statement. So what we can do is we can add a where clause to the statement to make sure it's only executed if the event really is for a pod. The where clause acts as a decision maker. If the condition is met, the statement is executed. Otherwise, the statement is skipped. Finally, we wanna take advantage of the native OTTL log field severity number. The field represents the severity of a log. The case events we're collecting can have two types, normal or warning. If the event's a normal event, we wanna set the severity to info. And if the event is a warning event, we wanna set the severity to warn. We could take advantage of conditions again to ensure that for any log, only one of these two statements is run, and therefore the log severity is only set one time. Severity number is actually an integer, but hard coding a nine or 13 into the function wouldn't be very descriptive. Instead, we can take advantage of OTTL's built-in enums severity number info and severity number warn to represent these numbers. These enums reflect the enums available in OTTL P and keep the statements easy to read and easy to write. With the transform processor and the filter processor in place using OTTL, we can achieve our goal. The top event is a backup event, so we keep it in transformant. What started off as a basic log, which is the event in the body of the log becomes a log rich with data that we've filled in, filled in all of its OTTL P fields. It has a k8spot event name as the semantic invention on its research attributes. It has the use in useful reason attribute that could maybe help us visualize the data later, and we've set its severity number, which we could maybe use for querying or something. The second log is the completed event, and we've determined we don't want those, so it's dropped during filtering. We do the filtering first so that we don't waste any time transforming data that we intend to drop. So what does it actually take to configure all of these statements? Turns out, not much. The configuration is pretty minimal. This config shows a snippet of a working collector config, and this is just a specific filter and transform processor sections. For the filter processor, each condition gets its own line, and then for each log record, the conditions are run in order, and any condition that matches the log's data means that the log is dropped, and any remaining conditions are skipped. Similarly, the transform processor, each statement is run in order, updating the log if the statement is executed. Again, we use a DSL to make sure that this can stay small and compact, and that our config stays readable. So we've been talking about one really specific example through the whole presentation so far, how to use OTTL to mess with your logs, but it can do way, way more than that. OTTL enables data operations like interacting with lists. You could transform your instrumentation scope, changing its attributes or its name or whatever. You could extract new attributes from a string. You could do intricate time manipulations on your metrics. You could generate new metrics. You could interact with your span events, change its attribute, change its name, and so on and so forth. All of these data operations are possible because of our log standard function library. Listed here are just some of the functions available to you. Those functions with the uppercase first characters are pure functions. They don't change underlying telemetry, like the convert case function we used earlier. Functions like this, you can do things like combining two attributes using the concat function. You could use parse JSON to turn a string from the body of a log or from an attribute into a map for future use. You could truncate the length of strings. You could limit the number of attributes on your payload. You could calculate span duration using OTTL's built-in arithmetic capabilities, hash attributes, and so on and so forth. These aren't even all the functions available for you to use. It's only what we could fit on this page. And if you start using OTTL and you feel like there's not a function that you need, just let us know. And we can work on integrating that into the library. So where can you use OTTL? It's available in several processors already, with the emphasis on the transform processor being the place that actually does the transformation of telemetry. The filter processor and the tail sampling processor use OTTL conditions to make decisions. The filter processor we talked about earlier uses the condition to determine when to drop data. And the tail sampling processor, you can use OTTL conditions to determine when to sample a trace. The routing and the count connector can also use OTTL conditions to make decisions. And in fact, be on the lookout for more components to start using OTTL conditions in this way, as it's one of the ways we hope to start standardizing configuration between processors. If you've got a custom component and you think that you could take advantage of OTTL to do transformation of data or to make a decision, great. OTTL is a package. It's available ready for use. And it even supports custom functions should you have a really specific need. So what's next for OTTL? It's been undergoing a lot of change in the past couple months, and it's currently considered alpha, but it is ready for general use. At this point, we believe the language has all the features it needs for the future and is in a good state. And we believe most of the breaking changes to the API have been completed. Once we're confident that the API is in a good place, we'll promote it to beta. The transform processors development closely follows OTTLs. We'll promote it to beta once we promote OTTL to beta, but we do feel the transform processor is ready to handle most common transformations. If you have any feedback or questions about anything related to OTTL, please open Github. It's really, really helpful for us. It helps us determine the priority of our users and to keep our quality high. Once we've promoted the transform processor to beta, we'd like to start deprecating redundant processors. We have an issue open already to determine exactly how we're going to approach this, but the processors we're targeting right now are the attribute, logs transform, metrics transform, resource attribute, and a span processor. Our goal is to make this deprecation as painless as possible. If you've made it this far and you want to know more about the things that we talked about today, here are some links to the different components that we talked about. Our goal is to always make our documentation for OTTL better. We want people to be easily to learn how to use OTTL, how it enables their use cases, and just like what it's all about. If you look at any of these docs and you want to provide some feedback, please let us know in CNCF Slack or in Github issues or whatever. Like I said earlier, it's really, really helpful when people engage with the community. Thank you for listening to us talk about the collector and we're opening it up for questions now. Yes, that's a great idea. And I'm part of that. And I should have known to promote it. Yes. So if you like open telemetry or if this is interested to you, if you like OTTL or whatever, on Wednesday at 2.30 is ContribFest.  We'll have a bunch of curated issues in the collector. A lot of them are OTTL, new functions that you could add. Come join us and make your first contribution to open telemetry or make your second or your third or whatever and just come help out the project. I don't know how questions are working, so. Is it working? Yeah, so it looks great. I have one question that is it possible in OTTL to create like a user defined function? Like in your example, the parsing, you know, the warning messages can be warning, warning for the back cell and I don't want to, you know, like define all these lines in every configuration. So OTTL allows custom functions. So like there's the set function that we looked at. If you wanted set to do something really specific, instead of using our set, you could define your own set and then your component could give that to the package and say this is a function available for the DSL to use. Does that answer the question? Yeah, just the one for what? So I have to define in the go programming or define as, you know, like a set of set functions? You would have to code it up and like build it into your, like write it in go, build it into your component and then it's compiled like with your component and OTTL is like a package. Great, thank you very much. Yeah, we do not have dynamic function imports yet. There is an open issue about that. If someone really likes Wossum, like that could be a cool way to do it. I'm curious about connecting like an orphan span via transfer process. I know that doesn't work now, but is this like a feature that could be imagined to work someday? It's tough to answer that one. I think it would depend on your case. If you have the information around the trace ID that you want to connect the span to, yes, but I think that you would need state around previously seen spans within that trace in order to do that. The transform processor is stateless so it wouldn't support that right now. But OTTL could be used in a stateful processor in the future to support that use case. Do you have any explicit design goals around things like safety? How do I make sure that a possibly untrusted user is provided some OTTL?  Yeah, so OTTL is like what is available to the collector is compiled into the collector and so you can only write the DSL to interact what's available. So you wouldn't be able to inject custom code or anything like that. There's no availability for dynamic functions or anything like that. If someone got a hold of your config and mess with your statements, it would change what the transform processor was doing but they would have already gotten into your system and the collector's already compromised at that point. So I'm thinking about building a system. Maybe I wanna build a system where I can provide a user to offer dynamic OTTL so they can define OTTL run time. Is there like, are you intentionally not providing Turing completeness to make sure that they can't run something forever? Anything like that? Yeah, I would not call the language tranquilated. It's very constrained. I should mention that each statement doesn't, there's no state between each statement. You can use the cache, you can put values in a map and then refer to them later but there's no way to set variables or define functions or update any kind of like environmental context. Hey, how's it going? Great talk. Do you have any mechanisms of validating transforms where you could sort of like give an input and output and say, okay, this is the set of transforms and this is what I expect to happen. Almost like unit testing your transforms. That is a really, really good idea. And the answer is the debug exporter right now which isn't a great answer. But right now it's like run it through the pipeline and see what the collector spits out. It would be nice to have some more rigorous like validation in like capabilities in the like package itself. That would be nice, yeah. So I wanna make sure I'm not mistaken but you mentioned that the logs transform processors was the things that's being deprecated in favor of this? Yeah, so the logs transform processor uses stanza which is another type of transformation package in open telemetry. So the logs transform processor right now is in development and the main concern around it is that you need to translate things from Pdata into a stanza specific format and then back into Pdata and the inefficiencies around that conversion are what we're concerned about. What we're trying to make OTTL like functionally equivalent so that there's not gonna be any loss there. Yeah, I was wondering if there was anybody who would, I don't know how popular hotel log usage is right now. So maybe not, but like the process of like moving all of, if you have a lot of complicated like expr transformations that you used to do and now it's being deprecated was there like a warning ahead of time and did people know this was coming? Are there anybody who's like upset by the? Yeah, we haven't started it yet. So like nothing's happened. Okay, so this is the warning. Yeah, well, this is a statement that like we'd like to get to it. As Austin talked about earlier, statements in 2019 may not be completed till 2023. So like no timeline. It is 2023. Right, so like this statement is being made now. Maybe it'll be done in 2027. Like I don't know. The logs transform processor is currently in development. It's not included in the contrib distribution.  That's great. When we get to that transform process or the logs transform processor, like we are looking at ways to take, like take in the config and spit out the OTTL equivalent statements. So like filter processor right now you can do OTTL statements or you could do the old style of config. We have already written the converter of the old style config. It'll read it in and like turn it into OTTL. And the next step is to like spit out that statement and like that'll be part of the deprecation process. There is also way back, way back. I think like two years ago when we first started doing OTTL work, there was a question of logs transform processor OTTL. Which one should we do? And Dan, you can ask him more about it but he did a huge comparison between Stanza and OTTL and like would it be possible? It's all written up in a really good issue. And the answer is yes it's possible. It's a lot of work. So like we're still working towards it which is why nothing's like officially marked deprecated or we haven't done any feature gates or anything like that yet. So I wanted to add on to that. For things that are using EXPR like the filter processor just add on to what Tyler said. Like when we convert those OTTL statements the plan is as part of the deprecation process the warning message that we print out that says like hey this is deprecated. We're planning to print out like here's the config that you should move to like just copy paste this and you're good to go. We're trying to make it as painless as possible. So there is like a conversion story between expert. Exactly, there is a conversion story. For the filter processors use of EXPR we're getting kind of specific and we should get off the stage. Yeah, you're right. For the EXPR filter processor specific config maybe not because that one is like a full language and like converting that full language to our full language would be really, really hard. So maybe not that one but also the EXPR stuff there was like the filter processor didn't build in too many capabilities. If you've got some really complex situation like I'd love to know more about it. I'll message you. All right, we'll pass on to whoever's next.


We are in the extraordinary moment of history. This is the age of nationalism, this is the age of isolationism and tribalism. It seems that the world is just overwhelmed by pessimism. No, we should look in a very confident way into the future. We cannot continue with an economic globalization that creates these big inequalities. We would solve so many of the issues that we have in the world today if we created a little more empathy for people. We need a collective stewardship of the entire planet if we're going to succeed with the world economy in the future. We have the potential now to create much more engaged, conversational institutions where we're able to take account of the ideas and the wisdom and also the know-how of people. The Fourth Industrial Revolution is not something that's done and over. It's a great opportunity because it enables us to deliver services of very small quantities to a very large number of people. There's tremendous progress that has been made, but we have to be mindful that the pace of change is going to accelerate. Technological disruption, globalization, climate change, demographics are coming together and changing the future of jobs. And I think that is where there is that reform needed to ensure that people are able to continue to fulfill their human potential. Basically, what is the meaning of work? Why will people work? There's always been an idea that if we work less, that's great, but work isn't just about income. What is the meaning for those people? How do they self-define? That's a harder thing to achieve. We have to manage that balance and that transition and provide retraining capabilities for our employees, provide great opportunities for them as they leave the company to find other jobs, and ultimately have to be sustainable in every dimension. By making business models more service-based, we as a company can actually much better ensure that the raw materials in our products can be repurposed for a second life. Economic globalization is a double-edged sword. The real enemy is not our country, but the weak, the poor, the ignorant, the superstitious and the biased. My story is one of a million. I know refugees who lost their mothers on the way. Some people are saying that refugees don't have to belong to the community. What I'm fighting for is that refugees have to have the rights that normal people are having. We believe that migration will create a net positive value to the recipient countries, including in the fiscal side. So we've got to do better in terms of communicating the benefits of migration and also to attenuate the impact that it has in some local communities. It's critically important to bring more diversity into educational institutions, because if you look at the world, the conflicts that are happening are not because of nationalities. They're because of haves and have-nots. The arts play a crucial role in helping to put the questions out there that really need to be answered. The Afghan Women's Orchestra Zora, there's a group of young women playing and performing music in their country, represent inspiration for many, many other young women around the world. I'm the first female conductor in our country. I told to my father that I want to be a musician. My father said, okay, it's your life.  The leadership really does start with where you are, regardless of where that is. And the challenges that I find in my media community, there's something I can do about it. I don't have to wait for permission from anybody. That means not being afraid to act even when you're threatened. Not being afraid to speak truth to power. And above all, you must be accountable. It's not about me. It's about the people whom I'm there to serve.


So, this worked great, no? We can absolutely distinguish images from white noise.  Well, let us see about that. So let's focus on the generator part now. Now, what's the generator trying to do? The generator is trying to do, in this view, it's trying to minimize the probity of being caught. In other words, you can say the generator wants to produce images such that the discriminator, when it thinks that it's a real image, it's actually a fake image and vice versa. So, one easy way of doing that is to give it the cost function, which is just the opposite of the cost function of the discriminator. Mind you, there's these two components. This part is the probity assigned, the log probity assigned to the images if they are real. And here we have the log probity of one minus, the log of one minus the probity assigned for the images that are fake. Now, here's a slight problem there, which is if D would go to zero, then no, if D would go to one, then this term would go to infinity. So, what people often do here is they rewrite it. So, they replace the term that's log one minus D with minus log D. It's a monotonous function within the relevant domain. Now, like here the value of D will generally always be positive. Now, like it's going to lie between one and zero, usually that's because we have a normalization that only allows values in the domain. Okay, but the original cost function goes to infinity as we approach, to minus infinity as we approach one, whereas the newly rewritten one stays in a more meaningful range as we approach one. So, that is the cost function that people then often use for GANs. Now, what's the GAN learning idea when we talk about probity distributions? Now, like what goes in is a random number. Maybe what we want to have is a probity distribution like the green one here. And what we are learning is effectively a mapping of a random number into the SpaceX so that ultimately after learning the two distributions are the same. And therefore we have ongoing low loss. Now, how's that going to work? We're going to produce a vector Z of random numbers. We will either draw them from a Gaussian distribution or from a uniform distribution. Now, you can say any one vector maps onto an image. So, the network that will be training here through the generator network. Not like any vector maps to something. Why do we need those random numbers? Well, we need the random numbers to describe the variability that is in the dataset. Now, like you can say the poses of the heads of the cats and there's the size of their eyes and there's the position of their ears and so on and so forth. So, there's a lot of degrees of freedom within that image manifold. In fact, it's quite interesting with this cat. It's a little disturbing because it's unclear what happened to the rest of the body of the cat. In any case, it's a fun webpage. This cat does not exist.com, which will give you every time you go there. A new cat that doesn't actually exist.  And this is where the neural network sets. It gets its input, this vector. It produces its output, an image like that. And now that we have a discriminator, let us freeze the discriminator. Let us use a simple multi-layer perceptron generator. Let's give it a couple of layers with real or non-linearities and ask, can such a trivial generator win, which is ultimately make the discriminator no longer know the difference between real and fake images? And then, of course, there's this question that's underlying here, which is, are the images it produces actually in any way close to real images?


Hi, everyone. I'm very happy to be here today in Shanghai, giving this talk to all of you. So my talk is yet another talk about AI. It's such a buzzword, no? But I don't think AI is just a buzzword, because we finally have models that actually work. Models that can improve your productivity. And since most of us here are probably programmers, one of these tools is code generation models. These are models that can also complete your code. It can be a function, or a class, or a whole code block, just like GitHub co-pilots, for example. So these models have been there for a while. But they've only been more trained and interested in research when Codex was released. It's a co-generational model by OpenAI, which is being used for the VS Code extension for GitHub co-pilots by Microsoft. Before this model, there were some small co-generational models, like CoGPT, CodeBert. But these were small models that were trained on a little data. So they weren't very good at the task. But when Codex was released, they kind of showed that you can just train a code model like you do for a language model. You take a transformer that is large enough, and you feed it a lot of code data, and just learn how to program. Then DeepMind worked on Alpha code, and AWS worked on Code Whisperer, which was great. But these models are all closed models, meaning you have to send your data to an external API. You also can't find you in these models. So that's when the open-source community tried to step in and train some good co-generation models, for example, co-paras from Huggingface, and coder from Meta, and co-gen from Salesforce. However, there are still some open questions about these models. First, regarding the performance, because these models weren't at the level of Codex yet. And there was also a focus on Python and not the other programming languages. There's also an open question about transparency. Which data were these models trained on, for example? There are also questions about evaluation and user experience. So these are questions that we tried to address. So now let's see what does Huggingface have to do with this? We have the hub for sharing data sets and models, but we also train models. Last year, we released Codeparate, which is an educational tool for training these models, showing you how to scrape data, how to train these models, and evaluate them. So it was more for education, and the performance was not very great. A few months ago, we released StarCoder, which is a strong co-generation model trained on more than 80 programming languages, and it surpasses the performance of Codex on Python, but also on other languages. The model is also open access. So today, I'm gonna talk to you about the projects where we developed this and about the architecture and deployment of these models. So we decided to create a big collaboration called BigCode, where we have more than 500 participants from over 30 countries. This collaboration is led by Huggingface and ServiceNow, but it is open for anyone to join. So if anyone in the audience today would like to help, we're very happy to have you. Our goal is to train strong co-generation models, but in an open and responsible approach. What we started with Code is to address some of the bad challenges of closed large language models development. This starts with not disclosing your training data and the sources that were used. For example, today there are a lot of great open access models, but we don't know on which data they were trained. The other bad practice is not releasing the model weights at all. So if you want to find two in these models, you can't. And if you want to deploy them on premise, you also can't, and you will always have to send your data which might be sensitive to third parties. Another thing that is needless to say is that all of this makes results not reproducible and doesn't foster research and open source. So what we think open development of these models should look like is exactly the opposite of all of that. First, you should make your training data public and you should give inspection tools so that people can look into the data and see what's in there and what these models were trained on. You should also create opt-out tools in case people want to be removed. Because for example, you as authors of some data or some code, you should have the right to say, no, I don't want to be in these model trainings. The other thing is that model weight should be public so that people can fine tune them and also deploy them on premise. And you should also give full documentation of the whole process to make it easy to use and reproducible. So let's see what it takes to train these models from scratch. It takes hundreds, maybe thousands of GPU hours, but also terabytes of data, but that's not just all. And I will try to show you what goes behind the scenes for training these models. Everything I'm gonna show you is about StarCoder. This code generation model that we released this summer and for Star Wars fans, this was named after race and wasn't released on May the 4th. Everything starts with good data. So if you want to train a language model, let's say on Chinese or English, you train it on Chinese and English text so it learns the language. If you want to train a language model how to code, you need to train it on code. And what's the base source for that? It's GitHub. So we cloned all of GitHub repositories and then we did a lot of data cleaning to remove extensions we're not interested in. And also we did some license filtering. Because if you take a code repository on GitHub, it can either have a permissive license like MIT or Apache too. It can have a copy left license like GPL or no license at all. For us, we only took the first category which allows us to use this code. We also built a data inspection tool. So for example, you can go to our tool, you type your GitHub username and if it tells you, if any of your repositories are in the stack, and if you don't want to be included in our future trainings, you just have to fill a simple Google form and we'll make sure not to use your data. We also did a lot of data curation to make sure that our data was clean and that the model could actually benefit from it. The first thing was to select a smaller set of languages and not keep all 300 because some languages are no longer used or maintained. We also added other sources. For example, GitHub issues conversations, Git commits and Jupyter notebooks. The other step for cleaning we did is deduplication because studies have shown that if you train the model on many copies of the same files, this is gonna hurt performance. So it's important to only keep one copy of each file. We also did decontamination. This means that if you have an evaluation benchmarks that you know you're gonna evaluate on after training, you need to make sure they're not in your training set. The last step we did for cleaning was removing personal identifiable information. Since believe it or not, there's still API keys and SSH keys and public repositories on GitHub. And we didn't want to train our models on that so that during inference, people wouldn't exploit this and use people's secretive data. If you're interested in how you can run all of these processings on very large data sets, then HackingFace data sets library comes to the rescue. This is the library we used for all our cleanings. It has some very nice features like applying filters and transformations using a method called map, which allows multiprocessing. And my favorite feature is batched mapping when you can leverage the power of map and using batches and make the filtering go so much faster. Okay, so we've seen how to prepare the data. Now how do we train these models? How do they leverage a lot of GPUs and make sure our model is optimized? Our model is a decoder model. It has the same architecture as GPT-3. It has 15 billion parameters with some code optimizations, but we did some changes to kind of address what people actually want from a code generation model. For example, we used the technique called multi-query attention to make sure that the memory footprint is reduced in inference, especially when you have large batches. We also used an 8,000 context length, which is long and can allow you to feed more context into your model. This is a decoder model, so usually it can only process contexts that's on the left, but we have used the technique called filling the middle, which allows your model to attend also to text that is on the right. For example, if you have some code and you want to edit it in the middle, you can do that with StarCoder. And for the training setup, it only took 512 GPUs from the Hagen-Face cluster for 24 days. It was kind of a smooth sailing where the loss kept going down and we've only had a few automatic restarts. We also used Megatron LM from NVIDIA for the training. So if these numbers scared you, don't worry. You don't need all of that to do fine-tuning because with libraries like Peft from Hagen-Face that use techniques like Laura, you only need little computational resources to do the fine-tuning. The idea is that if you take your very large transformer which has billions of parameters, you freeze all of them and you only add a small number of trainable parameters and you only train those. And you will get the performance that is matching as if you were to do full fine-tuning. This also lowers the storage cost because you only need to store those extra adapter weights. So let's see what the big-code ecosystem looks like. It started from a small family tree where we had the stack data set and some star-coder model and became a whole ecosystem. Nowadays, almost everyone that wants to train a code model they would start from the stack. That was the case, for example, for stability with staple code, for Salesforce and the replay, for example. There's also community fine-tunings where people start from star-coder as a base model and then fine-tune it on instruction data sets. For example, we have wizard-coder from the wizard LM team and the pangu-coder from the Huawei team. Okay, so let's say we now have a good code generation model that we trained. How can we turn that into an actual product? For example, a VS code extension that can serve hundreds of thousands of users. That's when deploying large language models for code becomes important. At the end phase, we have the inference endpoint. So if you don't want to take care of the infrastructure and the MLOps behind these deployments, you can just use these endpoints and you can query them to generate text or code. And what these endpoints use behind the hood is text generation inference. This is our inference library that we use in production. It is available on GitHub and it has a lot of optimizations. It also serves most of the popular language models. And what is cool about TGI is that it is production-ready. It has a lot of metrics and a good tracing mechanism. So you can follow your latency, your throughput, the request that you receive and be notified when there's an anomaly. It also has a warm-up system. We run the full pipeline before you put it in production so that we make sure it never crashes. It also has some optimizations. So TGI is focused on optimizing latency. What some other inference libraries might be reporting throughputs that isn't necessarily what you want because what you care about is serving as many users as you want as fast as possible. So caring about the speed is important. It also has continuous batching for handling concurrent requests, meaning that your batch size varies based on the number of requests that you receive. It also has token streaming. So for example, let's say you want your model to generate 200 tokens. Instead of waiting for the whole generation to be complete and then display in the result, you can display them token by token. And this reduces the perceived latency because the users see the results earlier. And it also makes your UI more interactive because if the users don't like the generation, they can just stop it in the middle and ask for a new one. So these are some of the optimizations that are in TGI. Some of the users are hugging chat, our UI for interacting with chat models like Falcon Chat, Lama 270B, and the VS Code extension, which by the way serves StarCoder and other code models like CodeLama, which is HFL Code Autocomplete. Open Assistant also uses TGI as well as Node.Dev. So for my last slide, how can I be at KubeCon and Cloud Native Con Summit and not talk about Kubernetes? When Kubernetes is what powers all of the hugging phase infrastructure. We use it for all our services, the hub, the API endpoint, dataset servers, and spaces. We have eight production clusters in the 800 nodes. The clusters are very dense because for example, if you want to build a space which is a demo on hugging phase, we usually give each user two CPUs. But these CPUs are usually most of the time not used. So what we want to do is to be able to fit a lot of pods in one cluster, and we can have up to 250 pods in one node. How we do that is using memory swap feature, where we swap the disk space for the RAM and we're able to fit a lot of pods in one node. One other thing that our team implemented is the recompilation of container D to pull images faster. So for example, if a user wants to restart their demo or space, they would need to pull the image of the container again. So if we can make this faster, this improves the user experience. So what our engineers did is to improve the checksum operation so it is 30% faster. So this is an overview of how we use Kubernetes at hugging phase. I hope this gave you an understanding of how we're able to go from raw source code that is laying there on GitHub to actual product like hacking chat or HFO to complete extension. And that's behind the tool. There's what I like to describe as an iceberg, where we only see the tip. It's a lot of GPUs, but that's not everything. There's a lot that is underneath the iceberg. This starts with a lot of data curation. You need to spend time actually inspecting what's in your data and trying to remove as much noise as you can. There's also a lot of data governance. And this was at the heart of the big code project. For example, in the training, we only selected permissive licenses. And then when we released the VS Code extension, for example, we implemented a code attribution tool. So if the model generates code that was copied verbatim from the training data, it is highlighted in red. It points you to the original repository in case you want to give attribution to the author because permissive licenses doesn't exempt you from attributing the author in case the code is the same. We also worked on inference a lot, as you've seen, because a lot goes behind the hood. So for example, the VS Code extension now serves up to, I think, 12,000 users, which is really important, as well as for the endpoint. And then there's a lot that goes in evaluation because you need to make sure that you evaluate your model as much as possible to have a comprehensive understanding of your tool. So that was my talk, and I hope you enjoyed it. Thank you, everyone.