Dr. Ladd, thank you so much for joining us today on This Is Purdue. I know so many of our listeners connect with cancer on a personal level, and you've actually helped create a drug that can find these malignant tumors. Tell us, but let's start at the beginning. How did you get into the chemistry field? Well, actually, my father was a chemist and he was a faculty member here at Purdue University and actually a very famous one. And as an obedient child, I took the courses in high school that I was asked to take. And he always signed me up for advanced science courses. So here at West Lafayette High School, I took advanced chemistry, advanced biology, advanced physics, and all of the math courses I could take. And I wasn't stellar as a student. I mean, I struggled through those courses. They were difficult for me, but in the end, I really enjoyed chemistry the most. And so when I went to BYU to undergraduate school, I instead what I should, he thought I should major in. And he said, well, why don't you major in chemistry? I suspect there may have been some selfish motivation in the absence. He was a famous chemist. But also he pointed out to me when you finish with your degree in chemistry, you could go if you waged in the business, go into sales, you could go into medical school or dental school, or you could continue on in science and sharpen your chemistry skills and get an advanced degree in chemistry. And at the time, I really didn't know what I wanted to do. But eventually, all the ducks lined up for me proceeding on and getting a PhD in chemistry. That's great. So you've founded numerous drugs, companies, you have patents under your name. Is that kind of along the same lines as you followed your dad in those footsteps? Or what made you so interested in innovation? I think what attracted me to innovation was the opportunity to do something that matters. A lot of chemists are motivated by the discovery of basic principles in science that can have a fundamental impact on lots of different studies. I was more motivated by doing something that would save lives or help people in some way or another. And I was always very intrigued by life sciences, how cells work, what caused them to be able to do so many wonderful and interesting things that cells do. And as I dug deeper, I learned I think a skill that a lot of people don't learn in their research careers. And that is whenever you discover something new or read about something new, sit down and ask yourself the question, how can I use this information to do something that really matters? And as I practice that fundamental principle, I found that simply asking that question opened up opportunities that I would have never envisioned had I not stopped to think about the potential value to humanity of the work that I was doing. And gradually, that just led me from initially working on cancer to more recently branching off into autoimmune diseases. Now we're working in CNS diseases like Alzheimer's and Parkinson's, and we're also working on fibrotic diseases and bone fracture repairs and things like this. So it's actually guided me in many areas I would have never envisioned and actually was never prepared to pursue, but had to learn on the fly because the ideas that I was coming up with brought me in that direction. And ultimately, I think we have over 500 patents in all of these different areas. And of course, as you know, we've founded six different companies all of which are successful. And speaking of doing good, your creation most recently gaining a lot of buzz is Cytolux, which is a fluorescent dye. And will you explain to our listeners how this dye works? Yeah, we have taken a dye that is that he emits light that is transparent to tissues, that is, it's light that will pass right through your hand, for example, it's not light that's visible to the naked eye, we have to have a special camera that detects it. But it is basically light, not radiation and damaging term of radiation. At any rate, we attach that near infrared fluorescent dye to a smart molecule that homes in very selectively on cancer cells. So when they're when the dye is piggybacked on to the this homing, this smart molecule, if you inject it in, for example, into the vein of a patient that is scheduled to undergo surgery, and perhaps an hour, the dye circulates through the body, but only attaches to cancer cells.  And so this allows the cancer surgeon to identify, locate and resect all of the malignant or cancer lesions in the patient and avoid removing too much healthy tissue because the healthy tissue doesn't fluoresce, but the cancer tissue does fluoresce. And this is a very important problem, because if you look historically, and even today, 40% of recurrent cancers recur in the tumor bed that was resected or removed by the surgeon. This means that the surgeon left a lot of diseased tissue behind, and that tissue grew back. And that means either another surgery or more often than not, it means because it's only caught much later that the patient is not going to do well, maybe not survive. And so enabling the surgeon to very clearly see they can just shave until all of the cancer tissue is removed, all the fluorescence is gone, when they see all the fluorescence is gone, they know they have removed all the cancer tissues. Kind of like painting by numbers, colors, you know, you see right exactly what you have to do. It's a very bright visual aid to the surgeon on how to remove the cancer. And you touched on it's important not to remove, you know, blindly some of these healthy cells. Why is that? Why is it so important to just get that cancerous cells out of there? Well, that's a good question. In some cases, it's not as critical to be really highly precise and not removing healthy tissue. But in brain cancer, people are fond of their brains and want to have as much of that left over as they can. And frankly, in other tissues, for example, in breast cancer again, if the cancer can be removed without removing extra healthy tissue, the patient would like would prefer that. In some cases, resection of healthy tissue actually does serious damage in prostate cancer. For example, if a healthy nerve is accidentally severed, the patient can be incontinent and impotent after the surgery and being able to see exactly where the cancer is and not cut anything that's non malignant will really benefit patients in all of these areas. Yeah, that makes perfect sense. So can you tell us about the difference between this targeted marker and others on the market that aren't specifically like tumor targeted? Yeah. Well, there are some other fluorescent dyes that passively concentrate to some extent in the tumor tissue. But the contrast between the malignant tissue and the healthy tissue is blurred. And the boundary between the two is generally not very and always not very exact. In our case, because the cell that our dyes bind to are malignant and the cells that it doesn't bind to are non malignant, the boundary between the cancer tissue and the healthy tissue is really quite sharp. So it allows the surgeon to really make a very exact cut and preserve healthy tissue while being quantitative in removal of the cancer tissue. And can you tell us everything that went into creating Cytolux and how many years in the making? What's this massive project? Yeah, I mean, I don't know where you want to start on that question. I could go way back and begin with the accidental discovery of a tumor homing molecule. Okay. We have all day. An interesting story. I was studying plant cells and asking the question how these plant cells detected pathogens like bacteria and viruses around because a lot of plants will wilt and die because of a pathogen infection like bacterial infection. And so we found that if you ground up some pathogenic bacteria and spread the dust on the leaves of a soybean plant that that would prevent that soybean plant from getting a bacterial infection later on somehow immunize the plant. And we wanted to find out what how that information was communicated from this ground up bacterial dust to the plant to create this immunity. And so I asked a graduate student to see if these any of these bacterial pieces were carried were captured by the plant cell and carried inside. I told him to radio label it and then see if any radioactivity went inside the plant cell. And he went away and then came back and said Dr. Lau I'm sorry I can't do the study. And I said well why not. He said because I didn't want to work with radioactivity. I thought gosh everybody you know our Elvis is doing it and we do it all the time. But anyway so I I deferred to his preferences and I asked him instead to put a biotin that's a biotin and attach that to some of these bacterial pieces and see if that went in. He did and it went in. We celebrated. Hey that shows how this signaling is taking place. But then I told him to conduct what we call a control study where you show that it really is recognizing the bacterial piece. So I said now put biotin on insulin plant stone have insulin. And I told him to put it on bovine serum albumin that's just a protein in the blood stream of cows and show that it doesn't go in because we know they didn't do anything to plants. And he put the biotin on those two proteins and they went into so this is how science works. And he came back he was terribly disappointed and said Dr. Lau we were wrong the experiment didn't work the biotin linked cow protein and the biotin linked insulin also went in. I said Mark you have just discovered a a vitamin uptake pathway in plants the plants are taking up the biotin because they can take up a biotin linked cow protein a biotin linked human protein and biotin linked bacteria. To make a long story short we decided to find out if animals had that capability to so we looked at animal cells and then we jumped to folate another vitamin and by the way animal cells did have that and when we jumped to folate we found out that only cancer cells had the ability to take up folate and that was a breakthrough by accident I must confess but that started my first company endocyte we used folate to deliver attached drugs very selectively to cancer cells avoiding uptake and the associated toxicity when you know a good drug goes into healthy cells and that made a huge difference and so the whole process got started by an accident but the outcome was that we noticed that something useful could be developed out of that accident and from folate we went to lots of other homing molecules and now we have a whole toolbox of homing molecules that we can use to target drugs to as I said not only cancer cells but cells involved in all the fibrotic diseases the autoimmune diseases the cns diseases bone fractures even inherited diseases and things of this sort so we're covering a whole realm of human diseases based on the principle that if you can concentrate a good drug specifically in the disease cell and not have it taken up in healthy cells you'll improve the potency of that drug and reduce its toxicity to healthy cells make making the drug far better than it was before and all six of my companies are are founded or built on that principle wow and so through all this trial and error and all this innovation and science that's exactly like you said that's how science works so that was a great story thank you for sharing um so this drug just went through the clinical trials process can you explain everything that goes into that yeah um well initially the whole discovery process uh is usually begun with a concept that you have for example a homing molecule and why not use that to color cancer cells or make them glow like light bulbs so that the surgeon can see it and so the first step is to do the chemistry in the laboratory which we did and the second step was to culture cancer cells in a little dish and what we did the initial study that we did was we cultured cancer cells in a dish in the presence of healthy cells in the same dish uh and they were human cancer cells and human healthy cells we added our tumor targeted fluorescent dye let it sit there and and bathe all the cells both the cancer and the healthy cells and then after a few minutes we washed off the the solution on the top and looked at them under the fluorescent microscope and only the cancer cells glowed the healthy cells did not so we thought by golly we've got something important there right then the next step was to take it into animals and the animals that we used we implanted into them um human cancer cells so we used special mice actually that would not reject the implanted human cancer cells and we looked to see when we injected into the tail vein of these mice would this fluorescent dye only go to the growing cancers and indeed we confirmed that they just went specifically to the growing cancers then we went to some naturally occurring cancers in dogs that came into the veterinary clinic and just looked to see if we could help there and helping the surgeons there find the malignant lesions in these pet animals and it worked there and then we had to prepare it for human clinical trials that's a long and whole process you have to demonstrate that you can reproducibly manufacture your drug exactly the same many times over so that they know that if you run these trials every patient's going to get the same molecules next you have to show that it's stable then you have to show in two different animal species that it's not toxic so we did it in mice and in dogs then you have to show that it can be very antiseptically bottled and stored and that it's stable during storage and and and filling and we had to do that and finally we had to submit what's called an IND in an investigational new drug application we submitted that to the FDA with all of these and obtained permission to begin trials in humans but then when you start in humans you have to start at a very very low dose and a dose that's far too low to even have any benefit so the first few patients received so little of this targeted fluorescent dye that you can't see anything but we just wanted to make sure that they didn't have a fever have accelerated heartbeat or had you know any adverse events and so we gradually escalated the dose and in the process we reached a concentration of drug that was very effective in revealing the tumors but not causing any associated collateral toxicity that that ended phase one then phase two clinical trials focus on finding really the precise dosing conditions more exactly and to give you the best contrast between the tumor and the healthy tissue then the phase three trials were just to test in a large population of patients that exact condition that we believe works best and then we test that in 150 to 100 patients and we show that we are able to find malignant lesions in these patients that would otherwise gone undetected and then that results in the ovarian cancer trial that has completed and was given expedited review by the FDA because it looked so impressive the results were very impressive in other words the surgeon never knew it was there turned on the fluorescent lamp and wow there's some extra cancer in there they cut it out they sent it off to the pathologist the pathologist says yes that's cancer and so that confirms that the dye enabled the surgeon to find cancer that was otherwise undetectable in the patient i have so many follow-up questions i'm not sure which one to ask um how did you feel when this was finally you know being tested on humans and like you said it's a matter of it's quite literally a matter of life and death especially in these aggressive cancers like ovarian cancer what what were you feeling like when this all kind of was coming to fruition well it was i must confess a eureka moment um bringing it to this stage was certainly a difficult journey i was not very familiar with the process and not not at all actually and so to be honest i and my colleagues at both endocyte my first company and on target labs the second company to some extent went through the school of hard knocks making mistakes that a more experienced drug discoverer wouldn't make but um i learned a lot in the process i'm not making those mistakes with my subsequent companies and i think that i've you know what i've learned will greatly benefit me in avoiding similar pitfalls in the future and moving drugs more rapidly from discovery through to a clinical application and so with my more recent drugs in these other areas where the sailing has been a lot smoother let's just say that sure sure i took the data and showed it to a lot of surgeons and it was interesting the surgeons told me oh we don't need this we can find the cancer very easily we're we've been doing this you know for all our lives and i heard that over and over again for about eight years i would present the data showing these very specific fluorescent images of animals and where only the tumor floresced and they in in no way disputed the fact that we could cause the tumors to floresce but they were not convinced that they would help them and then a surgeon over in the netherlands saw the data at one of my talks and said oh this is terrific let's do something with this and so they introduced it into human clinical trials and it was in ovarian cancer patients and we published the data in a very prominent scientific journal called nature and the results showed that they were able to find five times more malignant lesions with the aid of the florescent diamond without it and that a hundred percent of these fluorescent lesions were cancer and did you bring that back to the us and say see i told you that really that really created a stir i mean it really proved um unequivocally in a well care well run clinical trial that the surgeons really couldn't find all the cancer and that there was a lot that was being left behind that they didn't know about and so that changed actually the field that that paper was the first paper using a a tumor targeted fluorescent diamond humans and um so at that point we were able to raise a lot of money and that that was back in 2008 we created a subsequently a much better fluorescent dye and i think we patented that in about 2013 or 14 2014 and that's the fluorescent dye that just received expedited review by the FDA and should be approved within a couple of months so this has been a very long process almost 20 years do you foresee this drug being used in other cancers besides ovarian yeah we're um it will it is being used currently in phase three clinical trials for um resection of lung cancer and the data there are equally impressive but i'm not allowed to disclose them but i can say that with some confidence that it's going to also change the surgery for lung cancer um after that we have a different dye that we've designed that is very specifically useful for prostate cancer and that's gone through the phase one clinical trial stage uh and the data are looking very promising there we have others lined up in um in the uh refrigerator basically that we've designed and tested in animals that look very good for other cancers and i think very soon we'll have a toolbox of these fluorescent dyes maybe four or five that collectively will cover essentially all human cancers it was there a reason that you chose ovarian cancer in particular to start with um yeah there were a couple of reasons first of all ovarian cancer had this the receptor we call it that was recognized by this smart fluorescent dye um the second reason was that ovarian cancer is a very um insidious cancer it um emerges or is first detected usually only in stage three or stage four where it's already spread and i think the average survival is five years and um that's because it's been historically almost impossible to find the the metastatic lesions it spreads throughout the peritoneal cavity and little uh small malignant nodules and uh some of them are buried underneath the viscera and the intestines the walls of the peritoneal cavity on you know in different surfaces and it's just almost impossible to find but if you turn on this fluorescent lamp and we have videos of this you can see the surgeon moving the viscera around there's a fluorescent spot so they cut that out and they move it around there's another fluorescent spot these spots that are these nodules are often just um you know a tenth of an inch or less in size but if you let them go they'll grow huge and um so they would normally go undetected because they look exactly like healthy tissue you really can't distinguish them when they're that small they're not a bump they're just smooth and they don't feel hard or anything like that so anyway it seemed like a particularly um difficult cancer to successfully resect surgically and so it was a good place to demonstrate the benefit of having a tumor targeted for us. Sure so you mentioned before that now that you've gone through this process once with with your other companies now coming up you have um a greater knowledge of all of this where do you think that drug discovery will be in you know five or even ten years? I don't know you know I'm getting old and my days are long still and everybody keeps asking me when I am I going to retire I have a problem I'd like to retire but I have lots of ideas and most of these ideas are pretty good and so I'm not sure exactly um what the future holds for me to be honest with you um I've um you know we've been very successful uh both scientifically and financially and I don't have any problem getting started companies started anymore not only uh do I have a lot of ideas but it's a lot easier once you've been successful with one or two companies to go out and raise money for funding the um drug discovery projects and the additional companies um for example our third company is developing a tumor targeted immunotherapy that takes a patient's immune cells and puts in a new gene that enables them to very aggressively attack only cancer cells and we went out and within a few months raised uh over 250 million dollars for the company oh my and when we first uh just in comparison when I found at endocyte my first company it took me quite a while to raise one and a half million dollars so now you're just a prop so you know we're well it's uh it's easier once you've been successful uh the major investment banks and venture capitalists call you instead of you calling them it's going right back to your dad being a famous chemist it's another famous chemist so um what do you think the most gratifying part of all of this has been for you well I I think it's been very gratifying to be able to help people um I just uh watched a video of a lady from the Philadelphia area who was so thrilled that uh the surgeon there Janos Dr. Janos Tanya was able to find uh significant extra disease tissue in her that would have gone undetected otherwise and uh she listed off all of the different activities she was engaged in and how important her you know her ability to interact with these um charitable organizations and these volunteer activities and so forth and her grandchildren and so forth it was I was just uh I guess overwhelmingly uh rewarded by just seeing that you know what we likely did there was to give her an extra you know maybe 15 years or whatever to be able to carry out all of these wonderful activities that she was engaged in I think that's the most most rewarding aspect of it all you know and I've heard you have very strong ties to Purdue you talked about you went to West Lafayette high school your dad was a chemist here at Purdue why did you choose to come back here and work here um well it was the it was it was a good job I mean when I uh when I first started looking for faculty positions um I only had one other job offer and uh it was obvious my parents lived here I grew up in the town I liked uh West Lafayette very much so I decided to come back and uh yeah it was the obvious choice and I'm sure other people were trying to poach you as you got more and more successful yeah I've I've been obviously um given many offered many offers to move to other places and uh I um I grew up a very avid Purdue fan when I was a kid my dad used to usher at the basketball and football games and uh as a good father he bought me tickets so that I could go along with him even I and uh so I went to all the basketball and football games starting from the earliest days I remember and I grew up an avid Purdue fan it's interesting here's a here's a personal story that you might find interesting I played basketball at West Lafayette High School and uh was fairly good okay and um I got a basketball scholarship to BYU okay and I did play there okay and uh when um I got my job at Purdue um shortly afterwards BYU came out to play basketball here I had tickets for the game and uh so um um I wasn't sure who I was gonna root for and when the uh opening tip-off went up uh immediately I was a Purdue fan did you wear Purdue colors? I don't think I did it myself at the time okay but I do now you stayed neutral okay full wardrobe and as a matter of fact all of my kids have Purdue rooms and their homes where they have all the Purdue paraphernalia that we send them and so forth on the walls so are you still an avid sports fan here yeah I I I have tickets to both basketball and football games and what do you think um like when it comes to drug discovery work being done at Purdue versus other universities how does Purdue stay at the forefront of all of this? Well um I think Purdue made a decision and maybe it was just intrinsic in the nature of the university many many years ago to embrace the use of science for practical applications uh maybe this stems from the heavy focus on engineering which is really an application directed science um maybe it's was the decision of earlier administrators I don't know but in many uh schools finding a useful application for your discoveries is considered prostituting yourself for filthy lucre it's it's beneath the dignity of a good scientist to do something of that so in Purdue it's not crowned on as a matter of fact it's encouraged now in all fairness I will tell you that that attitude has changed 180 degrees at almost all the universities but when I first found at Endocyte at Purdue even Purdue was skeptical about the um compatibility of having a faculty member also engage in private enterprise and I had to go up through the department head first who discouraged me from doing it then to the dean who also discouraged me from doing it it went to the president and the president had just returned from a trip out to Stanford where they were already beginning to do it and he said well why not let him try and so I was given permission and we found at Endocyte I think it was the the first or not one of the not the first one of the very first companies it was 1995 and it was one of the first companies to be founded based on Purdue technology and since then as I say I've had over 500 patents all of almost all of them through Purdue University and Purdue is going to benefit financially a lot from these I think but also I there's been a lot of benefit to the university in terms of the training that students have obtained in my lab and in terms of the visibility that it brings to the university in terms of the money it brings into the university because it's my lab is extremely well supported now we occupy this entire top floor here and that money has come in because of the interest from industry and supporting all the work that we're doing and you know I think it's a win-win situation for the faculty member and the university to promote this within the you know appropriate limitations and restrictions I think those are important to just imagine that if you had stopped when those two levels of people said don't do it I mean look at all these things you've created so what would you say to a future student who might be thinking about coming to Purdue to pursue science oh I think Purdue is a great place for an education on science I think it's terrific they focus on the STEM curricula and the I think the atmosphere of encouragement for translating your discoveries into something useful is very prominent and helpful here my personal belief is it's the only way science is going to survive in the future the I think we have to demonstrate that public support of science is going to pay the public back it's in their benefit to do so and for many years it wasn't obvious we would discover new fundamental principles publish them in journals using language that only your colleagues would understand and no one who you know worked at the local factory and you know came home and watch ball games on TV had any idea of how that might benefit them and frankly in many cases it didn't but now with this emphasis on finding some practical use for your discoveries I think it will become very obvious that Purdue University and other comparable universities can become an economic engine for the state and local area and what does it mean to you we touched on this a little bit earlier but to to have all this success and you know like you said financially and you created all these different things but more importantly you've saved and you're going to save hundreds of thousands of lives what does that mean to you well I mean I I don't know how to answer it any differently than it is very gratifying it's very rewarding and it's nice to know that you have can leave a footprint on the planet when you leave and I think we'll be able to say that we've done that and feel comfortable that we've made a contribution to humanity and this is an easy one why are you proud to be a boiler maker I like Purdue yeah Dr. Lauer where does the name Cytolux come from well the prefix Cyt comes from the Latin for cell and lux comes from the Latin for light and so what you have is a lighted cell and that's where the name stems from one has to get these names approved through the FDA you can't actually claim any outcome or benefit with the name so usually go to companies that know what will be allowed and what won't be and you select four or five and send them to the FDA with your preference and the FDA that approves them or disapproves them and this was approved I like it I don't see it's easy to say some of the jugs that come out to me are like what so okay well thank you so much for your time I think it was wonderful um I am a true boiler maker that's a Purdue fight song on my cell phone I love it


Today on Nightline, a huge addition to the night family, a CMN road trip, and another highway hero. I'm Megan Phillips for Nightline. Two weeks ago, Nightswitch Transportation agreed to acquire US Express. The full truckload carrier, which is headquartered in Tennessee, has 14 terminal locations and operates a fleet of approximately 7,200 tractors and 14,400 trailers, primarily across the eastern and central parts of the country. They will continue to run as an independent brand and employees will continue on as usual. As with every acquisition, our goal is to make their brand the best version of themselves, not to simply make them part of night. We hope for this process to be finalized by the end of the second quarter. We welcome everyone from US Express into the night family. Glad to have you on board. It's Children's Hospitals Week, and fleet of Miracles driver Rosalinda Tejada and charity coordinator Whitney Buck took a load to Orlando, Florida for the Children's Miracle Network 40th anniversary event. I've been with Night Transportation going on over two years. I'm also a trainer here. What's not neat to drive for the fleet of Miracle is that I actually did some volunteer work with the Children's Hospital, and I'm also in the vaccination sites out here in Arizona. And I am riding along with Rosalinda out to Florida, and I am going to be more excited to go and deliver all the amazing things that are going to make Children's Hospital Week fantastic this year for their 40th anniversary. And here's our trucks. We've got Swift right here, and then we also got Night Transportation, and we got Whitney. Since 1983, CMN has raised $8 billion to support Children's Health. We're proud to partner with them and hope to raise even more in the years to come. Justin Kilbane was driving through Flagstaff with his girlfriend when he slid off the road and into a four-foot deep snow bank. Stuck in the snow on the side of a heavily trafficked interstate and unable to get himself out, he watched us truck after truck pass by. That is until a shiny red big rig pulled over next to him. James Reeves, a night driver out of El Paso, pulled over to help get Justin out of the snow. Justin said about James, he was a wonderful guy as nice as you could possibly get. We appreciate James for his empathy and kindness. As an RV veteran and now a truck driver, James has a service and duty-driven character. James has been a part of our night family for only a year, but has already proven to be an invaluable team member and friend. That horn means it's time to enter the speed zone. Facts flying in at 65 miles per hour. The next drivers' dine in and drive will be Friday, April 7th from 11 a.m. to 1 p.m. Pick up some free lunch from any of our terminals. The Truckload Carriers Association Scholarship Fund application is now open. The fund awards up to $6,250 per year per full-time undergraduate college student. Anyone who is an employee or the child, grandchild, or spouse of an employee can apply. You have until May 5th to apply. Check out the link below. And the Randy Knight Scholarship deadline has also been extended until April 16th. Any ASU student who is the dependent of a night employee can apply. That's two scholarships. Don't miss out. The winners of the biannual driver engagement survey are Hayden Crook from Phoenix Dry, Matthew Renteria from Tulsa Dry, Cherish Gray from Denver Dry, Christopher Jones from Lakeland Refrigerated, and Latasha Bell from Houston Port. Enjoy your gift cards and thank you to everyone who submitted. We appreciate the feedback and look forward to making positive changes. Next Step Load Preview is now available on KT Mobile. Check out the next step button to view the macros on the home screen of your tablets and personal devices. Make sure the information submitted is correct before moving on to the next step. The button will not change until it's correct. Hit the button as soon as it's appropriate and valid timestamp data is sent electronically to the customer. Have you taken this month's safety course in Night University yet? The topic for April is the CVSA Road Check Event. Remember, taking this course is required for you to qualify for the safety component of your monthly bonus. Don't miss out on more moolah. We have some old faces and new places. Mike Yase will be the VP of Planning and Optimization. Ren Night will be VP of Operations for West Dry Van. Mike Webb will be the VP of Operations for East Dry Van. Drew Zveck is the new Terminal Manager for Dallas Dry. Tyson Kitana is the new Terminal Manager for Las Vegas Dry. Spencer Edmond is the new Terminal Manager for Salt Lake City Dry. And Juan Martinez is the new Terminal Manager for Midnight. The goal for the Fuel Efficiency Bonus in the month of April is 7.26 miles per gallon. If you're looking for tips on how to increase your fuel efficiency, we made a video on that. You can find it on the Driver Portal. Congrats to our newest million milers. Each of these drivers have driven the equivalent of 40 trips around the entire planet. How crazy is that? Thank you for your dedication. And finally, congrats to our highest performing terminals for the month of February. Our highest performing large terminal was Memphis and the highest performing small terminal was Las Vegas Ports. Whether you're on the road or in the office, it's a team effort. Nice job to everyone from those terminals. Are you subscribed to Night Transportation on YouTube? If not, what are you waiting for? Get notifications for these videos and all of our latest stories. And check us out on Facebook, Instagram, LinkedIn, and TikTok so you can keep up to date with everything we do. I'm Megan Phillips for the latest in all news night. This has been Nightline. Good night and good truck.


I'm going to tell you about the success of the Dabawala's We are a 5000 people in our organizations with eight grade schoolings delivering two lakhs tiffins in a day within a period of three hours times We deliver the tiffin boxes to the office going people as well as the school going kids Which is a hundred and twenty two years old organizations? We are doing with non-stop. We are the Mumbai Dabawala's Mumbai's Dabawala's System is attracted too many people in the world. How the Harvard University is attracted They have done one case study which is published in their syllabus on the Mumbai Dabawala for their time management and the supply chain logistic Friends if you see the first picture a person Dabawala's carrying wooden crates on his head Which is weighing 60 to 65 kg in the next picture to the rear part of the bicycles is I have some tiffin boxes Which is an art itself to hang a tiffin boxes 40 tiffin boxes to the cycles friends You don't require any kind of fuel for the Cycles so it's already a pollution free Sorry our belief that we believe that work is worksheets friends Dabawala believe that Customer is the king for the Dabawala. Not the Singh is the king our secrets reveal This is the process that a people B people C people they are collecting your tiffins from your house They go to the nearby investigations there. They will segregate as per the destination wise We have given some coding code to the destinations that is journey road We have given the toll numbers accordingly. They will segregate the tiffin boxes from there that one number person will come to the Particular stations. He will pick up your tiffin and deliver it to your offices These are some coding systems that this is the upper part of the Dabaw Vlp stands for the will a parallel investigations in suburban in Mumbai that e person will come to your house He will go to the will a parallel stations from there. He will put your tiffin on the three-number bucket Which is going to a church gate There that nine-number person will come to the station church gate stations He will pick up your tiffin then he will go to the express tower EX stands for the express tower building which is in Narivan point on the 12th floor He will keep your tiffin box outside your office But even though daily having this was a his own tiffin. He's not able to identify his own tiffin That is why the name this was is written there Friends if you keep a thousand tiffin at a place, you will not identify your tiffin But if you ask any Dabawala, he will give it to your tiffins only These are some examples like that GS stands for the Ghatkopar railway stations BO stands for the Boriri railway stations T stands for the Tane railway stations Friends the precautionary measures you have to take no things has been taken granted in our organizations So it is the first-man duty to see that in transit your tiffin is handled by six to seven people as well as in the house You wash it and that time sometimes number become invisible. So it is very difficult to us also give a proper delivery So it is the first-man duty to see that number should be a visible if it is not visible He will go to the paint box. He will buy a paint box and with the finger. He will write the numbers there or Friends because we have a shake by our customer as well as the jay by our customer What will happen the shake-wise custom tiffin goes to the Jain by so that is why we have to take the precautionaries So we are going into the traveling into the local local train in the Mumbai So 12 coach train 4,000 commuters travels 8,000 disputed happens, but no excuses duty comes first This is into the local compartment after entering into the local compartment This is outside the church stations. They are just rearranging the tiffin as per the destination wise in the second feature They are just going on the delivery They have put at the tiffin boxes into the crates After finishing your lunch frames again your empty box goes to your house So this is the return journey collection of the tiffin in the first picture is going on and in the second picture They are just segregating as per the destination wise If you see in the picture the friends on the second feature that person has set on the rack and making some notes We are very disciplined people's In our organization leave has been not sanctioned without any prior notice wearing white cap is mandatory So it's just making that note was not where the cap who has remained absent on without prior notice So in every 15th in our organization office. We have the meeting that time. We are going to penalize these kind of people's This is our discipline during work hours smoke smoking and the alcohol is not allowed Wearing white cap is mandatory. No leave without any prior notice has been sanctioned. We provide employment to our citizens You know citizens also we have in our organization 72 years old of people's in our organizations We are happy to live in the Dubai to the every customers even in the times of the recessions and the depressions These are some facts about our organizations, which is a hundred and twenty two years old organizations friends We don't use any kind of Technology for because we have illiterate people in organizations for illiterate people's his brain is the Computer for him and this is a super computer. Once you cover your super computer with this kind of cover nothing Friends error rate is one in sixty one in one point six million transactions My father says that error is horror and don't look into the mirror So we have honored with the six sigma certificate friends. Let me tell you we have not applied for the six sigma But six sigma comes to us We are charging very nominal fees for the service that is three fifty to four hundred rupees per month Year later on over is 45 crores for the organizations per member is now getting seven to eight thousands in organizations And everybody is getting the same contributions right from the president to the worker levels So this is one of the I Told you about that the most of the world's people Get attracted from the Dabawala systems. How the Prince Charles visited on the 4th November 2003 Here among one of the against the attracted persons Richard Branson's here literally traveled along with the Dabawala's into the local compartment wearing white cap and having the Tiffin box in his hand. He has taken these kind of snaps and the photographs and he said that I'm just going to Display these kind of photographs in my worldwide offices to convey the message to my employees that work like a Dabawala's So, thank you friends


Hi everyone, my name is Jackson Huff and welcome to this presentation at LinuxCon 2022. In this presentation, I'm going to be making the Linux Performance Counter API really easy. So I'm Jackson Huff, I'm glad to be here at LinuxCon. We're going to be going through these slides really fast, so get ready. So this presentation about the Linux Performance Counter API is really important and interesting because there is no other Performance Counter API or PC API for short tutorial on the internet. That's crazy.  Maybe you'll be able to make your own after you watch this, but either way, I had to figure out all this out on my own, and this is a shocking discovery I made. And because there's no other tutorial like it, there's also no other demo program like this, like the one you'll see very soon. So before we get started, here's some quick self-promotion. So I, Jackson Huff, am a high school graduate in Orlando, Florida, USA, Planet Earth. I'm planning to attend the University of Central Florida later this year. I'm planning to study engineering and maybe music or Spanish. A fun fact, some people think I'm from Puerto Rico, but it isn't the truth. Pero no es la verdad. Just something you might not have known about me. I've also been involved in IT for a very long time. I've been in Linux for almost as long as that, probably over a decade in IT. I've known about Linux for about 10 years now, and I've been learning about Linux for all that time. And because I've been involved with IT for such a while, I've got to learn some stuff about networking and now earn me the CCNA certificate in August last year. Finally, you can check out more about me at my personal website at JGHUF.com, where you'll also be able to check out some of my other websites for other projects of life. Okay, so here's the painless overview of this painless presentation. We're going to start with the introduction to a PC API, performance counter API, why you need the PC API, how PC API works, implementing the PC API, the demo program. Got to put that in bold because that's going to be the coolest thing here. Brain dump, how can we take PC API even further? Brain dump? Well, if you're familiar with the CCNA, that's kind of a bad word. But this, but here it's a good word because brain dump lets us, another term for it is recall practice. It helps us remember important concepts and use them for the future. After that, finally, is the conclusion to this presentation. Okay, check OBS. Three minutes. All right, we're making good time. Here's a group question. What are performance counters anyway? I'm going to be giving you 15 seconds to think about this. Now, if you haven't heard about performance counters before, think about what the term performance counter means. And if you already know what performance carers are, I encourage you to think about what are they anyway? Okay, 15 seconds, go. All right, 15 seconds are up. So what is a performance counter anyway? So performance carers are a way to let developers and their designers check how fast CPUs are going objectively. They measure objective and truthful metrics in comparison to what a lot of marketers would want you to think. So CPU isn't fast or slow, 20% faster or 50% slower, but rather, the performance counters reveal all the metrics you could possibly want to objectively measure performance across processors. Phew, what a mouthful. Here's a quick, so from coughing up, started from coughing a lot because I have a cold and if I need to blow my nose, there'll be a jump cut. Sorry. So here's a quick history about performance counters. They started with a secret Intel processor feature. We're talking about like Pentium's from the 1990s. These processors had a few certain undocumented instructions and some clever people figured out with reverse engineering that these secret instructions revealed secret registers that revealed all the statistics you could want from performance counters and these performance counters expose certain useful statistics that the designers at Intel probably could have used to design better Pentium's, but they never revealed it to end users. However, these clever people made it so this new feature was so useful for end users, Intel documented them, AMD copied Intel, ARM copied Intel and AMD, and the rest is history. All right, quick. Yeah, okay, sorry. So what do you use performance counters for? Well, if you're a software developer, perhaps you've heard of something called cache optimization. So cache optimization is when you say sequentially align variables or compress your data so that all of the data fits in the cache or perhaps you're compressing down instructions so they all fit in the instruction cache. Either way, that's cache optimization. It'll make your programs faster. Performance counters can less check how exactly how optimized something is and performance counters less optimized, optimized for not just the cache but the TLB pipeline instruction pipeline install amounts and more. So performance counters can less radically improve our performance. And that's going to be important as we'll see here why we need better performance. So today, ironically enough, even though our CPUs are orders of magnitude faster than those original Pentium in the 1990s, our programs are even more orders of magnitude slower than they were. So over here we've got bloated electron apps like a glorified text editor or a chat app. What about streaming video delivery? Well that's actually a better use for performance counters because a little change here and there can add up to a lot of change overall. So for example, Netflix could save millions a year, millions of dollars a year if they just made their performance counters able to be. If they made their streaming CDNs more optimized, say 1%, that 1% adds up over their thousands of servers. I can't talk today. What about high performance computing? So HPC for short is probably the biggest use of all for performance counters and that's because if you can optimize your programs then your task will be solved faster and you'll be able to save millions of dollars because often you're using big supercomputers maybe cause billions and these performance counters will be able to let you save time using them. That's a win-win. And finally you'll be able to know the nitty-gritty about all your programs and how they behave. And actually there's a fifth reason why you need them that I didn't include here but the statistics from performance counters just look cool to a lot of people including me even though I know exactly what they're measuring. So that's another win-win. So here's the Netflix success story. So for a while Netflix suffered from unoptimized video CDNs. I digress. I already said that before. So to fix that Netflix hired someone who knew performance counters really well. I can't remember if it was Linux, 3BSD or something else but performance counters worked the same across all OSs because they're specific to processors and not the OS. So some testing and experiments and bar charts later Netflix is reaping the rewards and rewarding their stockholders. That's because they optimized their video delivery and efficiency metrics. So I could say they applied cache optimization. They improved the IPC. Actually the IPC measures overall processor efficiency. Cache optimization is just one means that I end. And you can measure both of those with performance counters and that's great. And today this person who I can't remember the name of is a legend in the performance optimization world. Okay. Now let's start with a PC API secret. It's not all about the Benjamins. It's not all about the Pentiums but rather it's all about the SysCalls. And that's because the only actual interface with the Linux kernel you use with PC API is with the SysCall using nrperfect that open the read function and I forgot to add it here but you also use the IOCTL SysCall to interface with Linux kernel to use the performance counter API. Wow. So these three simple parts are the only actual, how should I say it, connections to the kernel that you have to make. So if you're worried about, makes kernel interfaces or something like that you don't have to worry so much. It's just how you use the data you get. So here's another PC API secret. Until now you need to cast a memory buffer to devoid type, void C type, to devoid C variable type. Wow. I'm out full again for almost all reads you have to do in order to get data from your performance counters. However, modern C++ is here to help with these ugly memory conversions by giving us explicit conversions and a generic code, C++ are called templates, other languages called them generics. I prefer templates because they tell you what they are more. And so these two factors are the secret behind agile PC API use. Now predicting in the future it's going to be nothing but generic code in these explicit conversions that make PC API widespread. I'm hoping. My code's ganged to me, sorry. So here's another group question I want you to think about. Why have there been zero tutorials on PC API until now? Actually, I already asked you this, but I'm giving you another 15 seconds to think about this one. Okay. Get set. Go. And that's 15 seconds. So what's your answer? You might be able to put it in a chat because I'm recording this in the past. I don't actually know, but put your answer in the chat if you can do that. Thank you. So here's how the PC API works. Here's the big overview, big bird's eye view, how you use the PC API. So first up, find the pit of your process, whether it's user input or generated somewhere else in the program. And in this case, we're going to be doing the pit of the process that is not the process using performance counters. It's tricky to explain, but it just has to be a different pit than what the program running it has. Okay. Second step, find all child pits. This is surprisingly tricky, but there's an easy modern C++ solution as I'll show you later in the demo. Open a file script for each event for each child pit, and this presents another problem that I'll show you later how to fix. Fourth step, reset and enable counters. Really easy. Fifth step, wait. Really easy. Sixth step, stop counters. Still easy. Seventh step, read file scripts into buffers and read as buffers into usual variables. Tricky, but if you just copy my template, it just works. Last step, go back to reset and enable. And that means this is all an infinite loop, which is great. So here's an important choice we have to make. So as you can see here, counting counters count in some specified time period. Technically, whenever they're enabled, you go back here, they would be counting at the wait step. What about sampling counters? Sampling counters count based on some trigger or breakpoint. This could be a hardware or software breakpoint. So to deal with sampling counters is after trickier to use, and they require some really spooky memory management. You've heard of something called Mmap, that's exactly what you have to use. And you have to specify the exact breakpoints that you want to track, which itself is tricky too. So here we're going to be using the counting method for our demo program and the presentation too. Sorry if I don't know how to say that better. But, okay, let's move on. Actually, one more thing I want to say, there's a third type of performance counter, and this type of performance counter interfaces directly with the Linux kernel to measure how fast CIS calls and other low level actions take. However, that kind of counting is even funkier than sampling type counters. So I'm not going to be talking any more about that here. Maybe in a future presentation when I also do sampling counters. I digress. So here's the plan of action for this presentation so far. I'll show you the basic steps that you would use in C first. You can split those steps into C++ building blocks. I should say modern C++ building blocks because they use some funky and but useful modern C++ features. And this demo program will use these modern C++ building blocks that I'll show you individually. Some and because I'm recording this as a video, I'm just going to guarantee that there will be some way to get the complete demo code with the modern C++ building blocks at Linux. I don't know exactly how, but there will be some way to do it. Let's start with headers. So the PC API requires some very specific C headers. So this top header specifies the Nrperf event open macro. Second one defines more than just breakpoints, but also the events. Third one solves a problem that we're going to encounter with excessive file scripted use. Fourth one tells us how to use CIS calls. Fifth one tells us how to use the IOCTL CIS calls specifically. We're already 17 minutes in. Either way, let's get started with structs. So on the left here, the struct called read format helps us, well, it tells the PC API, no, it doesn't actually tell the PC API anything, but it's the format that the PC API uses to store data in a buffer. And when we read a file descriptor into this buffer, it will happen. It will be in this read format. So we're just specifying a struct so we can access the variables that exist within this struct. It's really wrong. I don't like it. What about the second struct here called pcount? Well, this is just a modern C++ abstraction. Sorry for the junk. I just had to blow my nose. So right here, we have the modern C++ abstraction struct. And here, we have the explicit conversion using reinterpret casts from a read format pointer to a read format data. That is funky. And I hate it, but it works. So if you needed just a function to abstract away performance counter stuff, this function would set up performance counters and do the main loop for counter processing, as you seen earlier here, where you have a reset enable counters and this 2D read FDs into buffers. So the demo program is just this function, but put into main instead. So everything in that program could be put into its own little function. Or should I say big function, but either way, it'll work the same. So what comes next? A problem. So how do we solve the problem of needing all the child threadpids of our input process PID? So I discovered on Linux, a PID is per thread and not per process, unlike a Windows and I believe the BSDs, which is kind of sad. So how do we solve this problem? So if you haven't looked here yet, think about it for a couple of seconds. Okay, so here's how we do it. We use a C++ directory iterator in regents that parts this directory. So plug in PID, say 1, 2, 3, 4. We parse this directory and we count up the number of entries found and we get the names of those entries and we return a list of all the entries. And because of those entries are child PIDs of this example here. And because this is so hacky, I believe Linux needs a better solution now, today, not tomorrow, today. On the other hand, my hacky solution works perfectly. And on my Zen free software development monster, it only takes two milliseconds to complete. I believe on my Intel laptop, it only takes three milliseconds. So it's fast either way. So what about problem two? Each of them per counter uses a single file descriptor. So I want you to think about what this actually means. Each event per counter uses a single file descriptor. And each counter is per child PID and each child PID is per thread. So imagine if we had 200 threads in a program and we wanted to measure 40 events in total, then that's 40 times 200 and if you don't know math, that's 8,000. So we need 8,000 file descriptors to store all those performance counters, crazy. So this prevents this presents a problem because by default, non-Linux users have a generous yet limited resource limit for file descriptors. And I believe on my system, it's 4,096. So not even close to 8,000. This means we run out of file descriptors and we track a program with oodles of threads and some programs like Minecraft servers come with hundreds of threads right out of the box. So oodles of threads isn't particular unusual here. So how do we solve this problem? We just get the hard limit and we resize the soft limit to it. Really easy. This implementation is so easy, I'll show it to you in the demo program. All right, moving on to 23, problem free, how do we decide what we want to measure? Answer the PCAPI docs providing an immensely long list of event macro options. Wow, is that wordy? So an event is one metric beach can measure. So on the right here, we can see for the type perf type software, that's a macro itself. We can select one of these config options, which is an event. So we can do perf count SW CPU clock, which measures the CPU clock. We can do task clog page folks, contact switches or CPU migrations. Wow, and that's just a tiny sample of what you can choose from because we've got at least 40 events in total. So you're covered in any either way. So how do we set up our counters? Easy, just fill in this template here for the initial sys call that returns the file descriptor of our counter. So file descriptor is a long end NR blah, blah, blah. Tells the Linux kernel that we want to make a performance counter. Settings is a struct that holds our performance care settings. And as you'll see later, this is kind of, we don't need to define the struct in our code, but it's a first struct that we use to go past settings. PID is the PID of our target thread, or process if your process is single threaded. Negative one means that we aren't targeting any particular CPU core. So you can set this to zero or above the target or particular CPU core of that same number. GFD is our group leader, PID, or set it to negative one to create a group leader. So all covered group leaders very soon, zero, whatever goes in zero is for sampling counters. So how do we handle errors with performance counters? Because as it turns out, performance counters on Linux are really flaky, and I made it. But it's easy, just check the file descriptor returned by this function is equal to negative one. So if it's zero or above, then it worked just fine. If you want to do an additional check of the error no variable to figure out the exact problem, and as it turns out, there are lots of exact problems you can do, I've covered most of them in a demo program. And it's important to check for errors because performance care errors can cascade the other counters, easy. So let's configure our, oh, yeah, let's configure the event. I shouldn't say struct conflict, because it's just one detail of the struct. So the PC API uses C macro constants to store the event type. Events are what you want to measure, and some events are really strange. And these all have to do with cache type events. I don't know why they do it this way, but here's how it works. So you select the type, per type hwcache, or yeah, hwcache, you select what type of cache you want. I'm selecting the data TLB, and then you have to bit shift this macro here, left 8 bits, and that's how this double Chevron notation works. It bit shifts whatever is here by this amount of bits here. So it bit shifts per point per count hwcache up right, left 8 bits, and it shifts this thing left 16 bits. Wow, what a mess. So overall, this measure is data TLB cache write misses, easy. So how does the group leader work? Group leaders provide handy abstraction to make counter control a lot easier. So to wait, so for group leaders, consider a group leader like a mini abstraction to make enabling and disabling easier. Well, we just did something weird. So where was I? So group leaders are like an easy abstraction for file descriptors. When called, they act on behalf of all members of that group. And an additional advantage of group leaders is that all members of a particular group will be scheduled on the CPU at the same time. And that means you can do accurate addition, subtraction, multiplication, and division operations without having to worry about the accuracy. Tricky. And sorry. So how do you use group leaders? In the syscall here, set negative 1 as the group leader pit to create a new group leader. And then this return value is your new file descriptor for the group leader. And then you plug in that file descriptor, not a pit, as I mistakenly said, in order to make a new, to make an event a member of that group. This is really tricky to work. I'm sorry again. Then enter the file descriptor of the new group leader in the same parameter in future counters. OK. And as I said before, group leaders are all or nothing. If one counter fails, all of them do. And that's because they all have to be scheduled on the CPU at the same time. So this is an unfortunate side effect of that. I'm sorry to say. So what about the infinite loop? Go to infinite loop once you make all the counter events for each pit of the process. This loop exists for as long as you want to monitor performance using performance counters. So that means if you want to, say, stop using performance counters, you would call all the counters as you see in the demo program. And then you exit the loop. So just simple break in CRC++. Also in C++, simple while true here works absolutely perfectly. Yay. So next step is to reset and enable our counters. So what's created are counters are in a state defined by what you can figure out, but their values are undefined. So how do we fix that? We first reset the counters so their values are a defined value, zero, and then we enable them so they can start counting. Here's how that works. Call, use the IOCTL syscall for all file descriptors that are either group leaders or do not have a group leader. So what does that mean? So when you call IOCTL for group leader, all members of that group will be affected. However, some counters might not be in a group, so you call them separately as well. So use these two calls to reset and enable. IOCTL, FD, blah, blah, reset. And this here tells us that we want to affect all members of a group instead of just one. Same except for enable in order to enable the performance counters. So now we sleep. We just let the Linux kernel code collect data with performance counters on the processor while we do other stuff in our program. We'll want to wait some reasonable amount of time, such as five seconds here, and an easy cross-platform way to sleep. Working Windows, macOS, Linux, anywhere is with this and C++.  What about disabling the counters? Disabling counters is also easy. It's the same idea as resetting and enabling just with a different macro. So IOCTL, FD, blah, blah, disable, affect the whole group. Easy. Reading, how do we read from these performance counters? So we're going to want to use the read. Is it syscall? Yeah, it's a syscall. And the return value of this read is going to be a long. So you're going to want to make a long size to store the size of the amount of data read by us. Invo, it's going to be no larger than an int. Long is just a saver. So you're going to be reading from the file descriptor of every counter. You're going to be reading the data into the buffer. And you want to read maximum of the size of the buffer. Read a maximum amount of data of the size of the buffer. I need a better way to say that. And you have to perform a memory check. If size is equal or equal to or greater than this equation. So the way this works is, if you remember earlier in the struct, there were two values for each entry in that array in the struct. Because both of them were long, longs, that's 16 bytes in total. So 16 times the number of events in that group plus another long, long for the whole group. So that's the equation there. If size is equal to greater than that, then it's viable to extract values from. Here's an example here. So reading part two. Now we iterate over each NR, which is the number of events in each read format struct. In that iteration, check the ID for a red value. Not a reading value, but the value that you just read is equal to the ID of the desired event ID. So if it sounds so technical, but that's exactly what we're doing. Then if this check is true, assign a user variable the read value of that ID. So we don't have this part here, but we do have these two elements here in this code. So we're doing check for greater than or equal to the equation. Now we iterate here over the buffer. We check if the ID is equal to the known good ID. And then we assign a variable to value of the thing pointed to by the ID. It's tough to explain, but as you'll see in a demo program, it's actually really easy. So how do we calculate the PID deltas? What is a PID delta? It's actually just the difference of PIDs because many programs dynamically create and destroy threads. And because they're dynamically creating and destroying the threads, that leaves performance counters trying to track PIDs that either don't exist or they aren't tracking PIDs that do exist compared to maybe five seconds ago. So here's how we do it. The complete, you'll be able to see the complete process in the demo program. Man, that demo program is getting legendary now. Here's how you do it. There's a lot of current PIDs, blah, blah, blah. I don't actually have enough time to say it all because we have to make time for the demo. So one more thing regarding performance counters is that the PCAPI causes security issues. Although the security issues might not actually be important because the only viable attacks known so far are proof of concept attacks. They can only extract a few bits, not kilobytes, not megabytes, but just a few bits of memory from other programs by using performance counters on those programs. And because it's a non-zero security risk, Linux locks the PCAPI for PIDs, Overt, and the column process. Remember, tough wording, but if you're trying to measure the process that you're using performance counters in, you don't have, there's no security issue there. So there's no extra steps you have to take. But if you are doing PCAPI for PIDs, Overt, and the column process, it, Linux locks it behind the cap assist admin capability, or since Linux 5.9, or maybe 5.8, I can't remember, but I believe it's 5.9, it's the cap perform on capability. So how do you do, how do you keep good security? First you should use a Linux, use a re-rescent Linux version as you always should, but you should also assign the cap perform on capability instead of assigning the cap assist admin capability or running the performance counter code as a root. So for my demo program, try using this command for it to add the cap perform on capability. So sudo, set cap perform on, forgot what P and E stand for, but you need them, and demo, which is what I call the binary of my demo program. So you might not already have set cap installed, but it's really easy to install. Okay, let's move on. Actually, that's all folks, that's the bare bones basics of using counting type Linux performance counters. So I might be covering sampling type counters in the future, but for now, we're just using counting type. So it's time for the demo I was telling you all about. Let's take a look at, let's put all of these concepts together using smart C++ building blocks in order to get a user generated PID and monitor another process. And the stem of steals almost all of its code from a real world project of mine. But I'm just leading the unimportant stuff to make a shining gem of something that you can learn from. Okay, let me fix OBS so you can see the program and I'll be back. All right, I'll work back. So here's a demo program in VS code, one of those bloated electron apps I mentioned earlier. Well here, I've got the whole program ready to go. So we start with our, we start with all of our headers, then we move on to all the C++ standard libraries, double check something, yep, it's going, then we have a classic main space FS equals STD colon colon file system that just makes the code cleaner. Then we have our instruct read format that we define. So the compiler knows on how the data is going to be structured when we get the raw data in the buffer, then we have our P counter abstraction here. So what this does is provide and it abstracts all the events that we want away. So we can just think about it per PID instead of per event per PID, that makes it a lot easier. So these four values store the event, the event ID, event value, and event file descriptors. And the buffer here, here's the buffer size equation. It's actually identical to the one I showed you earlier. So it's maximum bounce counter time 16, all plus eight. Then here's that explicit conversion using reinterpret cast. So here's get process child pins. Here's how that works. So we, we start by making a list of all of our PIDs, then a regex with proc some amount of digits, then task. We optimize this. Then try catch just for errors. Then we iterate over, over the directory using the final system directory iterator. And we select the directory proc, the PID we want to get all the child pins of plus task. And this is our directory entry here, constant auto reference directory. And then we add to PIDs using in place back, convert to a long and replace the whole path here with nothing. It actually just works. Then we return the whole list of PIDs. Easy. So let's set up a counter. I have a lambda function here to initialize all the arrays because I normally start with unknown values. Then we configure a struct because we have some common settings here. We have, we abstract everything into a lambda instead of having to configure a struct for every event we want to use. So we do perf type size of blah, blah, disable true. That's why we need the disable and re, no, the reset and enable. I have CTO pair early on. And then we specify we want to make, we have to use groups and the ID system. Then we set up the event. So we have the classic syscall template here. We check for errors. Now here's all the possible errors. There's a lot of them. Then we, for all of our events, we configure using, we configure the events using these lines here. So I've got two events here, one for CPU cycles and the other is instructions. Easy. We create our counters using a user provided list of counters and the user and the list of PIDs acquired from the, getting the list of current PIDs. So it just wraps for all of the PIDs past. We just set up new counters for all events for each PID. I need water, but I mistakenly put my water cup away. Can't believe I did that. So how do we get real counters when we're done with them? We use something called cold counters. So we have some insanely nested loops here, but it boils down to closing all of the group file descriptors that we don't need anymore. And we erase the whole performance counter from our, from our list of performance counters. Number mouthful. So what about resetting and enabling counters? So that's really easy. For all counters, we iterate for all groups within those. And for both beginnings of those groups, we do reset and enable, we apply it for the whole group. Easy. Now for disable encounters, it's the same exact thing above with the disable ILCTL thing too. Now how do we read from counters? So we set the long size for all S encounters. We check if it's above two. Now I want to tell you something. This was actually a source of days of useless vowel growing sessions because as it turns out, the Linux kernel gets rid of memory for you if you close out a file descriptor that is not being used because file descriptors by default have a value of zero. You know what else has a file descriptor value of zero, the standard input for a program. And because you're closing the standard input, whenever you enter like some string, you get a segmentation fault because the Linux kernel is getting rid of memory that should be there. And so we have to check if it's a sane amount for a performance counter. And I hope that helps you avoid days of vowel growing sessions as well. So after that, we do all the standard procedures that I showed you earlier. We get our size, check size, then we store all the ID values and the values acquired from those IDs into variables. How much time do we have left? We have six more minutes left. Okay. So this is how we solve the resource limit problem. So we set up a R limit struct from the resource inclusion here. We get our hard limit, we resize the soft limit, then we apply the new soft limit. Really easy. Then we set up our user variables that we want to use in our program here. We set up our, where we store all the counters and PIDs. We get a user generated PID. We get the current PIDs. We create counters for all the current PIDs, then we get into the infinite loop, reset enable counters, sleep five seconds, disable counters, recounters, set cycles to zero, instructions to zero because we want to reset them. And then for all the counters for each thread, we want to add the values to cycles to get the whole value for the whole program. And that's why we have the plus equals addition simplification here. And then this is how we do the PID delta process. New PIDs equals blah, blah, blah, clear, diff PIDs, temporary variable, get the difference, create counters for what's in new PIDs that isn't in old PIDs, sew on, vice versa, continue. Then we display our results here, remember to convert floats when you divide. And then we go back, really not easy. So now let's put this program into use, let me get this ready. So in my downloads folder, I have the demo program, enter a PID, let's enter banana. That's an invalid PID because banana isn't a number. So I didn't actually show you this before, but here's how you assign the cap perfmon capability. Set cap plus PEDemo, that's easy. So demo, let's try get cap demo. Yeah, we can see we have the cap perfmon capability. So let's use demo, but we need a PID to monitor. So I don't know, oh, let's get this guy's help. He's a musician from Columbia and he's really good. I know this audio player is called Totem, even though it doesn't actually tell you that here. So let's go to top, OBS, Xorg, blah, blah, blah. We also see Totem here with a PID of 52.273. 52.273, messed up that, 52.273. Wait five seconds. Okay, it looks like we got this many cycles and this many instructions. We have kind of junky IPC about one. So generally if your IPC is above one, then you're in good position. But if it's below one, then your memory limited. That's just a really general rule of thumb. So it looks like we're not too memory limited here.  Let's try OBS. OBS is 7657, 7657. All right, we're getting a lot more cycles, but comparatively fewer instructions per second and we have a really bad IPC of 0.6. So that means we're definitely memory limited here, which makes sense considering that my laptop has kind of junky memory compared to my zen-free monster. All right, what else can we check here? Actually, do I even have time? No, so let's go back to the Libre office. Let's take it further. I have these problems with the demo. Actually, these problems are opportunities to make the demo better. We only have two events in one group. There are four plus events available, so on and so forth. There's a non-zero delay to enable, disable, and reset counters, blah, blah, blah. And thank you for attending this presentation. Remember to check me out at www.jghub.com. See more performance cameras and action here at my GitHub repo, right, stole all the demo code. Thank you.