Welcome to the Dr. Gendry podcast. Well, by now, if you've listened to me over the years, you know all about the microbiome, the trillions of bacteria living in your gut. These tiny gut bugs, as you know, are incredibly important for the health of your entire body. But my guest today says it's not the whole story. There are fungi living in your gut that can have a dramatic effect on your health, good and bad. Dr. Mahmood Gahoon is an expert on fungi, and he's the director of the integrated microbiome core and the Center for Medical Mycology at Case Western University in Cleveland. He's also the author of a brand new book called The Total Gut Balance. On today's episode, he and I are gonna discuss the effect of fungi in the gut, the best foods for improving your gut health, and the three things that you can do today to boost whole body health. Dr. Gahoon, it's a pleasure to have you here today. Thank you for having me. It is going to be really fun because you are going to open the door on a whole class of other critters that are living in our gut that people need to know about. So how would you get started getting interested in fungi of all things? Oh, that's a long time story. What happened when I went to England to do my doctorate? My supervisor gave me a paper and he said, this is what you are gonna be working on. And guess what? It was about how candida is affected by antibiotics and steroids. And that's when my passion to work with candida started in 1974. Wow, that's a long time ago. It is, it is. So we're gonna get into candida or candida as many people say. So why did he get you interested in that? 1974, come on. You know, it was interesting because it's something in you. For a long time, we really did not have much role for fungus in general and candida in particular for diseases. But suddenly because the way medicine is changing, new antibiotics, new therapies and also people becoming more immunosuppressed. They have low, weak immunity. Suddenly we started to see more and more of these fungal infections. So to me, it was an exciting new area to try to understand how can we really manage this new emerging disease. So let's take a step back. So fungi are with us all the time. They're in us and they're a part of what I call the holobiome. Absolutely. As you mentioned at the beginning, a lot of people think of the microbiome as bacteria. But in fact, in our body, we have not only bacterial community, we have also fungal community. And guess what? They play together. If they play together nicely, it's good for us. However, if they are fighting, which as you know happens, it causes a lot of issues including gut issues. Gotcha. So these guys are actually down there having a good time throwing balls around and not like that, right? But they communicate with each other, right? Yes, they communicate with each other. And also, if fungi is present at a low level or low abundance, it actually is very helpful. So everybody is afraid of Candida. But 50% of us have Candida in us, both in our skin as well as in our gut and mouth for that matter. So as long as it is low level, what it does is start to break down ferment food which provides nutrients to the bacteria, the good bacteria, and then they are happy also growing together. The issue starts happening if, for example, as you know, somebody take antibiotic. If you have an antibiotic, you are killing not only the bad bacteria which is causing an infection, we are killing also our good friends in our gut. And their role, this bacteria, they are really good cops. They keep Candida under control. So when you kill them, Candida have a field day and start to increase in number or overgrow and then cause issues for us. So when people hear the word antibiotics that basically means anti-living things, but most antibiotics that we consume or take kill bacteria but they don't necessarily kill fungi. Is that right? Yes. A lot of the antibiotics, absolutely, as you said, they kill bacteria. To kill fungus, we have what you call antifungal. It's antibacterial, antifungal. And the compounds that kill or the antibiotics that kill bacteria, they target some structures in the bacteria which we don't have in the fungus. That's why they don't kill it. Gotcha. So we should have renamed antibiotics, I guess, differently. But, and I think it's interesting because back in 1974, I was in medical school and right around that time, the first broad spectrum antibiotics had been introduced. And for us in medicine, it was great because we no longer had to identify an individual bug and wait for cultures. We could just napalm everybody and sit back and say, boy, aren't we clever?   Is that kind of what happened? Exactly what happened. And I can give you a simple example which was true even then. When women take tetracycline, the antibiotic tetracycline to kill bacteria, guess what happens? They develop thrush because the lactobacillus in their body keep candida under control but this antibiotic is killing it. And then now we are having this wild candida go haywire and cause infections. So that's really true with many situations as you say, especially with the broad spectrum antibiotic because you are, as you say, napalm them. So these guys exist in us. They're in most of us and you're right. They kind of have a two-fro relationship with bacteria. I got interested in them researching my next book as you know, soil health also contains its own microbiome and the interplay between fungi and bacteria at the base of roots actually is essential for the plant to absorb nutrients from the soil. And you're right. There are, without fungi in the soil, bacteria can actually not get their job done of helping the root system. Go ahead. Yes, absolutely. Like they build a synergistic relation as you know. So they work together. It's like the kids playing in a sandbox. If they play nicely, they help each other. And a lot of the time in the roots of our plants really both bacteria and fungus work together which will help them take nutrients. So it's not only they help each other but also they help the plant to grow. So having both of them and this synergistic or complementary relationship is very important for the growth of plants and even, as you know, for breaking down all the dead leaves. Now we are in winter. So yeah, I mean, it's fantastic collaboration here. Yeah. And so we're gonna get to this. So I think one of your first points is that fungi are not exactly the evil empire that we have to fear taking over our bodies. Absolutely, absolutely. I think if our immunity is good, if we are eating the right food, the right lifestyle, we can keep it under control. And we don't have to be absolutely terrified of this. There is a way in our hand to try to control that. Got it. Okay, so these fungi that are in us are classified as the mycobiome as opposed to the microbiome. But I agree with you that our microbiome includes both our bacteria and the fungi and actually a bunch of viruses and parasites and wonderful single cell. Yes, yes. Really, it's, as you say, it's like a population. You know, different background, different races, they all live together. The same in our gut and our skin. We have, as you mentioned, bacteria, fungus, viruses, parasites, they all are living there and they all really play good and bad together. And that's why it's very important for us to understand this so that we can control them and give a total gut balance, as you would say, for when we have microbiome is in balance. Great. Let me give an example for our listeners and our viewers that actually brings this home on a, maybe on a more, not a microscopic level. Recently, as you know, wolves were reintroduced into Yellowstone National Park over a lot of objections.  And one of the things that people had noticed since wolves were eradicated was the elk population. Their natural predator is a wolf. And the elk population actually grew ridiculously. And what they found was that the elk population was eating small saplings during the winter, particularly near streams, and the beaver population in Yellowstone was decimated. They left. And one of the first things they noticed when wolves came back was they began controlling the elk population. And particularly in winter, the elks were no longer eating the little saplings in the streams. And now with wolves back in, the beavers are back and the songbirds actually lived in these little saplings by the creeks and the songbirds had left Yellowstone. You actually didn't hear birds. And now the songbirds are back. So you're right.  And we really ought to eradicate them, because they're evil. But you're right. This is an important part of the ecology of us. Absolutely. And I think let's just agree that, I don't particularly want to meet a wolf. But that wolf is a very important part of the ecology. Right? Absolutely. And so you're bringing it back to you. Fungi are incredibly important part of the ecology of us. Absolutely. The other thing I think just to make people be at ease when they think about fungus is everybody talks about Candida. But actually in our gut, we have good fungus as well, such as saccharomyces, which is the baker's yeast, which we use for bread and this sort of thing. We have it there and when you have it, our study shows that it can control Candida. Also, one study I published when I first started to work on the microbiome, I was interested in the HIV infected patients where we looked at oral complications of HIV, which as you know, better than me, it is thrush. So we did this study and we found to our surprise that if there is a yeast called picae, P-I-C-H-I-A, if it is there, Candida is not there. And if it is Candida there, picae is not there. So it's exactly what you said. There is an environment. If we had picae, we are able to keep Candida under control. So there are good fungus as well as bad fungus. Yeah, as a matter of fact, in heart surgery, I, against the advice of my colleagues, after we would give antibiotics to all of our heart surgery patients, I would give them capsules of saccharomyces to swallow beginning day one after the heart surgery. And my colleague would go, why the heck you're giving people yeast and fungi?  And nobody ever got thrush after I started giving them these capsules. And it's like, wait a minute, you're giving a fungus to stop a fungus? But you're right. Honestly, honestly, your thought was well ahead of the game. And now the good news is the science is proving that this is the case. It's good to have these yeast saccharomyces to try to control Candida. So I am really very pleased because now we have more scientific evidence to support what you've been doing well before. We started looking at next generation sequencing and this sort of thing. Well, I learned actually this trick and then I've mentioned it before. I did my children's heart surgery as a senior registrar at Great Ormond Street in London, England back in the 80s. And after heart surgery on babies, the sisters, the nurses would send the mothers down to the local greengrocer to buy fresh yogurt. And they would come back with the fresh yogurt and they would feed these little infants who I operated on yogurt. And smart American going, don't do that. There's things in that yogurt and we have to harden, we have to be sterile.  But I said, you know, they're probably right about this. So yeah, I learned that many, many years ago the nurses taught me. Absolutely, that's why like in your recent work and ours, we are saying, look, take some of these fermented foods because this is good. And yeah, I mean, it's really, I am very excited about this because now it's becoming clear what we need to do to try to keep these little critters under control and have a good health. Okay, so keeping little critters under control. There's a fancy word for that called dysbiosis. Yes. What, what the heck does that mean? Dysbiosis, as you say, it's really the scientific terms for imbalance. Remember, to have a good healthy gut with no digestive symptoms, you need to have the balance between bacteria, the fungus and the viruses for that matter, even though we don't know as much about the bacteria and fungus. So they are all happy, like as I mentioned before, playing in the sandbox. The problem, if we have nice kid playing in the sandbox, they helping each other. You get some little other kid who wants to fight with everybody, that's where the imbalance start. And in our gut, we are seeing, when we talk about imbalance or dysbiosis, we have an increase in the bad or pathogenic organisms and a decrease in the beneficial one. So what we need to do is try to bring back this balance so that they play nicely again. Okay, so how, I know I've written a lot about this and you write a lot about this, how did the bad kids take over? Did they just not get sent to the principal's office enough or what's happened? I think it is to do with the way they eat. What you eat, what type of food you eat, certainly can feed the wrong organisms there. It's like the garden. The garden, in the summer we love our garden, even though you guys in California have always nice roses, we need to feed and grow the nice roses and kill the weeds. And that's what we need to do. We give them nutrients, we give them the right filterisans. The same in our gut. If we can give them the right food and really avoid the ones that will encourage the bad guys, we are gonna have a better situation. Also, if we can have some really lifestyle approaches which are also important because our studies are showing is not enough just to control this diet by just the food or the different kinds of food. We really need to have less stress, more sleep, and this sort of thing. Okay, so wait a minute. You're saying that sleep and stress might affect our microbiome? Yes. Is that what you're saying, Professor? They are, and you know why? Because studies now are showing that there is communication between our gut and the brain. We call it gut-brain access. For a long time, we thought the brain is really the master switch. Tells us everything what to do. But now we know also the gut talks to the brain, so it is bi-directional, two-way communication. So if you have stress, you are gonna have dysbiosis or imbalance in your gut. That's why we really need to have less stress even though this day and age is very tough to have, but really it's very important because we had a lady, she had complete imbalance or dysbiosis. I looked at her food, what she eats, she eating perfect food which is gonna support the beneficial organisms and reduce the bad one. But then we looked at the questionnaire which she fills and we found that she is severely stressed. So I advised and that's what we really recommend is you need to do a little bit of meditation. You need to do a little bit of relaxation. Go out to Yellowstone Park, go out to nature because that all will help you de-stress and hopefully is gonna help your gut balance. Yeah, now you're absolutely right. In fact, in my last book, The Longevity Paradox, we make a big point that it is a huge two-way street and that things like yoga and meditation have actually been shown to even change gut bacteria for the better. And certainly we find that reducing stress allows the gut wall to heal. Okay, we mentioned Candida before. I can't tell you a day doesn't go by that someone doesn't walk in and tell me that they have Candida or they're being treated for Candida or that Candida is the cause of all their problems. All right, set the record straight. Okay, look, when we look at Candida, Candida definitely is responsible for a lot of infections, especially in people who are, as we mentioned, on antibiotics, cancer patients and any situation where you have imbalance and weak immunity, okay? That's true, but at the same time, Candida also lives in our body. We did a study when I was at UCLA many years ago. We found that 40% of healthcare workers in the hospital, they have Candida in their head and they are fine. And that's another story, I'm not gonna go there, but if we go back to the gut, we have Candida. As long as Candida is at low abundance or low level, the numbers is low, it's fine. In fact, it can help us. And I just... Oh, wait a minute, stop right there. Candida can help us? Yes. Oh my gosh. It can help us in at least two ways. All right, we're listening. Number one, it will train our body immunity to start to improve, to be able to respond to Candida when it really grows. That's number one. Number two, it can help our body also break down some of the food, the nutrients, which will help bacteria to grow and then that really complete the circuit. So as long as it is at what we call colonization level, it's just sitting there, not too many of it, less than we'll say one or 2% of the total fungal count, that's fine. When it starts to go up, overgrow more than 10% of the fungus there in your gut is Candida, then you will start to have issues and you need to control. All right, so this is a news flash for all my listeners and I tell my patients this, so it's not a news flash to them. So Candida in the right amount is not a bad guy. It's a wolf in Yellowstone Park. But okay, so how does this population grow to a point where it becomes a problem? What happens, what are we doing wrong? I think there are, it's multifactorial, a number of factors. Some of them, for example, if you are sick in hospital, then you are likely to have Candida because you are in antibiotics, you are all chemotherapy in various aspects. So that's one thing which we already will establish. The other thing which is really well known is what we eat. If we eat a lot of refined sugar, Candida loves refined and processed sugar and it can grow very fast. So if we are eating all the time sweets, we put so much sugar in our coffee and tea. I remember when I was a young boy, we used to have tea in the morning with my brothers and one of my brothers used to put so many sugar. My father used to say, look you are drinking sugar with tea instead. So it's very important that we cut down refined sugar because that is gonna be really an important factor for Candida to grow. Also, if we eat lean proteins, that also will keep Candida under control. Finally, we showed and others that if you have a vitamin deficiency, especially A, B and C, you are likely to have more growth of Candida. So you can see there are a number of dietary as well as, of course, factors which is beyond our control sometimes. So the average person walking around who has gut issues, is it from Candida in the most part or is it much more multifactorial than that? I really would say it's multifactorial. I mean, Candida will be one part but also, for example, if we have protobacteria, protobacteria, which is bacteria as we are saying, if it is increased in abundance and in its level, also it becomes pro-inflammatory or in other words, it causes inflammation. It's a good red flag for inflammation. So it's just like the balance of good bacteria and good fungus. If you have too much bad bacteria like people who eat red meat with high fats, guess what they're doing? They are feeding the bacteria that loves bile which is pro-inflammatory. It means it could cause more inflammation. If you have really lean proteins from fish, from plants then you are encouraging the good guys. Now, the same with respect to Candida. If Candida, we are giving it the food that it loves, it's gonna grow. Hmm. All right, so let's suppose you found a person with an overgrowth of Candida. Give me the steps that you'll recommend at Case Western to get this balance back again. I think now with this, the new knowledge of the microbiome and how these organisms microbiome and microbiome, which you mentioned the fungal community, the first thing we need to tell them is that really they need to try eating the right food. Try to avoid, we need to really restrict the growth or eliminate these organisms by not giving them food. Starve them to death. Starve them to death.  Make sure you take your vitamins. We need to also try to take some fibers because we want to encourage the growth of the good bacteria because those good bacteria can keep Candida under control. Now obviously, if somebody have an infection and whatever, that's another story. Of course, we have to go into antifungals and we are lucky these days, we have really some good antifungals. It's different from the days, as you remember, when we had only amphotericin B. That was it. It was terrible. Now we have better drugs for that case, but for ordinary person, I think if they can follow the right diet, starve the bad guys, feed the good guys, make sure you are really having food like cruciferous vegetables, which you love, you are gonna have anti-inflammatory, anti-oxidant components there, then you are gonna go a long way to rebalancing your gut. All right, let's myth-bust for a minute, and we haven't talked about this, so you're probably gonna bust my myth. There's a lot of advice on the internet for a anti-candida diet that you should never eat fermented vegetables, that you should never eat cheeses, and you should never eat mushrooms because mushrooms are fungi. I can bust that. Okay, bust that, would you please? So first of all, mushrooms, great food because they have fibers, there's a lot of fibers. Oh, thank you so much, thank you, okay. Yeah, no, no, I think this is definitely a misconception. Now, don't eat fermented food. As we mentioned before, saccharomyces, which is a good yeast, it helps us. So fermented food is a good thing to have. Now, of course, you don't want to go after the fields, go to the fields, and start picking mushrooms if they are poisonous and whatever. No, this is your problem. If you go to a good market or a place where they sell the right stuff, which is not toxic, mushrooms is great, as well as you are gonna be fine.  Just an editorial comment, okay. No, so that's great because I can't get over the number of people, and I have mushrooms on my diet, and I have fermented foods on my diet, and I have aged cheeses, and people go, well, I have candida, I can't do that. That's gonna promote the growth, and I'm going, where does this stuff come from? And thank you, thank you. All right, I wanna talk about one of my actual interests that I don't talk about, but we're gonna talk about it today, biofilms. Now, the Academy Awards are coming up. There are probably gonna be no biofilms that we're gonna watch.  Let me tell you, we all have biofilms. The best example is the plaque in our teeth. Every morning and evening, I hope, we brush our teeth. Why? We want to get rid of the biofilm or plaque. The plaque is a biofilm. Now, what are they, what are they? So I can tell you, in my studies, we showed that the first thing for an organism or a bacteria or fungus to start causing troubles, it sticks to a surface, such as the teeth, okay? Once it sticks, it starts to produce what you call matrix. So what is this matrix? It's like a jello. If you have a jello, and inside the jello, we have raisins or M&M's. So the raisins and M&M's are the bacteria and fungus. The jello which covers them is the matrix. So as if they are living inside a city covered by this matrix which protect them. You're talking about the film, the matrix. That's, biofilm and microorganism have a matrix, definitely have a matrix, which as you say, in a way it is like a film. It protects these organisms. They become resistant to all antibacterials, as well as we showed that they are resistant to our immune cells. They cannot go there and kill them. So now, let's go into the gut, okay? In the gut, not many people looked at that. Most of the studies were on the catheters and knee replacement stuff and whatever. We in 2016 published a paper where we showed that the bugs in our gut, which is, we found that Candida, Tropicalis, one of the species there, E. coli, which is a bacteria and Siracia marcescens, they all increase in these Crohn's disease patients, compared to their relatives with no Crohn's. And then, we found that they come together and they stick together. They start, as you said, communicating and talking together and they produce this huge biofilm. And we called it digestive plaque. It's like digestive biofilm, but it is a plaque because in, you know, to make it easy, it's all in our digestive system. Now, when they form that, that is a bad news. Why? Because then they start to change. For example, Candida, instead of staying as a round cell like Baker's yeast, they start to form these filaments or like threads like and those start to go and start damaging our gut lining. Okay? So that's why. So they can literally drill into our own cells. Absolutely. They can go in and they, I, before, I don't want to get too complicated, but I tell you, before I started to do biofilms, I started Candida is able to secrete enzymes that can break down the lining, you know? And we showed that it's a beautiful picture where he's, I know, I mean, I'm crazy. I like these pictures. I know, you're getting really nerdy on me here. Yeah, yeah, yeah. So you see the filament like this and ahead of it, you have all these spot of enzymes which breaking down and going to cause leaky gut, as you say. Now you've got everybody scared of Candida again. And we were both were trying to quiet everybody down. Okay, so how often are these biofilms happening?  So are you saying that that's happening down below and we don't know it? You know, it is there, but when there is the right harmony or this balance, when you have balance, the biofilms that forms are with the good guys, they don't have the ability to go and start causing trouble. The only issue when you are, as I give an example in Crohn's disease, for example, they overgrow and they are made of organisms that are pathogen or cause disease, then they are a problem. Gotcha. You know, believe it or not, you're probably aware of this that there is one theory of coronary artery disease that is an infectious theory of coronary artery disease and that bacteria and fungi and mouth organisms, some people want to call them nano bacteria, I don't think we have to go there, but that they set up these biofilms on the coronary arteries and it's these biofilms that are, we can't attack them with our immune system, but we try and it's actually, the plaque starts with biofilms. I kind of like that theory and I've actually written about that. Again, it's a theory. I get it, I get it. But, so okay, so we don't want biofilms. Candida is a good guy unless it gets out of control and if you feed it sugar, it potentially can get out of control. So give us three things that people can do today or start doing today to make sure that they're fungi or good guys and happy and everybody loves each other down in their gut. Yes. Number one, we need to cut refined sugars. Okay. Very important. Number two, it's very important that we have good vitamin levels in us, like vitamin A, B, and C, okay? Number three, we need to give resistant starch or fibers to feed the good bacterial guys that can keep Candida under control. Okay, I write about this a lot, but for our listeners, give me some resistant starches or fibers that we should be eating. Like you can find it in banana in the unripened side. Thank you for saying unripened, I appreciate it. Yes. We can say in oatmeal, you can see it in sweet potatoes. Yeah, all of these are great sources for resistant starch. And why really they are important to have resistant starch or carm? You know, I know you will ask me, I can now start, you and I are in the same wavelengths. Why are we saying about the resistant starch and not simple sugar? Because simple sugar, we can break it down in our intestine and they really can affect our glycemic index, they start to, if we don't, if we have too much of them, they convert into fats and we become obese. Whereas the resistant starch, our body cannot break them down. So they go down into large intestine where these good microbes are there and they start to eat them up and break them down. When they do this, what's happening, they start producing good molecules or good compounds. We call them metabolites. And what happens, these metabolites are great because then they can communicate with our brain and make our, increase our immunity, make us in a better mood. So that's why it's really possible to control these organisms and have a happy gut. All right. So you mentioned vitamin A and C and E and I would throw in vitamin D is really important. But most people don't know about vitamin A. Where do we find this stuff? I think there is a lot of vegetables that have vitamin A. Crosophers, vegetables have a lot of vitamin A. So, you know, broccoli, brocolini, all of those stuff, they have it. And vitamin A, why I mentioned it? Because studies have shown that kids who had candida issues in their gut, gastrointestinal issues, they really were deficient in these vitamins, including vitamin A. So having vitamin A is gonna help you to really fight candida in addition to the other factors which we talked about. And another great source of vitamin A is cod liver oil. Yes, yes. And unfortunately when I was growing up, I got a gulp of cod liver oil every day. And I must say the modern cod liver oils don't have that fishy taste, which is so don't be afraid of cod liver oil anymore.  Yeah, yeah, everybody had to do it. And you had to hide the flavor because it was pretty bad. So tell me, you've been at this a long time. What's the latest things you're working on? What's the latest breakthroughs that we should pay attention to? Because for most people, they never even heard about fungi and the gut. They know about candida. What's happening? You know, thank you for asking this question. I really, first of all, when we observe that, okay, they come play together the bad guys, they make biofilms. Now I'm starting to look at, okay, how can we develop ways to get rid of this biofilm? And I put a grant for the National Institute of Health and I can tell you, I'm happy to tell you that I just got funded. Congratulations. Thanks. It's a big deal, folks. To look at the interaction between bacteria and fungus in our gut, especially Crohn's disease. Because once we know this, we are gonna be able to come out with method, even better method than we currently have. So it's exciting to do this. Because you and I can't swallow a dental hygienist to go down there and pick at the biofilms. Absolutely, absolutely. We published a paper in April of 2019 and where we developed a probiotic. And I have for the sake of transparency, it's a company which I co-founded called Biome Health, BIOHM Health. And we showed that this probiotic, which not only have bacteria, but also have fungi, good bacteria and good fungi, is able to break this biofilm, you know? So it was published in Mbio, it's an American site for microbiology journal. And I think we are really moving in the right direction, not only in this, but also trying to understand in other situations of where the dysbiosis happens, you know? So, all right, so that's exciting and it's okay to give yourself a shameless plug. So, all right, so how do people find you? Where's the book? Tell us. Okay, you can find it, of course, in Amazon, bars and novels, but also there is TotalGutBook.com. You can find it also. If people would like to learn more, there is, I have a website called Dr. Microbiome, which really describes different ways how you can control your gut and also there is a community built where people help each other and this sort of thing. So do you have a lot of Candida suffering? People visit your site? You know, a lot of people, I must say, they call me and they tell me about Candida and I try to, as like you did, alleviate their fears, you know, and try to help them, what's the best way? We don't have to have a pill for everything. We are able to hopefully do something without that except in extreme cases. Wow, we don't have to have a pill for everything? Yeah, you and I both figured that out a few years ago, I guess, and good for you for telling people that we don't have. Actually, before we go, what about this notion that once you get Candida, you will never get rid of it and you will have chronic Candida the rest of your life and it's in you forever and that's it. Good luck. I don't agree with that. I really don't agree with it. I think because I did an experiment with my students once and I said, I want to take samples of your mouth, you know, just I want to see who has Candida and guess what? 50% of the students, they are healthy, sport playing guys, they have Candida and they are healthy. So to me, if you have Candida in your gut, it's okay if it is at low level. The only problem is if it starts to overgrow and you start to have, for example, digestive issues, constipation, bloating and whatever, then you need to start to look at that. But don't worry, Candida is a good guy.  If your immunity is down, you may have issues, but if you don't, don't worry, you should be fine. Follow-up question. What do you tell all these folks who watch TV and see every immunosuppressive drug known to mankind to treat every autoimmune disease, should they be more aware of Candida in them? I think if you are immunocompromised and there is Candida, as long as it is at low level, which you can know about it, there is a test. Again, I'm going to give a shameful, we, this company, which I mentioned by you, we do got testing where we are able to evaluate both our heavy profile of bacteria and fungus. And if it is low, don't worry, it's okay. It's fine to watch it, but you don't have to be really paranoia to have paranoia about this. Okay. That's what I think. Very good. All right, now at the end of our program, we always have an audience question, so, you know, we've got fans from around the world. And here's a question from Tore Schelebold, who asks, is IGF-1 in red meat equally present in game like elk and deer as in beef? Well, now, wait a minute. So IGF-1 is a growth hormone and it is present in, for instance, human milk. It's much more present in cows' milk and sheep milk and goat milk than in human milk because baby cows have to grow quickly to avoid wolves, for instance, and elk have lots of IGF-1 in their milk. When I write about animal proteins like beef or even fish or even chicken, the more animal protein-specific amino acids that you eat, in general, the more IGF-1 you will produce. But IGF-1, per se, the hormone is not the big factor in beef. So the answer to your question is eating elk or deer, you're going to have the same animal proteins that would promote an increase in IGF-1 as you would in beef. I think maybe your question is about the sugar molecule, a Nu5G, which is present in the blood vessels of beef, lamb, and pork, cattle, lamb, and pork, but it's not present in fish and chickens that we can have an autoimmune reaction to that causes another theory of coronary artery disease and autoimmune attack on our blood vessels. It's also present in deer and elk, unfortunately. So on the other hand, deer and elk, if you actually got it wild, it's gonna be far better for you because it's not farm raised on grains and soybeans and given antibiotics. But word of caution, most elk and deer that you buy at a restaurant, as well as bison, is farm raised, so just be cautious.  All right, so that's it for the Dr. Gundry podcast. Stay tuned for next week episode, but tell your friends about this because there's some really great myth-busting today going on, and I want everybody to hear this one. Thank you again, Dr. Gundry. Thank you very much. It's really good expression. Before you go, I just wanted to remind you that you can find the show on iTunes, Google Play, Stitcher, or wherever you get your podcasts because I'm Dr. Gundry and I'm always looking out for you.


please um all right sorry give me one second i'm gonna start my step watch okay thanks uh so today i'm gonna talk about um amp uh to characterize optimal type one type two error trade-offs for sorted penalized sorted l1 penalized estimators or or slope so this is joint work with some folks from uh the university of pennsylvania uh ji gi bu and wei ji su and then also um my colleague jason kluzowski uh from from princeton so um while that notion of sorted l1 regularization can be used in many problems i'm going to keep it simple today and just talk with talk about high dimensional linear regression so this is this framework where all you know kind of know and love right um we're thinking of output y equals x beta plus noise um my x matrix is going to be high dimensional so it's short and wide um i'm in the the usual asymptotic regime that most of us have been talking about where uh the ratio of the rows and columns um is some fixed delta uh we'll assume that the matrix is iid gaussian and um i'm also going to assume that my vector of coefficients beta is sparse with iid entries and in fact we have linear sparsity so a fraction of the the elements are non-zero and that fraction stays fixed as n grows so um again what i'd like to do i know y and i know x and i'd like to recover beta so um okay so for for this problem a well known and widely used uh procedure is the lasso right and so what does the lasso do well it produces an estimate of this coefficient uh vector beta uh by minimizing a cost function where the cost is the sum of a data fit term you know this is the the sum of the squared errors plus a penalty term where the penalty is the l1 going all right so um the the penalty term there uh forces a sparse solution right and the the level of the sparsity in the solution is is controlled by uh this parameter um lambda so the larger the lambda the more sparse our solution so um okay so what i'd like to talk about today is a new procedure uh referred to as slope or uh sorted l1 penalized estimation that's a generalization of the lasso and so the idea was slope is we want to break the monolithic l1 penalty in the lasso that treats every variable um in the same way by instead using a penalty that uses different regularization uh depending on the coefficient magnitudes so uh the motivation for slope was um to control thoughts discoveries um by sort of privileging correct support recovery over minimization of prediction error so uh when i say thoughts discoveries this is a notion i'll use throughout the talk um i mean elements in the output that are non-zero uh that correspond to zero elements in my my true beta right that there's also the notion of true discoveries which are going to be you know non-zeros in the the um the output that correspond to non-zeros in the the truth and so the way that slope sort of controls the thoughts discoveries is uh by incorporating adaptivity into its penalization sequence here right so it's like the lasso we have a cost function that's two terms a data fit term and a penalty but now my penalty is adaptive and so um what i mean by that is um it's sort of uh penalizes different values uh depending on on their magnitudes right and so the the adaptivity that it it sort of is motivated by is um that in strategies like benjamini-hatchberg uh the the procedure for multiple testing which you're going to you know compare like more significant p-values with more stringent thresholds so we're going to kind of marry those ideas with uh the the usual advantages of the l1 penalization that gives us that that sparsity so um it turns out that the sort of uh adaptivity in the the regularization sequence for slope um gives it a lot of nice properties uh for example it can be adaptive to uh the level of the sparsity so you don't necessarily have to know that ahead of time and um it obtains these optimal performance and certain problems and so um though the lasso is clearly a specific form of slope so right i could just take all of these lambdas to be equal and i get back the lasso um you know the lasso isn't going to have you know all of these these nice sort of properties so uh what we aim to do in this this work is to kind of quantitatively characterize the benefits of this you know uh freedom in in this you know sorted uh regularization and kind of just further contribute to um helping sort of answer the the question of what what what does this actually give us um by uh in particular exploring model selection properties of slope and uh we'll do this with the help of amp so i'm gonna sort of give the talk by um sort of telling us the the implications this has for for slope and then i'll talk a little bit about um uh the amp aspects of the the proof okay so um when we're in order to characterize uh model selection i'm going to talk about uh the true positive proportion or the tpp and uh the thoughts discovery proportion or the fdp so um this sort of gives some idea about you know uh the the rate at which our procedure is finding uh you know true discoveries versus false discoveries and so um in particular uh something that uh has been shown before is that in the case of the lasso there's a fundamental trade-off between uh this true positive proportion tpp and false discovery proportion fdp that's uh rigorously characterized so you can you know we sort of know the best um the lasso can do in terms of uh comparing these two values right so this is equivalent to like a type one type two error trade-off um if that's more familiar language so um okay so formally if we're talking about the lasso uh the situation looks like uh this so uh the work that um wagey did uh you know uh in in his dissertation sort of computes an exact boundary curve uh that separates achievable uh fdp tpp pairs from those that are impossible to achieve uh by the lasso um no matter what the signal to noise ratio is in the problem and no matter what regularization parameter you use so um kind of you know obviously right what we like is for the fdp to be low we want no you know low false discoveries and high tpp high true positives right so it's kind of like this is the region of the fdp tpp plane that that we'd like to to hit but it turns out you know that uh this work shows that this you know whole favorable region um can't be reached by the lasso so uh and the this image right the shaded area is the unachievable region by the lasso so in short um what what this work sort of implied is that you know no matter the choice of the regularization parameter and no matter the signal to noise ratio both types of errors can't be low simultaneously uh moreover um what they show is that that you know when delta is small so when my matrix my measurement matrix is really fat um or when epsilon is large meaning my signal is is dense uh the tpp is going to be uh asymptotically bounded away from one uh again no matter what you do with the regularization or the the s and r and so um this is sometimes referred to as a dano-hotan or phase transition and it's visible on the the plot on the left by this kind of vertical line here right and then um in the plot on the right we we just have the epsilon delta space where such a phase transition occurs so the the sort of epsilon delta pairs where you know your tpp for the lasso is going to be bounded away from one so um in problems where such a phase transition occurs uh i'm going to refer to sort of this largest achievable value of the tpp as the dano-hotan or power limit or the dt power limit so you know in our plot the dt power limit's going to be something like point point six here okay so um here i'm i'm plotting sort of the realized fdp tpp pairs for both uh lasso and slope on the same problem um where we have 10 independent trials and i'm just burying the regularization uh of both methods across the kind of um the the trials so um the message uh that that you know i want to to sort of convey with these plots is that you know uh if we look in the low tpp regime both methods seem to undergo a similar trade-off between fdp and tpp right so the red and blue points are kind of you know they look the same right but um what's more interesting uh in somehow is that in the high tpp regime uh clearly you know in each plot there's a certain tpp past which we never see red points right so this is exactly the the dt phase transition on the the previous slide so you know it's like point six here and maybe point four here but um you know obviously the blue points though aren't bounded away from from uh full tpp in the same way that the the red points are so it's like the the blue slope points um are able to pass this limit towards uh achieving full power so um kind of recognizing this right uh we're tempted to ask like why is sorted l one regularization better than the usual l one regularization uh for high tpp and um why do slope in uh lasso exhibits similar uh trade-offs in the low tpp regime and so um this is what we're we're going to try to to answer today okay so um what you know what we set out to do uh is to characterize the optimal uh fdr tpp trade-off curve so um what i mean by the optimal curve right is uh i would like to give the smallest possible value of thoughts discoveries um that's asymptotically achievable by slope uh with any regularization any snr um for every fixed level of the tpp right so i fixed my tpp how how low can i can i drop the false discoveries um so it turns out that uh we weren't able to actually do this exactly but we could give upper bounds and lower bounds uh for this this curve and so i'm going to introduce the upper bounds and lower bounds next and then talk about how a and p helped us uh to get um so if i give an upper bound and lower bound obviously the true curve somewhere in the middle so the goal you know the aim is that these are tight in some in some sense okay so in this plot uh i i give an example of um the bounds for just a single problem instance so the plot on the right is just kind of a zoom in of the the plot on the the left and in the the plot the blue and the green curve is our upper bound and then the red curve is the the lower bound and so we know that the optimal curve is between the two so um okay so i want to discuss a little bit about how we get these bounds uh and then i'll go into to some more details so uh when we consider the upper bound so this is the blue and the green uh to prove this upper bound our proof is is constructive so in the the sense that you know we specify a prior on the signal and a regularization sequence for slope and then we show that this asymptotically achieves that fdp tpp um point and so you know if we can achieve it with a certain uh setup right then the minimum fdp must be below below that value so uh below the uh dt power limit um the the upper bound is simply given by the corresponding curve for the the lasso so this is green on the the slide right and so um this is just you know recognizing that the lasso is an instance of slope the fact that the lasso can achieve these values means that the slope must be able to do it at least uh as good um and then uh the the other part of the upper bound above the the dt power limit which is the blue case um it turns out that this is given by a sort of the shape of the curve as a mobius transform and so i'm going to discuss the the proof of the upper bound um in what follows but uh as i mentioned it's it's constructive in the sense that it specifies a simple sort of regularization um and then it uses the amp analysis to uh demonstrate that such a regularization achieves these uh fdp tpp pairs um okay so then um i'll talk a bit about the lower bound but this is all i'm going to say about it uh in this talk so it turns out that proving the lower bound on the the optimal curve is actually a bit more subtle uh and so to to do this we developed a technique um based on a class of infinite dimensional convex optimization problems and uh while like i i think like the the actual proof is like um is pretty cool and uh it uses some novel ideas uh that probably you know i imagine will have um application and you know when we want to prove theoretical uh things about regularizations more generally um for the sort of for the sake of time i'm i'm going to focus on uh the amp aspects of the the proof um but uh before i i uh okay before i get to the proof um i just want to show one more um picture of of what these upper bounds and lower bounds look like more generally so on the last slide it was like one problem instance but now i'm trying to vary the epsilon and delta um pairs here so in this uh image the solid lines are the upper bounds uh the dotted lines are the lower bounds the true curve lies somewhere in between so you know it's like i'm happy to report on the slide that the upper bounds and the lower bounds are pretty tight um but you know i i kind of show this slide because uh there's something kind of interesting going on in the the plot on the bottom right so there in all of these plots we're fixing either delta or epsilon and then varying the other one and so um if you look at the plot on the the uh bottom right and this is something i don't really have a great explanation for but um you can see that what happens is we fix delta so we fix the dimensions of our measurement matrix and we're varying the signal density um and it turns out that the tradeoff curve isn't monotone there right so as the signal is getting more dense it's not like the problem is monotonically getting more difficult or something um which uh you know is perhaps something we would expect and is in fact the case for the the lasso um okay so i had one more plot but i'm gonna skip it and for the sake of time and just talk a little bit about um the proof uh so i just wanted to say at a high level you know um when we talk about the proof i'm gonna talk about mostly the upper bound um beyond the dt uh power limit and so um just at a high level to give you some intuition for how the sort of freedom in these parameters helps us beat the power limit in slope um i made this slide so the kind of idea is that it's well known uh with the lasso that um the lasso estimator will select at most in out of its p uh elements to be non-zero and um moreover you know a significant proportion of um false detections are always going to occur uh for the the lasso um in this like linear sparsity regime so uh and this is interspersed along the lasso path so it's always going to miss uh a fraction of the the true variables which gives that dt power limit so um on the other hand in slope uh we don't have the same sort of bound on you know in out of the p non-zeros the kind of corresponding bound is in fact on the number of unique non-zero magnitudes um so in theory you can you know kind of construct an extreme case where the the recovered slope estimator is completely dense in which case you know the tpp is automatically one so um that's kind of the high level idea about what what changes here um but what i want to talk about is is how we actually you know prove this and so uh the the the big idea is that um it's constructive so we use a simple like two level regularization sequence so um instead of just penalizing with one lambda we now have two lambdas and you know some fraction gets the the big lambda some fraction gets the smaller lambda and we're going to use amp to to characterize the tradeoff curve exactly for this regularization sequence and so um the the program kind of um that's satric uh laid out before is uh you know is the following we're going to you know construct an amp algorithm show that its fixed point solves our optimization problem and then prove that my amp converges to that fixed point and then once i have that i can pass the the sort of statistical guarantees from the amp state evolution onto um my my slope estimate so um this is kind of where i'm starting in my proof sketch my amp analysis gives me an asymptotic characterization of of the slope solution you know from its state evolution and then i use this characterization to character or to say something about the properties of the solution like the sparsity the fdp and the tpp and then this analysis you know gives me the conditions under which my delta epsilon pairs uh under which i can achieve various uh fdp tpp values um but to wrap up my talk i think okay i have like one minute left um i want to say i want to just shout out a a piece of work um that uh i have uh with sorry i'm skipping some slides um okay uh with uh a few colleagues at the University of Cambridge so this is with Oliver Fang, Ramji Venkata Ramanan and Richard Samworth and it's a tutorial article um on amp that was meant to kind of introduce amp to statistician so in general we wanted to um kind of derive and motivate amp without using uh replica or belief propagation since these are kind of ideas that uh statisticians tend to not be as familiar with um so if anyone has the bandwidth uh you know we're we're definitely looking for some feedback on um the the article and you know most of you guys are cited uh a lot so uh that's just a quick plug but one of the um things that i think that's cool about this article is uh we tried to kind of unify some of these ideas and one of them is this this program that Cedric used in his work and that i used here for slope um that's that of deriving exact asymptotic characterizations for estimators that are solutions to convex optimization problems so in particular this work provides kind of like a recipe for um doing this so you know anytime we have a an estimator that's a solution to an optimization problem of this form um what that paper gives is a generalized amp algorithm that has a fixed point that will solve this optimization problem so i'm still thinking of you know some sort of glm that's my model and i have a sort of a loss and a penalty that are separable and assumed to be convex but you know um these things you know well let's let's okay with the this simple sort of um optimization uh what we do is we provide the amp algorithm that will have the the um the fixed point that solves it and then you know the work on behalf of the the user is to then sort of verify the existence and uniqueness of the fixed point in the state evolution and then prove the the convergence of the amp to the fixed point and so um truly like the step three is the hardest part um as you know usually the cost functions are not going to be strongly convex at the solution so uh showing that we converge you know um takes a little bit of work but uh you know uh we give some strategies that other papers have have done to to uh to do this so what we're hoping is that you know with this recipe we kind of unified the work that you know um folks did for the lasso and the robust in estimators and you know pragya did for logistic regression and you know we did for slope here in a way that that sort of just gives a recipe that we think you know probably could be used more generally for glms um of this sort um so with that um I will stop uh thanks everyone for the the invitation and um I will pass the the baton to uh I think Joe's next so


the meeting at that stage. Thank you very much.


You got it, yeah, I got it. Excuse me, I thought one blip on me. There you go.  All the spook. How long has it been since you caught a fish? About a month and a half. First fish. A month and a half. Ooh. How did I call that, dude? How did I call that? He's small, man. They're in there. All right, tell me how I called that. I said my leg itches. I said my leg itches. I'm about to catch one. And then I literally hooked him right when I said that. There you go, second topwater fish. Summertime, topwater bassin. Let's get some more. What was funny is I was like, my leg was itching really bad, but I was like, I gotta keep working my spook. And then I was like, I'm about to catch one. And right when I said that, he hit it. That was pretty funny. All right, guys, here out on Lanier with Weston, I really didn't start a intro to do this video, so I'm going to do it now. But we're catching fish on topwater today. We came out here for a morning bite. We met up at 5.30. Now we're out here on Lake Lanier. I'm not even kidding. I haven't seen one boat. Like, we haven't seen one boat out here. But it's actually Weston's last day in town. He's about to move to Florida. So I told him, I'm like, we're going to go early in the morning. We're going to go catch some fish. I gotta go drop off my truck at 12. So that's why we came so early. But let's go ahead, continue on with this video. Weston right there. Dude, they're everywhere. Hold on. I'll get back to you guys. Did something hit my back or something? Is he on? I was just kidding. No, I was just kidding. I was like, I felt some weight on my line. Oh, I thought you still had the topwater on. I was like, wait, is he still on, dude? Literally, what's the chance of you going to show off the camera? And I have my port, it was kind of tight. And I felt like I had weight. I felt like I might have a fish on anywhere. Dun, dun, dun. You got another jig pole on? Yeah, you did. Oh, my.  What? Dude. Hey. You got him? You got him? Throw in there, Weston. This is a good one. Oh, I missed. Oh.  He's a good one. Oh, my God. That's a good one, dude. Look how fatty he is. I see it. Want to put him in the libel? Yeah. So we catch some good ones and we can take a cool picture. Dang, dude. He hit it right when I... Well, that's a slow and stocky, dude. Jumped three feet out of the water. Wait, right now?  All right, guys, here we go. Third fish. And since I... What was I even going to say, dude? I don't know. Oh, look at him. He's going behind you. He's going behind you. He's going behind you. Throw in there. Let's see if we can get a double up. But, guys, I want to throw this guy in the libel. He's pretty fat. It's actually Weston's last day of fishing. So we're going to try to get a cool picture going. So pretty solid fish. Let's see if we catch some more. All right, guys. So this is like a legit tip. If you're top water fishing or if you're just fishing in general, say there's a lot of bait in the area. We've been fishing this spot for about 30 minutes now. We've caught three fish. We're going to stop the activity slow down, but we know there's bait here. One thing to do is actually crank up your boat. This is going to sound really weird, but it's just what it is and it works. So when you take off, kind of make some ruckus in the water. Just start stirring up the water. What that's going to do is it's going to stir... Why are you laughing at me?   Therefore, the fish are going to bite. It's going to get more active. So seriously, just start... Why are you laughing at me? I'm being serious right now.  Hopefully catch some fish. I'm not going to lie. When I come to this spot, I usually catch one fish. It was bad for the longest time. And the last time I caught three here and not here, we got how many here, three? Potentially get about 20. I don't even know what I'm saying. What kind of math are you doing? I don't know. Are we going to catch three eventually? 20? No, not eventually. Potentially. Oh. Potentially we can catch 20. Because there's a lot of fish you need. But I think we should fish here and then come back, you know?  I couldn't see you because of my glasses.  Like Betty Crocker doesn't shrimp. What did you just say? I said... Who's Betty Crocker? I don't know who Betty Crocker is. Dude, come on. Tell me you watched SpongeBob, bro. I don't even think SpongeBob's in Betty. SpongeBob? I don't even think Betty Crocker's in SpongeBob. Betty Crocker, right, is a cook. And... You know what I'm saying? I literally have to add all this. I have to... What I got to do when I leave is I'm going to do donuts. This is a secret. I don't tell them. When I leave, I'm going to hit about 70 going that way, right? We're going to jack the steering wheel as hard as we can. Start doing donuts when you do a burnout, too. And listen, what I do is...  Then we'll start biting. You think I'm kidding, dude? I believe you. I'm kidding.  It's all good, dude. Pallets like parallel to the pallets. Yeah, I'm not. No, there's no way I could be in the line. Neighbors want to tow my car because I'm apparently in the way now. Which one? I parked this close to the curb as I can. Which one? Who is it?  The freaking pole right here. What is this thing, bro? Weston, I'm screwed, dude. We're just all screwed up right now. My phone's ringing. It's got, like, nine pots. Look at this, Weston. I literally swung the jig around. Look. And I'm screwed. Okay. It's gonna be so wild that it might honestly be put in this pole down. That's 9.1. Largemouth. Hold on. I barely got him, bro. I'm barely getting it. I'm getting it. It's a big largemouth. Brother, brother, brother, brother, brother. I don't believe you caught a largemouth. I hope myself, too. All right, you want me to get it? Dang, dude. I can't believe you caught a largemouth out deep. Topwater spook. I cannot believe you caught that out deep. Put it in the left-hand side. A largemouth, how do you do it? Honest spook. That's so crazy. Stop. All right, just grab it. All right, I got it.  Just grab it.  Just grab it, dude.  Weston, stop. Just grab it, please. I'm in a cluster right now. I just want to fish. We don't have that much longer for topwater. Stop. You really just do that? Yes. Why? Keep on. I was really about to say why. We've got to end this off.  All right, guys. Ended off the day. We've been out here for just like two hours. But I got to go take my truck to all South Thames to have some stuff done and get my speedometer checked and everything. But we have two fish in the libel right now. Weston's got one. I got one. And we're actually going to get a thumbnail with it. But let's see what we got going. Guys, these VT2s keep these fish so healthy. I'm not just saying. They really do keep these fish healthy. They're so hybrid. So wild. This is crazy. Here are the two fish. Nice spot. Nice large one. We're going to go ahead and get a thumbnail. Catch back up with you guys in a minute. All right, cool. All right, guys. Here's the two fish that we kept in the live well. That's a good large one out there. Guys, that was in 50 foot on top water. A large one out there. And that's rare on the near guys. I know you guys probably think that's a large one. That's just a large one out there. The near you really don't catch that many large ones, especially down south, especially out deep. We're going to go guys. Two fish that we kept in the live well, caught five all day. But it's only been a few hours. So let's release them. You want to get a snap real quick? Yeah. All right, guys. Ending off today's video. Sorry if it's a little bit windy out right now. But I want to go over what I was using. So I was using a bone white spook on 50 pound braid. And I have an Akuma Helios. And Akuma, Akuma Helios reel. And Akuma Helios rod. The rod is actually seven foot heavy power. It's a great rod, especially for this top water. And a heavy sounds a little weird. But for this rod, it hasn't given a tip. It's perfect for that top water. So that's what I was using, guys. I'm going to take my car. But I hope you guys enjoyed the video. Thank you guys so much for watching. If you enjoyed these top water videos, please hit the thumbs up button. I'll catch you guys in the next one.


This is Mauro Pesina, and welcome to my session, which is getting the up and down service efficiency that auto scalers won't give you. These are the contents we will cover today. We'll start by discussing the problem we are going to address and the Kubernetes challenges for ensuring application performance and reliability. We will then introduce the new approach we implemented in Moviri, which leverages AI to automate the optimization process. And they do that by providing real board examples. Finally, we will conclude with some takeaways. Before proceeding, please allow me to introduce myself. My name is Mauro, and I'm the head of performance engineering services line in Moviri. And performance engineering has always been my passion since I graduated. So let's start with a quick introduction about what the problem is. As you all know, applications are evolving. Monoliths are quickly disappearing, and modern and cloud native applications are features. Dozens or having hundreds of microservices spanning over a wide set of technologies. All these technologies comes with the tunable parameters and options themselves. And as a result, we must deal with hundreds of thousands of possible configurations. So how do you find the configuration that best suits our workload? Speaking of Kubernetes itself, there are more and more stories about how difficult is to ensure performance stability for Kubernetes application, as you may have seen also on speeches before mine. The Kubernetes failure stories, for example, is a website that was specifically created to share Kubernetes incident reports with the goal to learn how to prevent them. Many of the stories describe teams that are struggling with Kubernetes application performance and stability issues, such as unexpected CPU slowdowns, or even sudden container termination. But why is it so difficult to manage application performance, stability, and efficiency on Kubernetes? The simple answer is that Kubernetes is a great platform to run container-raised applications, but it requires applications to be carefully configured to ensure high performance and stability. Let's now get back to the fundamentals and see how Kubernetes resource management works to better understand the main parameters that impact Kubernetes application performance, stability, and cost efficiency. We'll go through the six main key facts and their implication. Starting from the first, which is resource request. You may have heard many talks before mine talking about this topic, but let's make a quick recap. When developers define a pod, they have the possibility to specify also resource request. Those are the amount of CPU and memory which are granted to the pod, or better, a container within the pod. And Kubernetes will schedule the pod on a node where the requested resources are actually available. In the example, you can see the pod A, which requires two CPUs, and is scheduled on a four CPU node. When a new pod B of the same size is created, it can be also scheduled on the same node. This node now has all of the four CPUs requested. If a pod C is created, Kubernetes won't schedule it on this node and as its capacity is full. This means that those numbers, a developer puts in a YAML file, accurately used by Kubernetes, to manage a real cluster capacity. A quite surprising fact is that Kubernetes varies no overcommitment or request. You cannot request more CPUs than those available in the cluster. This is very different from, for example, virtualization, where you could create VMs with many more virtualized CPUs than the real physical ones. And another very important fact to notice is that the resource requests are not equal to utilization. If the pod requests are much higher than the actual resource usage, for example, you will end up with a cluster that can be fully scheduled, even though its CPU utilization is as low as 10% for example. So the takeaway here is that setting proper pod request is needed to ensure Kubernetes cost efficiency. The second important concept is resource limits. Resource requests are the guaranteed resources a container will get, but the usage can be higher. Resource limits is the mechanism that allows you to define the maximum amount of resources a container can use, like two CPUs or one gigabyte of memory. All this is great, but what happens when resource usage is the limit? Kubernetes treats in CPU and memory differently here. So when CPU usage approaches the limit, the container gets throttled. This means that the CPU is artificially restricted and this can result in application performance issues. Instead of when memory usage is the limit, the container could get terminated. So there is no application slowdown due to paging or swapping or so on, but with Kubernetes, your pod will simply disappear and you may face serious application stability issues. The third fact is about an important, less known effect CPU limits have on application performance. CPU limits work by throttling CPU performance and you may think that this happens only when CPU usage is the limit, but surprisingly the reality is that the CPU throttling starts when the CPU usage is well below the limit. We did quite a bit of research ourselves and we found out that the CPU throttling can start when CPU usage is as low as 30% of the CPU limit. This is due to some configuration on some kernel parameters, but let's talk about what happens. This aggressive CPU throttling has a huge impact on service performance. You can get sudden latency spike that may breach your SLOs without any apparent reason and even at low CPU usage. So now some people including the engineers of Buffer, for example, tried to remove the CPU limits. What they got, you can see it in the chart on the right, was an impressive reduction in service latency. So is it actually a good idea to get rid of CPU limits? Well, it depends. The answer could be no, because CPU limits exist for the purpose of ensuring that applications run fine and coexists with other applications. So if CPU limits are removed, a single runaway application can complete this rapidly performance and expect the availability of your most critical services. This best practice, obviously, is also recommended by Google. So tuning these resources is really important and you have to think properly before doing it. Coming to the auto scalers. So we have VPAs, we have HPAs. Let's talk about VPAs. The vertical portal to scaler basically provides recommendation on CPU and memory request based on the observed pod or source usage. However, our experience with the VPA is mixed. In this example, you can see Kubernetes microservices which is serving a typical urinal traffic. The top right chart shows the latency of this service and it's a service level objective. While below you can see the resource request in terms of CPU and memory and the corresponding resource utilization. It's interesting to see that we left the service running for a couple of days with some initial resources sizing and then reactivated the VPA and let it apply the new recommended settings to the pod. It's interesting to see that the VPA immediately decided to reduce the signed resources. In particular, it cut in half the CPU request but this is likely due to some apparently over-provisioning of this service as the CPU utilization was below 50%. However, with this new setting suggested by the VPA, the latency of the microservices skyrocketed with the microservice no longer able to meet its reliability standards. What is the lesson learned here? VPA autoscaler is based on resource usage and does not consider application level metrics response time for example. We need to evaluate the effects of the recommended settings as they might be somewhat aggressive and cause severe service performance or reliability degradation. But we do also have HPA, the horizontal pod autoscaler. What about it? In this example, we wish a situation where HPA scales out when specific memory or CPU consumptions are met. As a result from one pod, we get two identical pods that provide the same service. This is great, so we have two times the resources we had before to compute the service. What could possibly go wrong? Well, things may go wrong because if for some reason you didn't properly set up your memory limit, for example, you undersized it, you will simply get into a situation where you get twice the out-of-memory kills you had before. But also, on the other hand, you may face a situation where you set your CPU limits to high, but your pod can find such resources directly impacting your service latency because of trolling. But let's say you wanted to play safe and overprovision all your environment and your pods. Everything works properly, functionally speaking, but at what cost? At least you may be wasting two times the resources you were wasting before the scale out if your pod isn't tuned properly. As we have seen so far, Optimizing Microservices application on Kubernetes is a real challenge. Given the complexity of tuning Kubernetes resources and many moving parts, we have modern applications. A new approach is required to successfully solve this problem. And this is where AI can help. AI have revolutionized the entire industries and the good news is that they can be used also in the performance tuning process. AI can automate the tuning of the many parameters we have in the software stack with the goal to optimize application performance, resiliency, and cost. In this section, I would like to introduce you to this new AI powered optimization methodology by leveraging a real-world scenario we have seen. So the real-world case is about an European SaaS provider of financial services whose Java-based microservice application are running on either Azure or AWS Kubernetes service. The target system of the optimization is the B2B authorization service running on Azure, a business-critical service that interacts with all application-powering the digital services provided by the company. The challenge of the customer was to avoid overspending and achieve the best cost efficiency possible by enabling developing teams to optimize their application while keeping on releasing application updates to introduce new business functionalities and align to new regulations. Those results were almost impossible to achieve with the tuning practice they had in place since it was manual and it took almost two months to tune one single microservice with mixed results, obviously, because it was always too difficult to find the right trade-off between performance, resiliency, and obviously cost. By design, the approach was the good old overprovisioning but as a result, what did they have? The result was a huge overspending and a lack of performance, operational efficiency, and business agility. So how did we address this customer? I would like to introduce quickly our AI-powered optimization methodology in practice. The process is fully automated and it works in five steps. The first step is to apply the new configuration AI suggested to our target system. This is typically done leveraging Kubernetes APIs and to set the new value for the parameters. For example, the CPU of the request of the container. The second step is to apply a workload to the target system in order to assess the performance of the new configuration. This is typically done by leveraging performance testing tools. In this case, we used a GMeter test that was previously built to stress the application with a realistic workload. We'll see it later. And the third step is to collect KPIs related to the target system. The typical approach here is to leverage observability platforms. And in this case, we integrated Elastic APM, which is the monitoring solution used by the customer. The fourth step is to analyze the result of the performance test and assign a score based on the specific goal that we set. In this case, the goal was the cloud cost of the application container, considering the prices of visual cloud. The last step is where AI kicks in by taking the score of the test configuration as input and by producing as output the most promising configuration to be tested in the next iteration of the same process. And so we can start again by testing a new configuration. So I've said that we have to have a goal. We can define a goal. In this scenario, the goal was to reduce the cloud cost required to run the authentication service on Kubernetes. As you can see, the optimization blur is declarative. At the same time, you also wanted to ensure that the service would always meet its reliability targets, which are expressed as latency, throughput, and error rates as follows. So how can we leverage AI to achieve this high-level business goal? In the AI-powered optimization methodology, the AI will change the parameters of the system to improve a metric that we have defined. In this case, the AI goal is simply to minimize the application cost. This is a metric that represents the cloud cost we will pay to run the application on the cloud, which depends on the amount of CPU and memory sources that are allocated to the microservices. The methodology that we used also allows to set constraints to define which configuration are acceptable. In this case, we state that the system throughput, response time, and error rate should not grade more than 10% than the baseline. So to assess the performance and cost efficiency of a new configuration suggested by the AI optimizer, we stress test the system with a load test. In this case, you can see the load test and scenario that we used. It was designed according to the performance engineering best practices. The traffic pattern reproduces the behavior seen in production, including API call distribution and think times. Before looking at the results, it's worth commenting on how the application was initially configured by the customer. We call that the baseline configuration. Let's look at the Kubernetes setting first. This container powering the application was configured with the resources requests of 1.5 CPUs and 3.4 gigabytes of memory. And the team also specified the resource limits of 2 CPUs and 4.4 gigabytes of memory. Remember, the requests are the guaranteed resources that Kubernetes will use for scheduling the capacity management of the cluster. In this case, requests are lower than the limit, which is a common approach to guarantee resources for the application to run properly, but at the same time, allow some room for unexpected growth. Besides looking at the container settings, it's important to also see how the application run time is configured. The run time is what ultimately powers our application. And for a Java application, we know that the JVM settings might play a big role in application performance. But the same happens for other languages, for example, Golang and so on.  Notice that the max heap is higher than the memory request, which means that the JVM can use more memory than the amount requested. As we are going to see, this configuration will have an impact on how the application behaves under load and the associated resiliency at us, the ghost. OK, we've covered how the applications is configured. OK, let's now look at the behavior of the application when subject to the load test we've shown before with the baseline configuration. In this chart, you can see the application throughput, the response time, and the number of replicas that were created by the autoscaler. Two facts are important to notice. First one, when load increases, the autoscaler triggers a scale out event which creates a new replica. This event causes a big spike on response time which impacts service reliability and performance. This is due to the high CPU usage and CPU throttling, obviously, during the JVM startup phase. When the load drops, the number of replicas does not scale down, despite the container's CPU usage may be idle, as in this case. It's interesting to understand why this is happening. This is caused by the configuration of the container resources, the JVM running inside, and the autoscaler policy, in particular for the memory source. The autoscaler, in this case, is not scaling down because the memory usage of the container is higher than the configured threshold of the 70% usage of the memory request. This might be due to the JVM max CPU being higher than the memory request, as in this case, as we've seen before. But it may also be due to a change in the application memory footprint, for example, after deployment. This effect clearly impacts the cloud bill, as more instances are up than the required ones, but shows that configuring Kubernetes apps for reliability and cost efficiency is actually a tricky process. So what did we do? We now know how the baseline behaves. Let's start optimizing. What we did was starting experimenting with different Kubernetes and Java configurations suggested by our AI algorithm and measured the results. Let's have a look at the best configuration identified by our AI with respect to the defined cost efficiency goal. This was found at experiment number 34, after about 90 hours of trials, and provides a 49% improvement on the cost with respect to the baseline. First of all, it's interesting to notice how our optimization increased both memory and CPU request limits, which is not all obvious, especially for Kubernetes, because Kubernetes is often considered well suited for small and highly scalable applications. We will dive on this into a minute. The other notable changes are related to the JVM options AI picked. The max heap was increased by 20%, and is now well within the container memory request, which was increased to five gigabytes. The mean heap size also has been adjusted to be almost equal to the max heap, which is a configuration that can avoid garbage collection cycles, especially in the startup phase of the JVM. So let's now see how the application performs with the new configuration identified and how it compares to the baseline. There are two important differences here. Autoscaling is not triggered in this configuration, as the full load is sustained by one pod, which is clearly beneficial in terms of cost. Response time always remains within the response time as it allows, and there are no more peaks. So this configuration not only improves on cost, but also is beneficial in terms of performance resilience. Let's also compare in detail the best configuration with respect to the baseline. Here we can notice that the pod is significantly larger in terms of both CPU and memory, especially for the request. This configuration has the effect of triggering the autoscaler less often, as we have seen. But interestingly, while this implies a kind of fixed cost, considering the pricing of the container resources, it turns out being much cheaper than a configuration where autoscaling is triggered, and this also avoids performance issues. The container and runtime configuration are now better aligned. The JVM MaxSIP is now below the memory request, and this has beneficial effects, and it also enables the scale down of the application should the scaling be triggered by high workloads. Let's now have a look at another configuration found by AI at the experiment number 14, after about eight hours. We labeled this configuration high resilience for a reason that will be clear in a minute. The score of this configuration, while not as good as the best configuration, also provides about 60% of cost reduction. So this can be considered also an interesting configuration with respect to the cost efficiency goal. As regards the parameters, what is worth noticing is that this time, AI pick at different settings that significantly change the shape of the container. It is now has a much smaller CPU request than the baseline, but the memory is still pretty large, which is interesting. The JVM option were also changed. In particular, the garbage collector was switched to a different one to parallel, which is a collector that in some cases can be much more efficient and on the use of CPUs and memory. But let's not compare the behavior of this configuration with respect to the baseline. There are two important differences here. The peak on the response time upon the scaling out is significantly lower. It's still higher than the response time SLOs. However, the peak is less than half the value for the baseline configuration. This clearly improves the service resilience. And now auto scaling works properly after the high load phase replicas are scaled back to one. This behavior is what we expect from an auto scaling system that works properly. Notice that the response time peak could be further reduced. It's simply a matter of creating a new optimization with a new goal of minimizing the response time metric, for example, instead of application cost. Let's not also compare in detail the high-resilient configuration with respect to the baseline. Quite interestingly, this configuration has a higher memory request and lower CPU request, but higher limits, than the baseline. As you may remember, the lowest cost configuration instead had a higher CPU request than the baseline. Without getting too much into the analysis of this specific configuration, what these results show is that the optimization goal changes and CPU and memory requests and limits may need to be increased or decreased, and that multiple parameters at Kubernetes and JVM level also need to be tuned accordingly. This is clearly a confirmation of the perceived complexity of tuning Kubernetes microservices applications. As we're here just discussing just one microservices, but think about tuning hundreds of microservices. In terms of customer result, what did we get? The resizing of the service pods allowed a huge cost reduction in the Kubernetes environment, but first more, we allowed them to automate the tuning of their services, all services, in a matter of hours instead of months, enabling not only the cost reduction, but also enabling improvement in the latency of their application and overall, a better user experience for their customer. There are many interesting configurations that we found out with our study and they maybe were discussing, but I think it's time to conclude with a few takeaways. First one, tune, tune, tune. Any inefficiency is not going to be addressed by your Kubernetes cluster. If you don't think about how to optimize your platform, nobody else will. And the second takeaway is, today's application are too complex. You simply don't have time to optimize everything. AI-powered optimization works and can be the solution to your optimization needs. Thank you. I have a microphone for questions. If you're leaving, do so very quietly. Thanks a lot. This is, for me, at least very, very interesting. One question, can you speak a bit more about the AI use? Is it like black box optimization? Like here are some parameters and just find the combination that seems to work best or? Yeah, so we could talk about the approach we had. I'll just share some insights. So basically that is based on reinforcement learning and we provide as input the parameters that we apply, so the value of the parameters that we apply and the score. So the score of the last iteration that we had. Then after analyzing such a score of the iteration and the parameters, the algorithms that we've used learns how it has to behave and propose the new iteration. If you want more details, you can discuss separately because it's a long topic as you may imagine. How did you took into consideration the CPU memory ratio of the node? Like you had at some point like the high resiliency was up to something like two CPUs and five, six gigabytes of memory. So I didn't really get the question. So like you have the nodes in AKS or something like that? So the nodes in the cluster have a CPU to memory ratio. Okay. And that kind of like if you have like the pod with two CPUs and six gigabytes of RAM, like you can only put one pod on the node. Yeah. So I did take that into consideration the CPU to memory ratio. Okay, so I got it. So in this particular case, what we did is that was focused on the pod sizing not on the, let's say on the node sizing or we didn't think about auto scaling of the nodes. But what we usually see is that the experimental approach allows us to monitor how the system behaves in terms of node capacity while we're on. So we are able to let's say identify if we have a shortage in the CPU and memory utilization on a specific nodes. And doing this experimental approach, we can do it on one single microservice or we can do it, let's say full stack for every microservices which is running on a specific node. So every container which runs in a specific node. I don't know if I answered your question but we can discuss it further if you want. Hi, thank you. I think my question is a bit similar to the last one. It's like if we always, this example seems to optimize versus one service. But I think a lot of times to minimize the cost you need to optimize against all the servers. Otherwise, I guess the one of the example is if it has, it gets a very large pod, three CPU. And if you're not my mean, your node cannot accommodate any other pods. And but I imagine if you start tuning against all the microservices that becomes a huge parameter space and then the AI becomes very hard to run or maybe even inefficient, how does that work in practice? Yeah, basically it may take longer to find a suitable situation but we can test multiple services together. Also, I can add in specific case constraints related to the sizing of the node. So if I want, let's say, to be safe in terms of node allocation, I can add more constraints in terms of, okay, you don't use much more of, let's say, the 30% of the capacity on a node. But I can add it as a constraint in order to, let's say, define a limit in the space that you have to search against. So there's a lot of people following online. A couple of questions came in, I'm gonna read one out. Does the AI provide a predictive ability? Yeah, the idea is to predict the possible configuration that could lead to an improvement, okay? So that is an iterative and predictive approach. If you have followed also the talk before mine, they were talking about predictive approach. This is the case, I mean, this is a predictive and iterative approach, okay? Hey, in one of your slides, you said that you found that if you disabled the CPU limits, the response times actually reduced. As far as I remember, there was a Linux kernel bug in 2020, if I'm not mistaken, and there was already a bug fix about that. And afterwards, presumably setting the limits again would actually eliminate the bug and fix the bug, actually. So my question would be, do you still think that turning off the limits is a way to go? Or? Okay, in my opinion, I mean, I know that in some cases many kernels fix that issue on the throttling. The idea is that this argument of removing the request, they're so slim, it's some kind, by the way, it's something like a war between the people who want to remove the limits and the people who says, oh no, you have to keep that limits. The nice part of what we do is that we don't need to take, let's say, a position since our approach is experimental. So if we find out that removing the limits has better improvements on our application, we can remove it. Obviously, by removing the limits, we take the risk associated with it, okay? Okay, two minutes left. Hi. I have a question on the problem of overfitting to the data. So in machine learning, you typically have this problem that you can, your classifier or whatever can become too sensitive to the training data and not perform well on the real data. So I see we are using performance tests or some artificial tests to set the parameters. So how does it behave well in production and how do you solve this problem of being too fitted to the performance test? Basically, the accuracy of the test is very important. Yeah, basically, I mean, I come from a long practice of performance testing, so performance testing is what I do for a living since I started working. And basically, the requirement to this approach is to find a way to shape a proper model of the, let's say, of the performance test. So you do have, obviously, to use the same version of application, the same version of the operative system, comparable, let's say, sizing in the pre-production environment where the test is configuration. So that is everything that you have to take into account. And for sure, that's the whole problem of how significant is your performance test. So this approach is well suited for customer, for people who are very into performance engineering and know how to shape their traffic well. But we are also investigating the field of doing it, let's say, in production with the real data. Okay, last question. With the GVM-based application, usually you have a problem when you try to base your request to your actual utilization because of the startup time. Yeah. When the application is start and the CPU usage go up. Yeah. How you could solve this. And the other thing is when I bought this resource and usage actually almost the same, I will need to scale fast. All right, so I will need to have to ability to scale fast, which I still need to add more nodes. Yeah. So basically what we did with the GVM topic is that in this particular case, because it's not, let's say, one fit for all solution, okay, in this particular case, obviously the best configuration was the one we didn't trigger the replica. But if you see into the other configuration, which led to some kind of 15% reduction in cost, and that configuration, we managed to find a sizing of both the GVM in terms of mini-pan, money and max-hip, which were increased, where the GVM wasn't suffering, let's say in terms of throttling at the startup of the GVM. Obviously, a GVM always suffers in terms of CPU usage starts. But that specific configuration, terms of mini-pan, max-hip and memory request on the pod allowed us to cut, let's say, the spike in half. But that is something you will have to face ever.  Okay, we are out of time. Thank you for being here, and get in touch soon. Thank you.