So, welcome to the Dr. Gundry podcast. Over 20 years ago, scientists began experimenting with this technique to see if they could actually grow plants in outer space and it worked, which got scientists thinking maybe just maybe the same method could increase energy inside our cells. It's called red light therapy and it's currently being used by thousands of elite athletes and other hyperformers to improve their energy and recovery. And there's emerging evidence that it could actually help you improve the complexion of your skin, diminish wrinkles, reduce morning stiffness and soreness, improve joint health and even regrow hair. So, my guest today is Scott Nelson, the co-founder of Juve, a company at the forefront of red light therapy. We're going to talk about the science behind red light therapy, how it works and how you can use red light therapy to improve your energy. Scott, so glad to have you on the Dr. Gundry podcast. It's a pleasure to be here. This is great. So, let's start with red light therapy.  How does it work? So, we'll start high level and then maybe have the opportunity to go a little bit deeper into the science as I'm sure you'll want to. The best analogy that I can use for red light therapy or in the world of academia it's referred to as photo-biomodulation is to use food as a parallel. Most people that are watching or listening to the show are familiar with how our bodies metabolize or respond physiologically to different macronutrients, right, proteins, carbs, fats. Well, the same sort of concept holds true when it comes to different wavelengths of light. So UV light versus bright white light versus these types of wavelengths of light, red and near infrared light. And so, those specific wavelengths of light actually help stimulate the mitochondria in our cells to produce more energy. And because of that core mechanism of action, you get a wide range of clinically proven benefits that come with this type of treatment. So help me out here. So most of my listeners know about mitochondria. But are you trying to tell me that the mitochondria eat this light and produce energy or come on, help me out? Yeah, so not necessarily per se. It's definitely more of a stimulation. So this is a relatively new field, photo-biomodulation or light therapy. So researchers as well as us on the industry side are still continuing to learn more about the mechanisms of action that are at play. And they seem to be wide-ranging. But at the core is during the cellular respiration process, which is the process that our cells go through to produce this energy, there's certain enzymes that are activated or stimulated by these wavelengths of light in a very healthy way. And because of that, our cells actually begin to work more efficiently to produce more energy. So that's going a little bit deeper at a cellular level. So this is a new field, pretty new. So what are the benefits of red light therapy that are proven? Let's start with that and then maybe you can fantasize about things that aren't perfectly proven yet. Sure, sure. So again, this is a relatively new field. You started the show off with kind of hinting at NASA's early involvement, which helped actually spur on a lot of additional clinical research in this field. But I would say there's probably some core benefit areas that are very well proven. Some of those would include skin health. Very, very clear that these wavelengths of light, primarily red light because it's absorbed in the superficial layers of your skin, your dermis and epidermis, very clear that these wavelengths of light help stimulate healthy skin or help produce those types of results. Another big benefit area would be the reduction of pain and inflammation, primarily participant groups that are suffering from arthritis as an example. Very clear evidence that's peer reviewed and published that showcases that near infrared light probably more specifically helps reduce inflammation. Another big benefit category that's really well documented would be kind of athletic performance and muscle recovery. And you mentioned that it's one of the reasons that a lot of elite athletes and people, fitness enthusiasts, et cetera, are seeing really good results is because they can get on the field that much quicker, whether it's recovering from an injury or just recovering from a workout faster. And so those are probably three big benefit areas, but there's a lot of emerging research as well. So are you saying that this is going to be in a training room in the NFL or the NBA, just to use an example? That's happening right now with our systems as well as others, but teams like the San Francisco 49ers, as an example, are using our big full body systems as part of their recovery regimen. There's several other teams, some I can say, some I can't, but yeah, it's definitely a therapy that's picking up steam. And I'd say it's for two primary reasons. One is because the science is there. If someone just does a little bit of diligence and looks into it, just do simple searches on PubMed as an example, it's pretty overwhelming that there's proof to the pudding, so to speak. And then the other kind of core reason why it's beginning to really pick up steam kind of within those circles is it works. If you give this a shot and use it consistently, there's pretty resounding benefits that most athletes specifically will notice right away, primarily because they're that in tune with their bodies. I know after a certain workout, this is how they normally feel, but after the use of consistent full body light therapy, they're noticing a difference with their responses. So that would be probably two major reasons why we're seeing the demand really increase there. Interesting. Now I know Ben Greenfield, many of my listeners know, is a big fan and he's certainly talked about it and I think he appears on your website. He does. Very good. So help our listeners understand, can you do a double blind study with this device? If you're shining red light on somebody, wouldn't they know it? That's a great question. And kind of going back to your point of Ben, I think he's actually been a great partner of ours, which we're very thankful of, but he's a great example. He's so in tune with how his body typically responds to different modalities, treatments, etc. And that's one of the things he noticed right away when first using light therapy. So I think that's a great example that you brought up. But kind of going back to your questions, how do you effectively study something that delivers bright red light? I think using red light specifically would be very tough to accomplish that. In fact, I think you probably have to get pretty creative. Very creative. But near infrared light on the other hand, near infrared light is invisible to the naked eye. So that's very easy to actually accomplish a double blind type of study. In fact, we're actually enrolling participants right now in those types of studies where you're using a sham treatment that's technically on, but the user, the participant doesn't notice because again, near infrared light is invisible. So fairly easy to kind of study that for sure. So I know you're big boxes and we're actually, for those of you listening, you brought in your little portable box, which I actually have and I'll describe maybe later what I've used it for. You have a portable box that can be either a red light box, but you also have a near infrared light box. You got it. Correct? Yep. But your big ones have both, both the red light and the infrared. Is that correct? That's correct. That's correct. This one is, we call it the Juve Go and it's our handheld kind of convenient portable option and it delivers either or, right, either red or near infrared light. But our larger systems, the modular systems as we like to call them, that allow you to kind of build out this full body system, yeah, those give the user the ability to switch between the wavelengths, red or near infrared or deliver both at the same time. And I would say, you know, 95% or more of our customers use both at the same time because there are benefits to using both. And really the only difference between the two wavelengths is depth of penetration. Very similar mechanisms of action, but, and I mentioned this before with respect to skin health, red light is absorbed in the superficial layers of our tissue where near infrared, on the other hand, has the unique ability to penetrate a lot deeper into our bodies. In fact, it does penetrate through bone even. A lot of the energy dissipates as it travels through bone, but it does have the unique ability to penetrate into deeper levels in our body, which is why a lot of the times when we look at clinical studies that are involving, you know, enhanced cognitive function or reduced symptoms from people that have suffered from some type of brain or traumatic brain injury, typically near infrared light is used because it has the ability to penetrate deeper. Okay, so a lot of people are hearing this and they're going, oh, you know, wavelengths and deep penetration and I hear that we got to be really worried about 5G with all that sort of frequency, help me with this. How is this different than all those scary radio frequency things we're hearing? Yeah, it can be kind of abstract and it's interesting that you mentioned that. We were in the process of doing a customer survey with our customer base, trying to learn a little bit more about like what, how did you first learn about light therapy? What did you think of it? And that's one of the things that has sort of popped up is like, when you first heard about this, what did you think? Like, was it abstract? Was it hard to understand? Were you all in, et cetera? And that's kind of the general consensus is like, I keep hearing people talk about it, but I don't, I can't quite get my head around it. And it's tough because it's like wavelengths of light. It's a lot of physics and, you know, I'm a biology guy myself, but physics is always kind of tough. You know, I think not just me, but maybe for a lot of people. But again, I used the kind of the food analogy before, but another good kind of analogy is that, you know, we're here in Southern California. You see a ton of houses with solar panels on the roof. In fact, it's hard to find, it's hard to find a house that doesn't have a solar panel. And we all kind of understand the concept that solar panels on houses are able to capture energy, right? Light energy from the sun and repurpose it elsewhere. That, again, that same parallel or that same concept holds true within our bodies. We have chromophores throughout all of our bodies. In fact, nearly every cell in our body has some type of chromophore that responds to these wavelengths of light. And in essence, they're, when I say responding, they're almost, they're capturing it, right? And the mechanism, we can get deeper into the mechanism like we kind of chatted about previously, but they're capturing it and repurposing it elsewhere. So our bodies have more energy to heal, to recover, to kind of function the way, you know, we're intended to function. Now wait a minute, Scott. You say we have chromophores in our body. Is that kind of like a chromosome? No. Not really. Yeah, chromophore. That's kind of a nerdy, nerdy type of description. But one of the mechanisms that I sort of alluded to earlier is within the world of photobiomodulation, one of kind of the more well-understood mechanisms, although it's not entirely clear, but it's fairly well-understood or well-appreciated, is that there's an enzyme during the cellular respiration process called cytochrome c-oxidase. And that's a chromophore. That actually responds very favorably. It resonates with red and near-infrared light. So that's an example of a chromophore. And because we're stimulating cytochrome c-oxidase, it allows more natural or more efficient cellular respiration, particularly during the fourth phase of cellular respiration. So yeah, so there's just different proteins and enzymes that respond differently to these wavelengths of light. But at the end of the day, the mitochondria in our cells are just functioning more efficiently. They're producing more energy at a peak level. So are you saying at a basic level, we're basically a light-seeking and light-sensing organism? Yeah, I think that's a great example. And most people understand that our bodies respond to UVB wavelengths from the sun, both good and bad. Access UV wavelengths of light can be harmful. Our skin responds and burns, and we get sort of an inflammatory response. But in turn, our bodies actually respond to UVB wavelengths and produce more vitamin D. So that's kind of a well-understood concept. It's the same kind of thing that's going on with red and near-infrared light. Our bodies are responding very favorably to these wavelengths of light. And then the positive ramifications are increased energy systemically throughout all of our bodies. Are we not getting enough anymore? It's a very good point. And I would say affirmatively, yes. In fact, there's pretty well-documented evidence that, at least here in America, we spend over 93% of our lives indoors now under artificial light. And before moving to Southern California, we were from the Midwest, Minneapolis, specifically. And that stat may seem kind of alarming at first. 93% of our lives indoors. But for me, personally, that was true. In fact, it probably was worse in terms of my exposure to natural light. And that's so drastically different than how our grandparents lived. And because of that difference, I think it speaks to the fact that we are sort of light-deprived in a way. And so, yes, to answer your question, I'd say we don't get enough of these healthy wavelengths of light. OK, so if this stuff is so great, a lot of the skeptical listeners are saying, Scott, OK, this is amazing stuff. If this is so great, why aren't people using this? It's a great question. I've thought a lot about that. And unfortunately, I probably don't have a definitive answer as to why now. Why is this now seemingly taking off? And I would say it's not necessarily taking off. I would say, yeah, I mean, I think we're still very early. But just like any sort of field of medicine, there's a significant uphill climb to any sort of treatment. I know your cardiac surgeon and that's your warehouse. I actually came from the cardiovascular space on the medical device side. To see a therapy or a pneumodality adopted, we're talking years, 10 years at a minimum, at a very minimum. And so I think when it comes to light therapy, I think there's an interesting blend of it's still a very niche community in terms of on the practitioner side, the researcher and practitioner side. It's a very small community. It's not well understood nor covered or reimbursed by traditional payers or insurance companies, which makes a big difference there, too. I would say there's not, and I'll give us our company plug here, there hasn't really been any sort of company to try to raise more awareness for the science of light therapy and the proven benefits of it either. I think we're seeing more and more, maybe, companies kind of come to the forefront. But we were one of the first companies to really bring a lot of awareness and education around the benefits of light therapy to a wider audience. And so I think there's probably a number of different ways to probably explain that, but that's sort of my mediocre answer to that question. All right. Well, so another skeptic would say, well, so the sunlight, obviously, has red component, and it has near-infrared. So couldn't I just spend four hours outdoors every day and accomplish the same thing? That's great. I would say you're exactly right, Joe skeptic. In fact, with people that come to us, whether it's through would-be customers that come to us through email or online chat or some sort of whatever, regardless of the channel, that maybe don't want to go all in with a light therapy device, one of the first things we say is just try to get more natural sunlight on a daily basis, maybe 10, 15, 20 minutes even. You're going to be in a lot healthier environment doing that versus not doing that. That's an easy, it's a free step, et cetera. But I think really the question is, do we have enough time, do our modern lifestyles allow us to get outside on that sort of a consistent basis? And in some scenarios, we don't really deal with this in Southern California per se, but there's a lot of areas throughout the world that just don't get a lot of natural sunlight either. Minneapolis that I mentioned where you come from is a great example of that. Not a lot of natural sunlight. And so I'd say to kind of answer that question and come full circle is that I could say the same thing about Whole Foods, right? You don't maybe necessarily need to buy a ton of supplements, just get it from Whole Foods, but some people just don't have the time. They have busy lifestyles, they have kids, they have work commitments, et cetera, it's just hard to do. And so using a high quality light therapy device is an easy way to sort of supplement that into your lifestyle if you can't afford to get out and expose yourself to natural sunlight on a consistent basis. Got it. So now my good friend Joseph McCollum, we've had debates about vitamin D supplements and he said, well, you know, you can get all the vitamin D you want by spending two hours every day out in natural sunlight. And I said, not too many people, I know him, spend two hours in a speedo walking the beaches in Fort Lauderdale every day of your life. So that's an impractical recommendation. And I think you're saying the same thing. You're exactly right, yeah. Speaking of Joseph McCollum, he would say, just go to your hardware store and buy an infrared light and just don't burn yourself with it. Is there a difference between buying a heat lamp at a hardware store? Yeah, you're hitting on some great questions, Dr. Gundry, I can tell you. No, you've done your research. No, it's another one that really comes up quite a bit. What's the difference between something like this and an infrared heat lamp that you can pick up at a Lowe's or a Home Depot? Great question. And there is quite a bit of difference in short, but to get a little bit more granular, I've referenced these red and near infrared wavelengths of light throughout our conversation so far. And clinical research has shown that it's actually very specific wavelengths of light that actually help us the most help us to the greatest degree, namely red light, visible red light in kind of the low to mid 600 nanometer range, as well as the wavelengths of near infrared light in the low to 800 nanometer range. And so when you look at that sort of that window, there's also clinical studies that showcase that wavelengths of light in the 700 nanometer range really don't have any sort of biological impact on our bodies at all. And it kind of all goes back to how do our cells respond to different wavelengths of light? And so our cells, it's very, very clear based on thousands, literally thousands of published peer reviewed evidence that you need very specific wavelengths of light. And if you take a heat lamp example, most of the wavelengths from a heat lamp are actually not near infrared light, but actually mid and far infrared light. That's why they produce more heat. And so in essence, if by using infrared heat lamp, you're not really getting those clinically proven wavelengths of light. And most of that energy is dissipated as heat because it's mid and far infrared. So it's just a different biological response and really quite not very different than a light therapy or photobiomodulation device. So how did you actually, let's talk about juve and how that came about, how did you actually, first of all, why did you form the company? And how did you figure out how to make these wavelengths of light? Yeah, so I've spent, I mentioned this before, I spent most of my professional career in the traditional medical device space. And in fact, that's why we were based in Minneapolis. I was with Medtronic at the time, primarily the peripheral vascular space. So probably interacted with some of your peers. They make all of my catheter. Yeah, some of your peers. God bless them. Yeah, yeah, so when I, I think it's a nice transition kind of going from this idea of like a skeptic to kind of our story because I was that skeptic. I was in what I thought was real, real, real stuff. Yeah, real stuff. Helping, you know, doing real things. And so when my wife and her sister, Melissa, bought or purchased a quote unquote red light therapy package at a local spa in Minneapolis, I was, you know, they're into, they're more accepting of newer things than I am. And so I was completely skeptical. I thought red light therapy, that's kind of, you know, everyone does the natural chuck of like, that can't be, you know, that can't be real. Totally woo-woo. I had the same type of response, but they had really good results. They went consistently for about eight to 10 weeks to the spa four to five times a week and saw really good benefits, primarily related to their skin health, you know, reduction of eczema, age spots, stretch marks from pregnancies, et cetera. And so I thought, they were really into it. I thought, okay, well, maybe there is something to this. And so I went right to PubMed. I did the same thing that Joe Skeptic is gonna do. Go to PubMed, search for red light therapy or photobiomodulation and another popular acronym is LLT or low level light therapy or low level laser therapy and you will find endless numbers of studies. In fact, there's over 3,000 published manuscripts on light therapy. So I was quickly blown away by that by the sheer amount of data. And, you know, so the next step for me was like, who makes these devices? Where can you get them, et cetera? And what we found is that at the time, this was back in 2015, at the time there was very small underpowered devices available on the market or super expensive laser-based devices. And there was this kind of gap in the middle of like, you know, why isn't anyone producing kind of a device that you can use at home? Because that's really ideally where most of us should be using the therapy at home that delivers clinical grade power and ideally over a broader surface area, you know, because our kind of core thesis as a company is that full body light therapy would be ideal. If it works for small areas, it's got to work for your entire body too. And so we didn't intend to start a company. We just thought, huh, there's no one really talking about it. There's not really, there seems to be kind of a product gap here. And so we started doing some early prototyping. My brother-in-law is an engineer, so he started kind of messing around with some prototypes. And kind of, we were off to the races in early 2016. So it's really kind of a combination of kind of quote-unquote stumbling into sort of a therapy that was based on a lot of science, but yet no one was really talking about and the fact that there wasn't really devices that you could use at home that treated large areas of your body. So I hope you didn't quit your day job. I had a pretty good job, so it took me a while, took me probably a lot longer to quit.  And we were fortunately, did that fairly quickly. And so we saw some early traction in the marketplace, particularly in the kind of natural health kind of longevity communities. People that listen to this type of show and are interested in optimizing their health in the most natural way possible. And yeah, we saw pretty strong early traction and it's been kind of a nice ride ever since. How did you break into athletics? I mean, trying to convince, you know, 300 pound lineman to sit in a box of red lights, was that an easy sell? No, I definitely wouldn't say it's an easy sell. That's definitely a much more difficult community to convince, but there's a guy on our team who's here in the audience, actually Wes, who's been pretty interested in kind of trying to cross the chasm from a therapy that's supported by science and very quickly being adopted in kind of the natural health community to elite athletes, you know? And it's interesting, a lot of those elite athletes, whether they're with the San Francisco 49ers or the Chicago Black Oaks, are actually listening to a lot of shows like yours. They're listening to your show. They're listening to Ben Greenfield's podcast, Dave Asprey's podcast, who I know you've had on recently. So there's an interesting kind of overlap between these two communities. And so we started to see some interest from those types of people in the professional sports world that would listen to your type of show. And we've seen kind of a slower kind of traction that's really begun to build over the past probably six to nine months within those types of circles. Okay, so you've got a box that everybody needs. So what are the challenges of convincing everybody that they need this? Yeah, so we talked a little bit about this. It seems like there's a lot of people thinking about or talking about red light therapy as of late, but I think we're still very early, right? And so this challenge of convincing a much broader audience that light is actually a healthy thing, that they should consider light maybe equally as important as food in kind of their daily routines. That's not an easy conversation, especially when we're trying to do that sort of in a virtual kind of digital environment. So it's definitely a challenge. And I think just one of the core reasons is because light is sort of, like we talked about earlier, it's kind of abstract. Most people just naturally think of it as I turn on the light to illuminate a room and have no idea that can actually be pretty beneficial for our overall health. I think we're making progress. And as a company, we allocate a lot of resources towards education. So I think we're making some progress, but we've got to stay after it. And if you're still kind of skeptical after listening to this conversation, I would say don't listen to us, don't listen to me, go right to Pamen. There's a lot of science that supports this type of therapy. And if you go into it with even a little bit of an open mind, I think you'll be pretty impressed, just like I was. The athletes are using it and the super performers are using it and Ben Greenfield is kicking ass and taking names.  Yeah, so I think there's more and more awareness thanks to people like yourself, people like Ben and Dave Asprey and other I'm gonna use air quotes here, health influencers from a wide variety of sort of arenas that are beginning to help us raise more awareness about light in general. We started to see some early signs and I think it's semi well understood that excess blue light at night as an example can impact our circadian rhythm because it tells our bodies that we should be wide awake when in reality we should be beginning to rest for sleep and in fact you've seen even big brands like Apple begin to implement some certain things with even iPhones around screen time and allowing you to easily adjust like the brightness of the screen at night, et cetera. So I think there is this kind of burgeoning sort of interest in light but blue light is very different than healthy wavelengths of light and so what we're trying to do is help people understand that similar to food, right, similar to how they think about whole foods and macronutrients and how to maybe reduce lectins in their diet as an example, they need to be thinking about light as well. That should be part of that equation, sleep, fitness and training, food and light. Light should be in that equation and so how we're doing that, like I said earlier, we allocate a lot of resource towards education and at the end of the day, whether someone purchases a device from us, hopefully they will or someone else, that's one more person that understands the importance of light and so I think we're demonstrating it there and as well as just kind of at a grassroots level we have a lot of people that purchase even our small device like this and want to go bigger. In fact, I mentioned some survey data recently, there's pretty strong evidence that they're not only telling their friends and family about the impact that lights had on their health but also they want to go bigger, they want to go with a larger device so I think the combination of those two things is allowing us to kind of slowly push the boulder up the hill when it comes to raising more awareness about the benefits of red and near and fred light. So are you saying go bigger, go home? So what we're talking about this box, what's the dimensions of this box? Yeah, it's a little bit bigger than an iPhone, you know, slightly bigger than that, maybe a little bit thicker obviously but the Juve Go is a really nice option, delivers either red or near and fred light, completely wireless and rechargeable and in fact delivers the same type of power as our larger systems. So this little guy will do both, red or near and fred? Not at the same time, you have to purchase one or the other. In the future, I think we'll probably allow for that type of functionality but for now it's either red or near and fred but with our larger systems, the modular systems, those do allow you to use either red or near and fred or both at the same time and really the cool thing about that modular system is that I mentioned earlier our core thesis as a company is that full body light therapy is ideal and what's interesting about those modular systems is you can buy a smaller device, something like the Juve Mini that we call that's probably about, I don't know, four times the size of that and you can start with that one and then add to it over time and the devices connect together both physically and electronically and so one device controls the whole system. So it's kind of like Lego block. So you can start small, try it out. If you don't like it, you can return it through our free return trial period but if you do, you can add other devices onto it and build out this full body system over time which is kind of a really cool feature. Well, the reason I wanted to have you on is I don't know, it was probably a year ago you guys said, hey, why don't you try out our device and I've actually done a lot of research on red light therapy and near and fred light therapy with a company that I was a devising and I won't mention their name.   About the same time as actually this time last fall I was participating in a exercise of resistance stretching and I had an old knee injury. I had a meniscus tear 20 years ago that had healed without surgery. I actually couldn't even walk because of my left knee and I actually had to walk to another appointment and by the end, my leg was pretty much stiff and it remained that way and no matter what I did stretching, heat, you name it, didn't try NSAIDs but took all sorts of other temps and it wasn't getting any better and I'm going, this, my dogs hated it because I couldn't run with them and this thing arrived almost simultaneously with it and I said, you know, what the heck? And so I started focusing actually on my left leg and knee and within about a week or two weeks I'd forgotten this thing started getting better and it completely resolved and the only thing I changed was, you know, your stupid box and I'm going, wow, this is pretty interesting and so I became a fan just because of that and then a few weeks ago we were hiking in Deer Valley, Park City, Utah, at about 10,000 feet. We do a lot of hiking but not consistently at high elevation and we would hike for about four hours and usually after that sort of workout, my wife's pretty aggressive on these hills, my quads would be achy the next day so the first night out, knowing that my quads were going to be achy, I had actually taken your little Juve Mini and I just sat around afterwards and I just spent 10 minutes on each thigh, literally just sitting there, my wife said, what are you doing?  And so the next day and I would have sore quads I can guarantee you, my quads weren't sore and again, is it a placebo effect? Well I should have done one and not the other but you know, to be a true scientist but I was actually impressed and so I did it, we did four days of hiking and about four hours every day and I did it every night so I'm with the athletes. I'm not a super marathoner, thank you very much. My wife will routinely hold our near and Fred go at night on her forehead, the same thing at Dr. Hamlin who's on our scientific advisory board is a pretty well-known photo medicine researcher. He does the same thing, if he ever struggles at night he holds the near and Fred device right on his forehead so my wife is doing the same thing and you'd think I wouldn't be skeptical at this point but I'm like, really? Is that really doing anything? And sure enough, I started to do it now and my sleep scores are better at night. So it's kind of one of those things where it can seem so hard to kind of grasp. Are you saying that I can increase my heart rate variability by holding your near and Fred device near my forehead? I think there's pretty strong evidence that maybe suggests that and so I think I guess that the point I'm trying to make is that if you could be skeptical for a wide variety of reasons but look at the science and then give it a shot and I think you'd be hard-pressed not to find some benefits whether it's something related to like recovery or even like sleep as an example which is another very common kind of result that we hear back from customers as improved. All right, so I've got my aura ring. There you go. And I've got my whoop band. We're testing them out. You're not double-fisting it per se, double-handing. Double-handing, so. Yeah, so I'm gonna, I'll check that out. Now, tell me about this crazy idea that guys are increasing their testosterone by shining light on certain areas of their body. That's true, certain areas indeed. And I think Ben Greenfield's sort of popularized that treatment that's still very popular today. In fact, we see some pretty interesting images on social media as you can probably imagine but when you look at the, that's one of those emerging categories that I mentioned earlier where there's not maybe as robust science to support it now but there is some that's pretty promising and what it tends to showcase is that red light and near infrared light can be both beneficial for testosterone increases as well as sperm mobility for those that maybe struggle with infertility too. So it's interesting evidence but anecdotally, what I think is probably even more compelling is that we have a lot of, we work with a lot of health influencer partners whether it's someone like Ben Greenfield, yourself, Mike Muntzel or Luke Story, people that are already pretty healthy. I mean, they're doing all of the right things and some of them have gotten their blood work done or at least quite a few of them have gotten their blood work done before introducing full body light therapy and then after, after anywhere from 60 to 90 days and the results are pretty dramatic and they're showing increases in not only free testosterone in both categories of testosterone. So, yeah, so free in total and so it's pretty compelling that there's some, it does work, it does work for those areas and that's a category that we're beginning to study more and more in clinical studies. In fact, we've got some, we're enrolling participants right now with a group in Minneapolis studying both hormonal results or hormonal markers in women and men and the results are pretty staggering. We'll hope to release those in the near future but it's pretty overwhelming that light therapy has a pretty interesting positive impact on hormonal health. Now what about mood? Can this stuff help mood? Coming from Minneapolis, where everybody's ready to kill themselves in the winter. Yeah, and I think probably a good portion of your audience is probably familiar with the happy lamps, right? The sad lamps, the seasonal effective disorder lamps. I think there's some pretty, there's some decent evidence that suggests those do work for types of depressive disorders but I think what's even more interesting is that red and neon for light is beginning, we're beginning to learn more about it's benefits for those areas as well, whether it's just kind of mood in general or if it's a more symptomatic depressive disorder. In fact, there's some really interesting evidence over the past two or three years that's been published that really does showcase the use of near infrared light significantly helping people with symptomatic depressive disorders and I like Dr. Hamlin's quote who I referenced earlier, he said he thinks you get a little bit more bang for your buck with a red or neon for light device versus something like a happy lamp because you get a lot more wide ranging benefits with a device like that. Mood being one of those but also muscle recovery, reduction of inflammation, sleep benefits, et cetera. Well Scott, thanks so much for coming on the program. Where can people find out about Juve? So the best place is probably our website juve.com so J2O's to these juve.com. If you wanna check out reviews, we publish all of them, even the negative ones, check out our reviews. That's actually really, really interesting to see there's thousands of them now that really tell a pretty interesting story about the wide ranging benefits that people get from this type of therapy but if you're more into the science and kinda wanna nerd out on that, we do have a lot of educational articles on the learn library on our website as well so check that out. So those would probably be the two main areas and then we're at Juve social on social media. So Instagram, we're pretty active there so at Juve social. Because I'm sure you're misspelled all the time. Why two O's and two V's? Two O's, two V's. So this was back in 2015, kind of an interesting story. We were looking for short.com. We wanted it to.com, we wanted it to be kinda short and catchy and so we were trying to, Juve is a play on rejuvenate cause that's really what these wavelengths of light are doing. So it gets misspelled sometimes but I think there's a lot of people that kind of like to Juve now or they find themselves juving on a daily basis so it's kinda cut on. Very good. Thanks again for being with us. And now it's time for audience question. Jonas Tack on YouTube asks, wouldn't you have to drink a lot of wine to get any of the polyphenol benefits? That's a great question, Jonas. It turns out that if you wanted to get about 100 milligrams of resveratrol which is one of the polyphenols in red wine, you would probably have to drink 150 bottles of red wine to get that much resveratrol. But there are also lots of other polyphenols in red wine besides resveratrol and one of them is another small molecule called curcetin or curcitin depending on how you like to do it. So it's the combination of all these ingredients. Now, no one would suggest that you should be drinking 150 bottles of red wine to get the health benefits of resveratrol and that's why there are some pretty doggone good resveratrol combinations out there. But just because you're not going to get 100 milligrams of resveratrol every day from your red wine doesn't mean that if you do drink, red wine is probably the best source of polyphenols in alcohol. And like I always say, if you don't drink, don't start. And thanks again for joining us on the Dr. Gundry podcast. We'll see you next week.


Well it was funny because I was actually under way when she called so like everyone on the ship was going crazy trying to find me and I was out on liberty at the time. The main elements had to be a serpent, six stripes for the six original frigates and you had to have the motto, I swit my strike. I was pretty excited, my wife was super excited for me.


Good morning. Just give it a couple more minutes or a few more minutes, so some more people to join. Okay, thanks for the link to the demo. So please set yourself as an attendee. All right, so it's 802, almost 803. So thank you everyone for joining. So today we have CORE and the project and Clive is here, and thank you for deciding to show us the project and what it's all about. Excited to learn about it. I think this is probably the first AI machine learning type of project that has presented in cigarette time, so we're pretty excited about it. Yeah, so go ahead and take it away. All right, cool. Yeah, great. I'll share my screen if I can. Let's see. Cool. Let me see if I can move this. Cool. Can you see that? Yeah. All right, cool. So great to see you. So I'm Clive. I'm CTO Selden, so I'm just going to give you an introduction to some of the projects that we work on, and yeah, good to get your feedback and see how it connects to the things that you're interested in. That would be an interesting thing for me. Great. So I'm just going to go through some sort of rationales of what Selden's trying to do, sort of set the landscape. So one of the things is this paper from Google from 2015, which is really setting the scene for what we were trying to do was that saying, you know, when you do ML code, you know, the data scientist sometimes think, or even the whole organization thinks it's just, you just got to write your little algorithm, and that's it. But actually, there's a whole set of other things surrounding it, and this was a paper by Google from their own internal analysis, and some of you might know it's got quite a lot of fame afterwards, where the size of the boxes, the amount of code they had to write for these other things surrounding the ML code. So there's a lot of technical debt that gets created in an organization in real Selden, and that was sort of inspired us to start doing what we're doing is that we wanted to solve those parts of the technical debt in terms of serving infrastructure, analysis tools, and monitoring of your machine learning, et cetera, and so really help our organizations in that area. We've seen a trend in recent years that the organizations are looking for best to breed parts for the whole ML lifecycle, you know, from initial data analysis, and food to the training, and then the serving, and that's really the direction we see that's taken, obviously also with the cloud native world and tools fitting into that, we'll discuss that a little bit. So that's one of the rationales for Selden to help organizations in that area for these things with their ML code, so that data scientists can focus on that part, the core ML code part. So another way of looking at what we do and sort of the issues that we find in organizations is one, you have the data scientists on one side and they've got their own set of tools that they know a lot about, you know, obviously all the training tools, and TensorFlow, Nvidia, Spark, et cetera, and then you've got on the other side, you've got the DevOps who've got their own tools in the cloud native world and all the tools they work at well, and there really is sort of quite a hard divide between the two, the data scientists don't really care sometimes too much about the DevOps, what the DevOps are doing, and the DevOps are scared about the machine learning and all this stuff which they don't really understand, it's not so simple as a normal app, they're going to just deploy onto their Kubernetes cluster, and so you really get this issue of these two teams working together, and that's also what we're trying to help, trying to put some DevOps into data science and data science, help them move this stuff out so they can get it into production to really speed up that time to get those machine learning projects from just being a project out into production and making a difference for companies and organizations. So that's another rationale, this one obviously should make a lot of sense to you guys, this is, you know, we're saying nothing new, you know, a person as a data scientist they've got toolkits, they want to really deploy their model scale and update it, you know, onto a compute cluster with CPUs, GPUs, and TPUs, and obviously we all know about the sort of stack that's been built up over the years from the container one times, then obviously Kubernetes is coming in to orchestrate more complex projects on top of those container one times, and then projects such as Istio for the service meshes, which allow you to do all the things that that allows you to do in terms of handling that sort of network management, and then interesting projects like Knative and other ones to really build on top of that for serverless, and there's that sort of gap in between that and what the data scientists need to do. And so really the issue is if you want to do use that stack, you've got all those things that I think we all know about it, but data scientists are probably less interested in, so I'm not going to go for the big list, doesn't think it will all be familiar as well, so Kubernetes, like aficionados here, but you know, it's quite a challenge for people to do that as data scientists or companies to get all these things when they're trying to get their machine learning out of the stack. So really, what we're trying to do here is with our projects, Seldom Core and KF Serving, is to build on this stack and sort of bridge that gap to our data scientists, sort of focus tool, which is sort of cloud native and allows them to get their machine learning into production. So that's another question now for us. And then one of the last things is also very important to us is the whole ethical side. So obviously there's a lot of interesting society of how AI is going to be used when it's put into production. And so that has built up a lot of momentum. It's obviously key for all the projects and companies that started to use AI and apply it to their customer bases. And then there's now beginning to be a lot of regulation in certain areas, you know, sort of GDPR in Europe and other regulations coming in, which need to be applied and trying to build up some sort of rules for how AI can be applied. But there's really still, there's still a space of how there is actually going to be how companies need to apply those regulations and what is the best practice when you actually put your machine learning in production, how do you know that it's actually not harming your user base and you know, it's really doing what it says and can be audited, etc. And so there's a gap there and that's also what we're trying to fill with some other tech. So this space is really emerging, you know, from how society is reacting at the base to AI as it affects them more. And then in the middle layer of companies and regulations being applied and how that's working, I think that's still being worked out, you know, how that's going to apply and then the tools at the top. But it's a space that we're in and we're trying to also help out there. And then finally, I just wanted to bring up one more sort of academic side is that obviously there's really big conferences every year on machine learning. One of the biggest is ICML. And the first the first time they had like a workshop ICML on challenges in deploying and monitoring machine learning systems. So it's really being viewed also in the academic world, what are the challenges when you try to put machine learning into production. And we actually were lucky enough to get two talks into that workshop or one on serverless inferencing on Kubernetes by myself. And then there was another talk on the work we're doing in Alibi for some monitoring and explainability of models in production. And there's a lot of research areas that are being applied in this area of what are challenges when you're actually deploying machine learning models. So I just wanted to sort of highlight that as an emerging area that's coming also from academia with a lot of research being done. So okay, that's the background. So if you understand what we're trying to solve at Southern and what's the projects that we actually work on to help to achieve some of those goals. This is our stack of projects that we work on. So starting in the layer in the middle, this is all open source in the middle. So the Southern core, which is probably the most mature project, we have four or five years, maybe I can't remember now, but it's got a lot of traction. That's going to into more detail. That's providing it like a abstraction, a custom resource value to define how you want to put your machine learning model and or whole inference graph out into production, managing it, updating it, scaling, et cetera. And then more recently with in the last year or so, we've been working with a group of companies on KF serving, which is building it's very similar aims. It was building on the stack of K native. So looking at serverless, how can serverless help in the area of machine learning deployment? So that's sort of the middle layer. And then at the bottom there, which feeds into that middle layer, a suite of projects that we've been working on to focus on some of the things you need to do. Once you've got your model out, which is obviously the core concern for a company, creating a data science model and putting it out there, then the things that surround it. So things like explanations. Why is the actual model giving the response? It's actually given try to understand that for various stakeholders, be it customers, be it auditors or the actual data scientists themselves who want to understand how the model is behaving. So I'll discuss a little bit about that. And then we have no project called alibi detect, which is focusing on the ways you need to monitor models when you put them into production. So things like drift detection, outlier detection. These are like adversarial detection and stuff like that. And these feed, this is also open source, you can find them. I'll give links to them on GitHub. And these feed into, and then these projects are buffed to actually allow you to actually deploy those models on communities and add these techniques surrounding your model. And then at the top, obviously, as a company, we need to make money. We have a project which is our core enterprise project, which is bringing this all together for companies to provide a full enterprise stack for machine learning, bringing it with this open source, like an open core company and feeding it in and then providing a full solution so companies can scale and manage their models. I'll give a quick glimpse of that at the end. I'll focus more on the open source. So going to a bit more detail on some of those projects. So we have sold and core. So what is that trying to do? That's basically allowing you to build up a whole graph of containerized components that are doing inference and put them together, allow them to be reusable in different projects and then manage that for you. So that graph is defined by a custom resource. So we have our own operator that's running that. And you define by that custom resource the various components, various customizations you want in terms of pod customizations and what the model is, what type of model it is, and how you want it to actually connect together. So you might define just a single model in your inference graph, or you might have much more complex things. We have customers, for instance, doing large inference graphs with things like multi-on bandits, which decide in real time based on the input traffic, which of the underlying models this particular request will go to. So you might add that in with a suite of different models and then you might tie in further things earlier in the inference chain, maybe some feature transformation that needs to be done before the request gets the model, maybe some transformations that need to be done after the request has come back from the model, then adding in other things like I discussed, like outline detection and explanation. So basically we allow you to define this whole graph in YAML, obviously YAML or JSON as a custom resource and then deploy it and then manage it and update it. So that really handles the core thing of what data scientists want to do, deploy, scale, update their model, and they do that by updating that custom resource with the various definitions. Then the next question I suppose is how do they get the core components to actually derive these sort of containerized boxes in their inference graph? There's really two ways that we provide. One is out-of-the-box machine learning servers. So all they need to do is say, okay, here's my artifact on S3 or Google Storage and then we'll fire up a, say, TensorFlow serving or Triton server for that artifact and manage it and tie it all into their inference graphs so that the actual API can be used through that. That's one way and that's very popular. That's obviously the most simplest way that they can use, sell them once they've trained a model and got that artifact onto some location, either on the cluster, as I say, or in the cloud in some bucket. Then the other way, which is surprisingly actually quite popular as well with lots of organizations, is if we have particular language SDKs. So in different languages, if you have custom code and what we allow is the data scientist just to focus on the prediction codes. Say in Python, we have a Python wrapper, as we call it, and they can just focus on the predict code of the Python wrapper and maybe some of the stuff to set up them all at the start. And then we'll manage that in terms of wrapping it up into a microservice, allow them to containerize it, add in the metrics, tracing and other parts so it can easily be slotted in here as part of the inference graph. And that's actually quite popular with a lot of our customers who have like custom code in different languages where people using Java, some people using R, there's actually a talk at the last KubeCon Europe about guys from NASDAQ using sold and core, and they've got some models in R. So we've various language wrappers that allow you to wrap up your code, so allow the data scientist just to focus on the machine learning code and then put it into the graph. Yeah, I have a question. So yeah, you mentioned the models on the right side, and they can be like artifacts in S3. But once those models are created, the models are created, are they loaded into memory or some other place where the inference actually can happen faster? Or is it still in S3 or is it still, or like maybe EBS storage or maybe typically some of these models may be kind of large, right? So yeah, that is quite a good point. So what we allow people to do is define a location where it's from, as you say, S3, and then the actual model server will download the model from S3 onto the local volume and then run it in memory. So there is still work to be done in terms of very large organizations that maybe have lots of using the same model from different locations, how you can use caching, and we're looking into that sort of caching layer that sits between S3 and the actual local model server, which will have the model in memory. But at present, we provide sort of standard downloaders, like a in-it container that runs once the model server starts up. It has the in-it container as given the location of the model and it understands how to talk to S3 or GCS, et cetera, and then it downloads it locally. And then the server starts in the main pod and reads from that sort of local volume and reads the model into a sort of memory. Got it. Cool. Thank you. Cool. And so one other thing we add as part of what we do is a service orchestrator. So obviously, you just define this graph and you don't need to decide how this graph connects together. Sorry, that's not right. You can define how the graph connects together but will manage the sort of request and response flow through the graph. So we add in a component which you don't see here, which is the service orchestrator, which is going to take the request and manage that flow. It's going to say, okay, first I need to call, say, the feature transformer. And then once the response comes back for that, they've defined this to go to a multi-unbanded and I'll send the response to that. And then the multi-unbanded says, okay, I want to send to say model A and in this case, I'll send the request to model A, the model A responds and it will send it back for out the graph. So that's an extra component, a sort of sidecar that we add in to the graphs. But apart from that, they've got complete flexibility so they can define parts of these to be in certain pods that scale in a sort of different way. So to say the feature transformations could have like a HPA that scales in one way and the models can be on like another pod that's going to scale on a sort of different matrix. You've got a lot of flexibility in how you define it as well. I mean then in terms of how people use it, so in terms of what we've done and in terms of our sort of life cycle, we've focused initially more on so RPC use cases of real-time machine learning inference and that's probably how most of our customers come to us and use us at present. But also what we're seeing is customers do want like a unified solution. So once they've requested the model, they don't just want to expose it via RPC or they want to also send as a batch request to it or use a streaming but I say Kafka or Knative and all that same that. So we allow them to do all free basically and it's very easy to use the same components irrespective of how they're going to sort of consume that model in their organization. So that's sold and core and I'll just go on if there's no questions. Sorry, I don't know if you can hear me though. Sorry it's a bit low volume but yeah. You mentioned something about containerization of their model initially. Is that part of what you provide as well or do they they are assumed to have used S2I or whatever when they? Yeah so that's also a good question. So we provide in our docs examples of how to use S2I and we have actually S2I builder to allow them to easily use S2I that brings in the appropriate sort of dependencies say in Python and stuff to actually wrap the model easily. But we also provide them docs for how they can just use a standard so Docker file to actually create their model and we have people using different methods because obviously not everybody wants to use this S2I. There's some people like it some people don't and there's obviously many ways of building that container. So we want to be quite agnostic to that so yeah there's not like a unified solution that's that we provide them with different sort of resources to really create that container in the way that they want to. Okay I was just trying to figure out the scope of it. Okay great. Okay cool. So that's sort of cool. So as I said there's never an area that we work on which is sort of looking at the things you need to surround your model with and that's two projects that we work on. One is AlibiExplain and one AlibiDetect. So AlibiExplain is looking at explanations so looking at once your model is out there how if you get a particular prediction back why did the model give the actual prediction? We have different techniques for this both black box and white box. So black box has advantage it just purely treats the model as it says as a black box and you just talk to the model over an API. So the big advantage of that is you don't care how it was created what technique. It could be like a deep neural network. It could be like a tree based model. It could be just a sort of simple linear regression. It doesn't matter because all the techniques do is just query the model many many times normally by changing some of the inputs from the initial input and trying to understand how the model is actually responding to those slightly slight changes in the input. From that it builds up a picture of how of what the model is taking into consideration and then it can give like a human understandable response. So that has a lot of interest and especially in organizations who want to keep it quite separate. So the team that they built the model they can keep it completely separate from the team that needs to explain it. It is obviously also more challenging though because it's treating as I said purely as a black box so you're just talking over an API to it. So we also have white box explanations which is focused on if you know how the model was created so you actually have access to the model weights and if it's a neural network you have access to the Keras saved model then you can load that saved Keras model and then you can look at the weights and you can do analysis of that. Again probably doing various techniques to actually understand how it's working or if you've got like a tree-based model or you can actually load that tree-based model and try to understand it and give an explanation for this particular prediction. So each has both their pros and cons and it's not like a single way to do it. And also just to quickly jump into the Alibi project just to illustrate the sort of different ways of looking at it. We have a large number of different state-of-the-art techniques and the key thing is those techniques give different ways of viewing those explanations. Also they have different focuses on the type of input data so it could be that some techniques work with classification. Actually most of them here work with classification but some are more focused on regression and also some are more focused on certain types of input data. So if your data is just tabular then fine and also if it has sort of categorical variables you know is it some work better or worse with that. Some work focus more on text, some on images and other features like that. So there's not really one way of solving the explanation question in terms of machine learning models and also the way these models give their responses is also quite different and so some are suited more for data scientists where I say some like sort of counterfactuals are maybe suited more for the actual customers. So for example counterfactuals would tell you what do you need to change. So if you have like a loan system say that sort of rejected your loan it's like an automated system and if you use the counterfactual explanation technique then it could tell you what do you as a customer need to change for this system to change its mind and actually give you the actual loan and so it's easier to understand that way whereas other techniques might help the actual data scientists understand what features are being taken into account in terms of how the models. So they all have their pros and cons and we need to work with the organization that's heard into which one would be most appropriate for the actual models being put out. This is also quite an active research area and also the other challenge is there's also quite complex so you need to train some of these on the actual training data. I'm saying even though some of them are black box so they don't care about what technique you use to train your model some of them need to understand the training data so if they're going to actually sort of perturb the input then they need to know say if it's an age if this feature is an age then it only makes sense to sort of perturb it between say one and 110 or something it's not putting in the value of 1000 and seeing how the model behaves because it's going to give you strange results. So there's certain you know things you need to look at in terms of that. I have a question so yeah this is very interesting so um do you actually integrate with some of the rafting tools I would imagine like some people want to understand you know some of how these models actually behave so for example like something like Tableau right so people want to see how people are using the model or how or how it's making its decisions right so oh yeah I'm absolutely I think there's quite a challenge there I mean it's one of the large challenges and so explanations is really how you show that data to the user and how you tie it back into the actual models being used and actually maybe I can quickly sort of dip into our enterprise part just to show you the explanations in action hopefully I can just give a quick demo so this is one model this is our like a top-level viewing panel cell and deploy which shows the different models you've got running I'm not going to go into too much detail here obviously we're focusing more on the open source but this will help hopefully answer your question so I've got one model here which is the sort of incomes of loan categorization model which is based on making predictions based on sort of demographic features is actually a standard sort of data set we have it's actually taken from the US census in 1986 so stands for open data set so we've got various demographic features and it will predict whether the person's going to have high or low income so we can take a particular example send it to the model so it's a like a binary classifier so it's saying this person is 86% chance of having low income and because we've added an explainer to this model it can basically try to investigate that model and understand why so this is using a technique called anchors so we've got the core demographic features here and basically this is showing that this model is quite focused on two it's saying mountain status separated and sex equals female so it's saying 95% of the time if you just had those two features the model would low income so obviously this is key because this is saying what is this acceptable to put this model out because it looks like it's quite biased in terms of gender and so it may be that you this is completely acceptable to put this model out you need to get more or it may be that you need to get more data for various sections of your model so I mean this technique anchors which is the technique I'm showing here really gives you the core anchors that the model is using but you can set a threshold in this case it's 90% so try to look for features obviously there's a lot of features here you know there's features on what gets white colored the race and other things where they are United States their age and other things like that and it's trying to find core features that from that initial request that we're trying to explain that really made the model go in one direction and here it seems to be mostly mountain status but because you set the confidence to be at least 90% of the time it should be true this answer this explanation then it tried it went further and tried to find one more feature which in this case was the gender feature so is the underlying model here is it you know random forest is it linear regression how do we see this yeah this model is actually just a very simple I think it's some linear regression and this technique is black box so it doesn't actually as I say depend on the actual underlying model but it's very simple model just illustrates that it's quite biased and actually this data set from the sense of state is actually very so unbalanced there's actually only one percent of people who have high income so most people have low income in the data center it produces very biased values this is where we're using it as an illustration of things you need to do to of how you can use explanations and it can help you understand your model more and then obviously get early indicators of some of these things okay so like if you have linear regression and you are asking for a year that's outside the model it might tell you look this is outside the range and so for the linear regression is invalid or something like that um yeah I suppose that's I mean that's slightly related actually this technique is actually needs to have access to the training data so it can understand the range of I think this is the I'm not sure which one is the age age thing but it needs to understand that if it's say in what it was doing it would have probably sent several hundred requests to the model it would probably change this feature a little bit it's 65 you know 55 and stuff so it needs to know that there's no point doing 600 because that's going to give you know strange results won't help the explanation so it it sort of understands those aspects of the actual training data and then it fires off lots of requests to get this answer okay so it's not really rule-based stuff uh-oh you're doing linear regression in the wrong way or something it's not going to yeah no it's not like that it's really just as I say treating the model as a black box and trying to understand but by um understanding how the model is behaving just around this particular value so it's changing features and seeing if the model changes its opinion and um what are the core things that make it focus on that result very cool yeah yeah yeah it's it's cool sort of techniques and so here down here we just give access to the actual training data so you can see examples which actually fit that explanation and then there should also be some examples which don't fit that so that these are people who have those features separated and female but have high income as you can see okay maybe I need to expand out my training set by getting more of these you know to sort of label these and stuff and you know it just helps the data scientists this technique is obviously as I said you need to be very understanding of the actual stakeholder that needs to look at this if you gave this to a customer I think would probably be very confusing to them so this is a technique that's probably more focused on the data scientists or maybe the auditor as opposed to some of the other techniques yeah just one comment that I think this is a great feature I mean some of the the things that I've been hearing about some of the models is is that they're biased and that's this is a great way to see that you know they're biased and then maybe telling some of the data scientists that they need to change their models right or they need to in certain ways so they are very less biased things yeah absolutely so it can be used at various different stages of the machine life cycle so it can be used in sort of development and sort of so auditing but also obviously in production as well that you know you might get requests maybe this is gone live let's not say hey well was I you know rejected you know for a loan and then we say oh look this is these are the reasons the models look here you know and so you know you can get early get early warning of that I'll just show you one more this is like a different model this is a sort of text-based explanations this is a model that takes a movie reviews and tries to decide whether it's a positive one or a negative sentiment so hopefully I can find an example and I'll predict and I'll explain that example and we can have a look at so this is a movie review so visually exquisite but narratively opaque and emotionally vapid experience of style and mystification so obviously that's some negative and so what this explanation here in the text thing is saying what are the words that the model was looking at to really make this sort of negative and obviously it's highlighting these two emotionally vapid as the two core things so it sort of shows like another angle that you need to have different explanation techniques for different types of data and and how those work for different types of what should be different so you know so obviously this is different you know to the tabular case and it's sort of highlighting things in like a different way but it's actually using the same sort of technique but for text-based data rather than sort of tabular this is sort of anchor text because it's called an anchor tabular technique cool so yeah so that's alibi explain and then we also have alibi the tech which is looking at so more the monitoring side so looking at outliers obviously because you don't want to actually give responses from your model if it's an actual outlier because you know it's quite likely the model is going to give you the very strange results if you have an outlier you know you you should probably throw that result away from the model then other things like adversarial attack detectors and there's also an area that we do research on obviously it's quite niche there's you know obviously particular areas that it's important for and this is an example taken you know from sort of a traffic sign detector so it's just a classic one you know where you've got a stop sign here and the model saying stop great but if you just attack it in certain ways by adding a few pixels it's still pretty much looks like a stop sign to us but actually the model gives a completely different answer and the same for these apotrophic signs always wondered whether stop sign was blue i think it's slightly taken from a german data search i don't know maybe stop signs are blue in germany i'm not sure um see is there a new stop they're in germany yeah that's true yeah so so that's um we do that we also look at things like drift detectors which tell you when do i need to retrain my model so it's the input distribution the model's saying completely different to what it was trained on because again you it's probably going to be giving bad answers you probably need to retrain it on on the new type of data and it's the same sort of thing so this is the alibi detector github repo and again we there's a suite of different techniques again based on some of the different um sort of um modalities and and sort of decision points you know is it is your data tabular image time series text it doesn't have categorical features you know do you want to do online outlets section do you want to have outlines at the particular feature level you know so for like a sort of image the feature level would be sort of the level of pixels rather than the whole image as you need to make decisions based on that and similar for the other sort of things abasile detection and uh a sort of drift detection that's obviously the key is that we we do this data science then we bring it back into seldom core so you can then deploy your model using seldom core and then you can add these explainers etc um explainers outlasts textures to your model to give the things surrounding it and that's obviously the goal to allow allow these organizations to do that um so cool so i'll go on to another project we welcome which is care serving so this is like focused on using some of the things from say k-native so we're focusing on scale to zero because obviously you know things like uh GPUs and other aspects of machine learning inference are quite costly so if you if it's if you got the model it's not being used wouldn't it be great just to get rid of the infrastructure from your Kubernetes cluster so i mean it's really great stuff they're doing k-native and so this is building on top of that everything is we're looking at gpu autoscaling in this project because actually gpu autoscaling is is quite a challenge i think i'd talk about it in the next slide just how that how k-native solves that and also what we're trying to do we actually founded this project with some very large partners google bloomberg microsoft and ibm and we're trying to really create some standards for machine learning inference and feed them across the whole industry so one of the things we've done is is created like a standard protocol for machine learning inference and we're starting to get people from these organizations to actually buy in to actually using it so obviously that takes time as all standards do but it's it's an interesting direction that's part of what the project was created for this project is part of q-flow so q-flow is actually you've heard of it it's like a sort of ecosystem of machine learning projects right from sort of data analysis training at scale and some hyper parameter optimization and serving and so this part of that so it's like a great sort of location to join together with these other projects in this ecosystem to work on machine learning on top of kubernetes to just focus in one thing in care serving that we solve so one thing that's really difficult is gpu autoscaling why it's difficult because when you've got gpu so models using gpu you've got various metrics you'll have metrics from the actual server using the cpu and you'll have metrics from the actual gpu itself and also those gpu metrics are sometimes hard to get from your kubernetes cluster and it's also then harder to combine all those into a single rule if you want to do autoscaling you know if you want to say okay my cpu or my server gets over this amount and my so gpu stats say this then I scale up and it's actually quite hard for people to do so I'm so luckily we can simplify that by using the ideas from k native which just are going to basically look at the number of in-flight requests going to your actual pods in this case machine learning obviously k native is quite generic so all you have to do is say what for these pods what is the amount of concurrent requests they can manage maybe this model server can handle 10 requests or maybe it's just one and so basically k native takes that into account it actually uses various sidecars and stuff it puts in like q proxies to understand how many requests are in flight and then from that it can take those stats from that and how many requests are coming in to actually decide should I scale up or should I scale down and so it makes it much easier for the actual user in terms of machine learning who sits at the top who just wants to have their thing scale automatically not really to do much all they need to do is say how many requests can I serve my actual model server serve at the same time and it really solves that obviously there's never great things in k native they if like there's a burst of requests then they get stored in a component called the activator before they get pushed on to the actual replicas when they've scaled up so you don't get like too many requests hitting your model at the same time so it's really interesting and we're trying to build on top of that technology for machine learning interest one question here is so when it comes to handling many many requests we're talking about many millions of requests so so the standard practice is to keep share storage you know for these for these spots or and so they can they can actually do the serving from that storage I mean if you if you have a really big model because spinning up parts for every single request and then creating us you know models storage for the model you know every for every single part then we take a lot of time right so yeah absolutely and that's a very good point you've actually I don't think I have it on these slides but I gave a talk at the ICML on just on kf serving and so one of the slides was the challenges presently with with using it and that is exactly what you said the challenges are that once you scale up you've got that big lag time so you've got all your requests waiting for this replica to start and so yeah I mean that is a clear challenge and actually even some people if so at the moment the way to solve that is is to where there's nobody many ways at present there off to solve that but one is to have your model locally so it doesn't have to be downloaded so you should get rid of some of the network time but even then you're still going to have time for the model server to start up or the other thing people do is just to get rid of the scale to zero and just say okay I want to have at least a certain number of replicas but then that's obviously getting rid of the whole point of k native so yeah it's definitely an open challenge and that is something we're actually looking at as part of kf serving or so with the k native community you know how can that be solved yeah got it thank you yes you put your your your finger on the one of the key points uh cool um so yeah so I've sort of showed you some deploy which is our sort of closed source architecture and that's really bringing everything together so it's it's it's it's allowing you to use core and kf serving and I love I explain the detect and tying it all up with standard components things like so things like githops which were like a big believer in which if you don't know absolutely you do that everything gets stored onto source control before it's it's put into the cluster so you've got your sort of declarative representations you know as you define a model in deploy it's pushed out to get up a bit bucket comes back on using things like argocd um which really allows you to give that full audit trail stuff like that and we tie it together with um you know metrics and elastic stack and and off levels with decks and key cloaks tying to L there where you've got enterprise api so that's how we tie everything together for like enterprise customers and obviously that's quite key so like you know in terms of um some of the models I've got like a model running here um where you get all the stats but the the key thing is just to highlight the githops you've got that full um you can go to see what all the actual actions you've done on the model because it's all stored in in github every everything you did see all the canaries you created or have a waste updated and you can see what's the difference between that and any sort of documentation and obviously if you feel there's a mistake you can go back to a particular point in this little chain of of your github repo and just restore to that state if you wish so it really allows to do that so yeah that's what we're really doing um cool so that's the enterprise product and that's pretty much my last slide I just wanted to finish on some things for the future and things which may interest you I'm not sure but um so some things which have come yeah last slide sorry the one that had argo yep someone is already using tecton you know instead of argo can they should they just you and they wanted to use seldan would they have argo and seldan on their system or like can you replace argo in that picture with tecton yeah this is um so actually something slightly different this is I'm like another project in like the argo ecosystem called argo cd so it's it's all about sort of um so githubs and sort of syncing things from a sort of source control onto the cluster okay okay now this is my point I got it yeah yeah thank you it's but it is actually confusing because like actually when I created this slide I couldn't find there wasn't any logo for argo cd so I just stole the actual argo logos this is why like exactly what you said happens people think it's just argo yeah so I think the argo cd people need to get a logo um so yeah some sort of final um points so some things we're looking at so so GPU sharing is comes up quite a lot in terms of you know because you've got these very costly um GPUs and you might have say with some of our partners like um some bloomberg and kf server you've got thousands of models which are say very slightly different and then all skykit learn model sale or something and and what they need to do is really share on a single server all those models to sort of decrease cost and also some of them might not be used be used very much so that's a definitely a challenge and actually part of the kf server project we're looking at um extension of that project to do so multi-model serving is what we call it to allow you to have an multiple models on a server and sort of pack them in um it's one server um and obviously there's other things we're looking at like volcano which has a GPU scheduler to allow you to share GPUs and stuff so it's very early stages and I think the actual volcano GPU schedule is also very much in alpha there's interesting stuff there that are coming right from our customers that we need to look at um stuff like edge i'm sorry that's like a company and also the code that you've seen we're not really sort of edge focused at the moment but that's definitely a air which which obviously we all know is growing and um it'd be great to get your feedback of what you're seeing from other people who have probably presented because i saw you had some cube edge presenting some weeks ago so it'd be interesting to get your feedback on that point actually rather than me i think we've we've certainly seen customers um in that area definitely um and so general model optimization system we're looking at into the future so you know so customers would sometimes want to just give us the model it could be just a TensorFlow model but then we can do various optimizations optimize it for various endpoints it could be for edge you know sort of and or have a way to optimizing the model you know perhaps take a skykit layer model and turn it into one that can be run on a gpu and stuff and so there's lots of interesting stuff there that we're looking at and then just one the final thing is to shout out about care serving and sort of general machine learning data plane we're doing so as you said we've created a general um machine learning protocol uh which was and is then going to be supported by various people in the industries it's supported by nvidia triton infant server present self and self care serving and and then hopefully in the near future torch server facebook doing some work there so that's an interesting development of cool so hopefully i haven't taken up too much time it's probably longer than i thought but they're happy to open up a discussion what any points thank you for the presentation it's it's very interesting space and yeah and you know i think a lot of a lot of people are moving more towards um you know having more of a cicd type of system where you can have like you know have the data scientists create some of these models and uh have some of the the cluster operators so the DevOps people in an organization handle some of the serving parts so so i think this kind of fits in in there to to fill that gap yeah absolutely yeah thanks yeah i think it there's quite a big need definitely it's hopefully like the slides at the start showed and so yeah definitely so what question that that i have that i is a seldom team um looking at maybe having one of the projects join the cncf for i'll be in part of the cncf uh yeah i'm certainly we are um thinking about it in terms of several projects but we can't say which but yeah we're sounding in talks with cncf and the guys at lfa i so yeah i think that it's probably a direction we'll probably think about for some of the projects definitely great anybody has any questions that don't want to be the only one to ask most of the questions i had a question about the kf serving it's part of kuflo yeah for other uh components that are also part of kuflo or is that that is i somehow had the idea that all of seldom was in kuflo but i guess that's not that's or all the open source parts i guess that's not right yeah yeah so seldom is like separate from kuflo but we have integrations in kuflo so if people want to use seldom core in kuflo they can and also the the project kf serving is actually in kuflo so yeah so it's a bit confusing so uh so kf serving is part of the kuflo so domain and that project's inside kuflo for serving so it's all both for the two projects one is outside but you can use it and one's inside being developed on there so if um seldom became part of cncf then kuflo might do that separately and they just interact together that would be the way to do that um yeah i suppose it's quite separate i suppose kuflo because it's so under so google i think probably it's up to them i think there's actually an open discussion in in kuflo of how the individual projects whether what the governance would be whether individual projects could then move into cncf or lfai or how that would but i think google is interested in keeping it together at the moment okay thanks yeah i think uh with cube edge there's also a good fit there you can maybe cube edge can be used to you know manage some of these workloads and then you know seldom can be used maybe to send the these workloads or the or the models and the serving mechanism over to the edge right where maybe you want to have faster response time and i've seen some of the use cases where you know you you do some of that um inference uh on say stop sign or or maybe license plates and uh at a tall booth for example right so yeah absolutely i mean it's it's not a um but i know too much about cube edge but it's something that's part of our sort of research path to see how we can get closer to those guys and see how we can collaborate yeah any other comments questions once twice got some people on the call that are very quiet well thank you very much okay no thanks thanks sir all the insights and yeah um let's keep in touch absolutely absolutely yeah thanks for having me here appreciate it all right thank you all