All right, folks, we're gonna get started. We have a really jam-packed agenda today, so I'm gonna try and keep us on time. It never works out that way for community events, but welcome to the third London OpenShift Commons gathering, the second time we've been here at the beautiful Savoy, IET, with this very posh atmosphere. It feels, for an open-source community gathering, it's pretty upscale, I must say, and so I'm really grateful that you've all chosen to join us here today and spend a day with us hearing about all of the things that are going on across the OpenShift ecosystem. So come in, grab a seat, and I can see a few other people wandering, so there. So my name is Diane Neuler. I am eternally grateful to all of the sponsors who are here today. This is unlike other Red Hat events, it's a community event, so everything is sponsor-driven and we couldn't do it without folks like Aquasex, Six-Dig, and Prolifics and Storage, OS, Accenture, Portworx, Snicks, Crunchy Data, UK Cloud, IBM, Z or Z, depending on where you're from, and Couchbase, so please, during the breaks, take a wander around, have a cup of coffee, chat with them, we are incredibly grateful for the work they do within the ecosystem, a lot of them have built operators which you're gonna hear about today, a lot of them have open-source versions of their products. It's a wonderful, very big and robust ecosystem and these guys are some of our main supporters, so we're very, very grateful for that. As I said, we have a really packed agenda.  So that's gonna be a nice, long-packed day. We promise coffee breaks and food up again where you registered on the third floor, so don't rush the elevators, but there will be food and plenty of time to network today. So I like to start off all of the Commons gatherings with really trying to help people understand what OpenShift Commons is all about. And I've done this talk a few times about, as I call it, the search for connections across the OpenShift ecosystem. And often when you're working with a vendor or an open-source project, you're doing it remotely or you're doing it in your cubicle and you're working with lots of people who they, you never see their faces, so a few people here I've seen for the first time. Wayne there, who I just met for the first time has the same name as the mayor in my town and now I've actually met him and I know in LinkedIn he's not my mayor. So there's lots of great ways to connect and I think really having these smaller gatherings, 300 or so people where you can spend some time on the coffee breaks, really meeting and connecting with people is probably the most wonderful thing you can do in order to move forward your adventure with OpenShift and delivering your infrastructure. So today is really about facial recognition. I know you didn't come to an ML or an AI event but my goal for all of you today is to meet people you don't already know. So I'm gonna ask you all to make sure that maybe you move around your chairs during the break, introduce yourself to someone else who's not from your company because we had over 180 different companies not including Red Hat and IBM now, people. So there are a lot of a good mix of people from every kind of part of the market sectors from banking to Edge to telcos to all kinds of fun folks. And we had 500 individuals register for today. So I know it was a free event. So there's a little attrition but you'll all come and at least eat my food and hang out and really try and take advantage of what the commons is because it's really kind of a new community model. If you work in open source for as long as I have, originally it was really about trying to get people to contribute to my project whether it was at CD or OpenShift Origin when it was called three or four years ago. And now it's become much more of an interconnected world where when you see the CNCF and all the projects that are upstream and downstream to OpenShift, it's much more difficult to really get a sense of all of the intricacies of those communications. So what we've tried to do is create a new community model which is much more peer-to-peer network, networking interactions and we have events like today. We have OpenShift Commons briefings. We have a very long catalog of YouTube videos where you get to see me introducing people and getting you guys to talk about your talk, you know, whatever's near and dear to your heart. So if you want to speak, we have plenty of podiums. We have SIGs on just about every topic, number of working groups that we host under the OpenShift Commons. We have a really active Slack channel which is active, sometimes good, sometimes bad because it's like two in the morning and someone from Japan is tapping me on the shoulder. There's some mailing lists. And yes, we do look for you to contribute code but we'll talk a little bit about that where the code contribution can come from. But really my goal for community development is about helping create connected communities whether it's the OpenShift OKD one or it is EtsyD or it's Operators or it's Quay or it's Kubernetes or any of the CNCF projects or any of the sponsors who have operators and things like Falco and other projects. So just trying to keep all of those projects across that so that we can sync up our release schedules, feature requests, and people know each other. So that's really the goal of Commons. It's really about creating those connections. I've done a little bit of research on GitHub contributions and you'll see these and whenever I talk you can Google it and I do a lot of, I call these the jellyfish diagrams. Each one of these dots represents a person who's made some contribution or pull request or issue logged in GitHub or in one of our bugzillas for the projects. So you can see Kubernetes up there and you can see OpenShift and just some of them if I picked all of the new CNCF projects and all of the projects it would be a big blur of that. But it's the way you can start to see people who are contributing to some, come on in. Don't be afraid, there's a whole front row here.  So it's really, today is just about trying to help all of you create a lot of connections because really the way that I measure the health of our communities is about the number of connections that we create and how interconnected it is with the upstream, the downstreams and all the vendors. Because at the heart of everything is open source, the open communities and trying to make all of the collaboration that we do come out in the open. So that is what makes innovation in the open source world work. So as I said, open source is the source of everything. There's millions of projects out there in GitHub. There's lots of them. There's no way I could put them all up here because again, it would be a blur, but we have to really recognize the interconnectedness and interdependencies. So I like to say, if you don't know, the name of the open source repo and the open source project for OpenShift is OKD. And I jokingly say that means OK, Diane. But it is a function of Kubernetes plus everything else. So it really, and you'll see the pan to pop up a little bit. And OKD is really an interesting, it's in an interesting spot right now and you'll hear a talk hopefully this afternoon, mid-morning from Christian Glombeck on where we are at with releasing OKD4, but it's a collaboration with the Fedora Core West community, Ignition and a number of other projects as well to bring OpenShift onto the Fedora Core West so we'll have a pure open source play. So there's lots of ways we collaborate. But the interesting thing is, is like for all the millions of projects and all the hundreds of them that I actually know about, there's tons of undiscovered ones. And I probably overdo the jellyfish metaphor too much, but it's, there's so many different things that pop up on our radar so that it's very hard unless you have some network of peers and ways to get connected with people to really validate which ones are the ones that you should be watching, you should be incorporating. And this is really what Common Strives to do because this is last night, late night snapshot of the CNCF snapshot, trying to make decisions about which ones of these projects you wanna deploy in production or you wanna start testing or you wanna start contributing to is really hard if you don't have people to help you and give you advice and that you can trust to do that and that's what we really try and do. So if you just look at a few of the things that are in the incubated or graduated status of the cloud native computing foundation right now, just that figuring out which ones of these and how they interact with OpenShift and how you should use them is really a difficult process.  You're talking about another 104 operators plus a lot of them are still out in the wild getting added into them. So what we keep trying to do within the OpenShift ecosystem is build strategies so that you can figure out how to vet and validate and use and find and discover some of these new projects. So with the operator hub.io is the open source site of things, there's another one that's built in and baked in and you'll hear about that today quite a bit, there's gonna be a nice talk on operators and some demos. We have the author of the recently published Kubernetes operators book. Jay Dobie is one of my colleagues is here. Did you get the published books? Did they show up? Are you gonna ask me that at the coffee break?  Yeah, and we haven't seen a hard copy of it yet. So that's coming. So the thing about jellyfish is really sometimes you have to really dive into some of these projects. So we look at trying to figure out the interconnectedness of this and some of the research I've done has really been how I as the head of community development and the person who tries to find some of these people. So these three little dots at the top are three little, not three, but three folks who have been contributing to both of these projects and some of them are, so these folks, some of these folks will be here today talking to you and some of these folks will be online. Greg Swift is a great example of this. He's been contributing a bit to Jaeger, a bit to Kubernetes and OpenShift and he was working at Rackspace and is now at LogDNA.  And so what I'm hoping today you'll do is you'll find some of those people, whether they're Red Hatters, where we overlap with IBM folks or there are other people maybe from Amadeus or Santana or Barclays or Deutsche Bank, some of the people you're gonna hear talking about this stuff today. They're here in the room, they're here to connect with you, they're here to share their stories and their best practices. You can ask them questions. If they're here and on stage, they're totally willing to talk to you and connect with you. So take advantage of that. And so you guys are the audience and you're maybe looking in and kind of looking at these strange jellyfish who are gonna come on the stage, but I promise you they don't sting and they won't entrap you. I might entrap you and try and get you to come and share your story at the next London event or one of the any of the upcoming ones will be an Amsterdam at KubeCon with another gathering, will be at Red Hat Summit in San Francisco, any of those events. If you wanna share your story or if you have an open-source project that's part of this ecosystem that you wanna talk about, please come find me on the break. I am happy to stop talking and to give away the podium today and any day. So I'm doing pretty good keeping us on time here. So I'm gonna try and make sure that each of the speakers is available during the breaks. Everyone's committed to staying here who's speaking the whole day. So you can track them down and there's a reception in the evening. So which is upstairs again in the third floor and I've seen the skyline from that room upstairs. It is gorgeous here at night. So come be jellyfish with us. If you'd like to join the Commons, there are I think 540 organizations. It's a member-based organization community. So you only have to join once and then anybody at your company can join. So just sign the paper, sign away and we'll get you into the Slack channel, onto the mailing list. We'll make you volunteer for SIGs and tell your story and we'll get you connected to the people and the projects and the engineers and the other contributors across the ecosystem and help you on your journey through the OpenShift world. Again, one more shout out to all of our sponsors who are most of them are probably upstairs, probably frantically still trying to get their booth together. Please do spend some time with them. They really, we couldn't do this without them. So I'm gonna thank you for your time.


Hello So my name is Julian. I'm the CEO and founder of we Fox and I'm a serial entrepreneur So I've found it around eight companies in my life so far and today I want to talk about my biggest single biggest Learning I've made over the last few years So of course what you always see in the media is how well businesses are doing Currently we have a couple of scratches, but that's generally the picture that is portrayed, right? So we have big companies, right our company for example in Since 2015 more than 600 million in revenues We have been able to grow to more than 1300 employees We are now the number one inter tech in Europe when it comes to size Valuation wise we are actually the number one inter tech in the world. So the media celebrates this We have been this year for example The number one in terms of how much has been raised in terms of inter tech globally and you see by far Right and these are the type of messages you would see in the media when you look from the outside in But today I want to be brutally honest to you and I want to share my most important learning This most important learning can be summarized like this No matter how glittery it seems No matter how straight the path towards success seems The success of a company is Limited by the personal limitations of its CEO The company is like a plant and The plan wants to grow The plan wants to become a big tree and The CEO is like a cage Around the plant Keeping the company from growing where it could and And the single most important job of a CEO is To increase the size of the cage So I want to tell you a little story about how I came to realize this So when we started in 2015 I Was not an insurance guy. I was in technology and I swore I never want to become an insurance guy You know why because my father is an insurance guy So I wanted nothing to do with insurance in The end we saw the potential we saw that someone has to do it So we started we Fox in 2015 and I called up my father and I was like that. Can you please help me? Can you come and work for me? So he started working for me. I was his boss That's a weird story and I could talk about it and my learnings for a long time but The most relevant piece of experience that happened was one employee one day coming to me and we were already 150 people or so and that person said Julian So now you've got your dad here and you're running the company and I can tell you one thing The family dynamic between you and your dad Can be felt to the last part of the company and you and your dad You are taking the company hostage By trying to solve your interpersonal issues inside of this company and Then it was like boom. I realized what I As a CEO have as a responsibility and what is the impact of me? my personal limitations my biases on the growth of my organization and And I realized that What I have to do is I have to work hard on myself To become the best possible CEO that this company deserves and this means I have to go where it hurts and The first thing that I did was fire my dad and Since then this is my credo Because we all know this This is the process of transformation and as a CEO, but also as a team leader If you don't become an expert at this process and Go through this process as a part of your daily routine You're not going to be the best possible version. You're going to keep your company your team hostage So we all know how this works We are in our comfort zone Everything kind of seems like it works. Well We go to work We feel comfortable with the people we meet we will feel comfortable with the tasks we do We will comfortable with the conversations we have we go to dinner with friends We go to bed and everything kind of works But you all know what I'm talking about There are these cognitive dissonances The things that don't really make sense the things that kind of make us feel uncomfortable and Our normal reaction is we push them away We try to ignore them and It's a sad story But most people live their entire life by trying to push away What makes them feel uncomfortable and try to stay in their comfort zone their entire life until they die and look Back at their life and regret it so what You have to do as a CEO as a good CEO It's you have to go and look For what makes you feel uncomfortable What doesn't make sense? Instead of pushing it away You have to go and approach it and what happens then is not nice It's actually the hardest thing for us humans we become fearful Because things That have always made sense start not making sense anymore Especially when you're a CEO by the way because something Must have worked well in order for you to be so successful in order for you to build up a billion-dollar business In order for you to raise one point three billion dollars in order for you to build a company with 1,300 people something must have worked well you must have found that success formula so now Challenging yourself and actually Thinking about Whether the way you have acted is Actually the way you should act going forward and whether certain behaviors patterns of yours that have made you successful You might have to leave behind you is a damn scary thing and Actually when I sit with my management That's what I request of them. I tell them people We are here because we have acted the way we have and Each one is here because of your unique skills But where we're going is A place where none of us has been and If we stay the way we are we're never going to get to where we want to be So what I ask of you is Question everything that you consider your strengths that you consider your identity and Jump with me into a dark cold water It's like a caterpillar a caterpillar that becomes a beautiful butterfly But the caterpillar goes through a lot of pain I don't know if you know that just before the caterpillar becomes a butterfly It actually looks like the caterpillar is dead For two three days. It's not moving All the energy is out Everything is gone. It looks like this is lifeless and this is what happens in the process of human transformation as well You will go through a process of confusion You will go through a process of identity crisis. You will go through a process of depression There is no way around it but then You start figuring out That there's something good in pain and Then you realize that you have learned your lesson and Then you start understanding that this is something that changes your life That changes the way you treat other people that changes the quality of your family life and that changes your qualities as a manager or as a CEO and then You recognize That you never ever want to be your old self That this old self that you thought was so successful Was actually so damn limiting and You are thankful for having gone through this journey and the difference between great CEOs and bad CEOs is That good CEOs Look for what makes them feel uncomfortable and they don't just wait until it's too late Because we all know that if we wait Life is going to put us into a position where we have to go through it. Anyways, it might be us losing our job It might be a dear friend or family member dying Life will push us into these moments So what I can recommend to all of you is go and look For what makes you feel uncomfortable go and look For things that you believe make you weak and vulnerable and go all in and make that your daily routine and Manage that process yourself become strong enough and find joy in pain Because without pain. There's no evolution And without going through this process You will not become a great CEO There are CEOs that have a huge amount of talent. I can tell you I'm not one of them These CEOs they are born with a huge cage So they can grow big plans without doing this work But at one point They will have to do this work Either they want to or they have to So I believe in the unlimited potential of every single human being I Believe everyone can be a CEO and everyone can build billion-dollar businesses and everyone can be the best possible version of a CEO to their organization and unleash their potential But you have to be willing To go through the process and there's no shortcut. So thank you very much


Thank you. Hello, everyone. Hi. So you probably have seen that there was two names at the beginning on this talk. And you probably have counted that there is only one body right now on the stage. So yeah, there was some last-minute changes. And I will be presenting myself. Let me first quickly tell you more about what I'm doing in Google. So yeah, my full name is Vatislav. Just don't try to pronounce it, Slavo is good enough. In the Google, I'm leading a small team that's trying to optimize TensorFlow in other framework for the hardware that we have on Google Compute Engine. So basically, if you're spinning up a VM on the Google Compute Engine and you're choosing special deep learning OS that we have, you probably will be using our custom version of TensorFlow precisely optimized for the GPUs and CPUs that we have on the platform. Before we're going to proceed to the talk, quick question. How many of you guys actually using TensorFlow? OK, it's impressive. How many of you guys using PyTorch? So why have you come to this talk? But anyway. OK, so let me first say kudos for authors of this presentation who are not here. Several of them are from the TensorFlow team. Several of them are from my team. Huge thanks to all of them, because it wouldn't be possible without them. Now let's jump to the agenda. As you might have guessed, we split the talk on two parts. First, we're going to cover the theory of distributed training. Oh, yep. There was any questions? OK. First, we're going to cover the theory of distributed training. And inside of the theory, I want to answer two simple questions. How you can do a distributed training, which type of distributed training you need to pick, and why? Then we will see in the practice how you can actually apply all of them with the TensorFlow. And what is more important, we will see how in the practice you can take some legacy model that was implemented five years ago, let's say, or three, or some old, old model, and use the modern practices of doing distributed training. So let's first start with theory.  These are official icons of Google Cloud. First of all, one is Virtual Machine on Google Compute Engine. The second one means that VM that I'm showing is high CPU instance without the GPU. And the third icon is GPU. I know that GPU icon is kind of look the same as VM and GCE, but these are official icons, so just bear with me. OK, so let's start with asking question why to distribute. If you want to do a distribution, you probably need to ask three questions. Question number one, if you have a model of hands, how fast do you need to train this model? So basically, how fast do you want to experiment? If you're trying one model per day, maybe training one day is fine for you. Maybe you need to train one hour or two hours or maybe one week. As soon as you have a realization how much time you need to spend on training the model, you need to answer the question, what the level of the accuracy that is good enough for your business needs. As soon as you have these two questions answered, that you, let's say, absolutely fine training is between two hours. And the level of the accuracy, let's say, 89%. The third question is, what are the cheapest way to do the training to satisfy these business requirements? And with distributed training, I basically saying that the rule of thumb is the same as with concurrency programming. I used to tell to my students, if you're thinking to do concurrent, and if you can avoid it in any possible way, please don't do concurrent programming. The same is here. If you can avoid distributed training in any possible way, please do avoid it. But OK, let's now speak more precisely why you might want not to do distributed, and then we'll speak how to distribute it if you're still willing to do so. So this is a chart that I'm going to show just to illustrate some compromise that you will have to have when you think about distributing. On the x-axis, I have the configuration of different type of instances. And I'm going to show you the speed of the training of a RESTnet with synthetical data on different type of the instances. So y-axis is actually the speed of training. It's not going to be up to scale. OK, the first instance is 96 CPU Skylake. If you're going to use special deep learning VM that basically we have, that have a special TensorFlow optimized for the Skylake, you might have 200 images per second. This is like an ideal case. Let's say that this is not what you want. You need to go faster. So you might actually spin up the instance with one volt GPU, and you will have around 800 images per second. Again, this is ideal case with TensorFlow 111. We recently have released 112, but these numbers will be approximately the same. These two numbers, what will we be calling baseline? Effectively, we're calling it baseline because in the second instance, we're using only one GPU. Let's say that even this is not enough. You need to go faster. Then you spin up a GPU machine. And on the right-hand side, these two configurations, what I like to call it is FattyTalesOn. And why FattyTalesOn? Because if you're spinning up a machine with 8 GPU, you're probably expecting the speed of training would be 720 multiply 8, which is roughly 6,000. This is reasonable to expect. You now have 8 GPU. There is no network with all of them and within one box. This is reasonable expectation. Even more, let's say even this is not enough, so now I want to spin two instances. Again, I probably would expect for 6,000 multiplying two, something around 12,000 images per second. This is my expectation. And again, this is FattyTalesOn. In reality, of course, you're paying the prices of the overhead. If you're using 8 GPU, you probably will get something roughly like 5,000 images. Again, this is close to EDL case. You might get even worse situation. So you're paying, effectively, 10% of the overhead just to go multi-GPU. This 10%, it's roughly, I'm trying to recall the actual price, might give you around $1 per hour that you're just paying for this overhead. Things getting even worse if you're going with network. With network, effectively, you're paying almost 25% of the overhead just to go in the distributed mode. Reasonable question that the business might ask, does it actually make sense to go to distribution mode? Here is another chart that basically shows you how much speed up you gain and how much more cost you pay. So effectively, you can see it's still better. It's still worse the money to continue growing the speed if you do need to speed up. At some point, this both line will converge. And the price will probably will go higher than the speed up that you're gaining from the training. But if you're still willing to wait, let's say you don't want to spin up two or three machine, you actually want to use one machine but to wait more, price will be actually lower. Because you no longer get paying overhead of the network. And again, if you can avoid doing distributed, you should avoid doing distributed. To answer the question whether this difference in the price that you pay is worse the time that you're getting, it's up to you guys and the business to decide. OK, so now let's jump to question which type of distribution exists in theory. And we will start with multi-GPU distribution. Effectively, how you can distribute the training if you have one physical instance and you have many GPUs in one physical machine? Let's say this is your VM. You spin up a brand new VM. You have 8V108V and you want to train your model. Let's say this is your model. I know it's not practical. Not many people in real life have the model with five elements in your computational graph. But for the sake of the example, let's say this is your model and you actually want to train this small model on V100. OK, as long as you have money, you can do whatever you want. One of the type and of the way how you can do distributed training for this super crazy graph is to actually copy-paste this graph to all the V100. Each of the V100 will be training the same graph. It will be executing forward pass, backward pass, applying the gradient and doing it again and again. Now, the question arise if all of these V100s doing the training on the same computational graph, how are you going to actually synchronize a graph between each other? One way of doing so is the called all reduce, the special algorithm that basically going to synchronize elements of the graph between all the GPUs on your physical instance. And once the most famous all reduce algorithm is ring all reduce. Ring all reduce basically says looks something like this. And this is slide from the technical blog of Uber. The Uber have done a drastic work to implement the Horowitz high level framework that actually based on the ring all reduce and heavily speeds up the distributed training of the distributed training on the floor. We will speak up about later in the practice phase. But the whole idea of the ring all reduce is that each worker could be one GPU or one instance in your cluster, sends a part of the gradient to the next worker, and at the same time receives part of the gradient from the previous worker. Effectively, this means that each worker connects to next neighbor and previous neighbor and forms something that looks like a ring or so-called whatever you like more. The good thing about this part that each worker utilizing both channels, download and upload at the same time. So you kind of can squeeze all the possible performance from the communication channel in between your instances. As basically you might guess, you need to build communication between GPUs something like this. Issues of the GPU and the instance should have precisely two connections to the next neighbor and previous neighbor in order to give you the high speed ring all reduce typology for your synchronization with ring all reduce. Now, if you spin up deep learning VM on Google Compute Engine and you will run NVIDIA SMI Topo dash matrix, this is basically special NVIDIA tool that gives you all the connectivity between GPU that you have on the hardware. This is something that you can just run this CLI, get this connectivity. And NVIDIA V100 has six NV links. NV link is a special technology from NVIDIA that allows to connect to GPU between each other, creating the high speed connections. In this particular diagram and this matrix that you see, important part of this part. You can see that each GPU basically has three NV link connection to one GPU and three NV link connection to another GPU. So each of the GPU precisely connected with two neighbors allocating all the connection speed to previous and the next neighbor. Actually, if we're going to graph exactly this particular connectivity to our previous picture, you probably will get something like this. This is six NV link that design on the hardware level to speed up wrinkle reduce in the best possible way. So this is one way how, in theory, you can do a distribution. You can copy the graph on all the GPUs. We will see how to do it in practice and then apply some level of synchronization. The second way, which actually came due to historical reasons, still sits around, called model distribution. The previous one called data distribution. Idea behind the model distribution that, again, you have the same graph with five nodes. Again, please don't train the five nodes graph on the 100. And this time, you're distributing the graph. You're no longer copying the whole graph on each particular GPU. You're copying a part of the graph. In this particular graph, for example, you have one GPU having one node, another GPU having another node, then one GPU have two nodes, and so on and so on. Effectively, this type of distribution de-mos complex one, because you, as a developer of the graph, actually need to decide how you're going to distribute graph across the GPUs. You need to take the account connectivity between GPUs, like if you have a ring topology that we saw, it's not practical to have diagonal connection like we have in this example, because there is no direct hope between them. So this is model distribution. As soon as you, as a developer of the graph, have created a distribution across the different GPUs, you might see, OK, I have eight of the V100s, and I distributed my graph across four of them. How are we going to use another four? For another four, you can actually apply now high-level data distribution. So now you have a logical graph that sits on four GPUs, and you have a copy, another logical graph that sits on four GPUs. And now you can do a data distribution. You have two copies of the graph. Both of them can train separately and use a ring to synchronize data between these two logical graphs. So these, so four, we have covered distribution within one instance. Small conclusion. So we have a data distribution. It's kind of simple to implement, because for you as a developer, usually it's absolutely transparent. You're saying, here's my graph. Please do all the magic. And usually, a framework that you're using knows how to copy the graph across all the GPUs and knows how to synchronize them. Then, usually, due to the popularity of this type, you have a hardware optimization, hardware that precisely designed to do ring call reduce. The second type is model distribution, which is actually hard to implement. Please don't do it. But due to historical reason, it's still around. And we will cover why. And then you have the most complex one, hybrid distribution, when you're kind of doing model distribution, and then you're copying this model that sits on four GPU and doing a higher-level data distribution. This one, the hardest one, but it's still used in the legacy code. OK, now, multi-node. Multi-node basically has the same idea behind the distribution. The first one is model distribution. Only in this case, you're spinning up not the many GPUs, but many instances. This type of distribution used to be around probably five years ago, ish, I think, 2012, six years ago. OK, so this is actually a picture from the disbelief white paper. Disbelief is a framework for deep learning that we used to have in Google before the TensorFlow came up. And the disbelief, by that time, was heavily utilizing CPU training. So in this particular case, we have four instances. Each has 96 CPU scores. And therefore, in order to create a huge model, you do need to have a model distribution. In this particular case, due to the nature of the model, usually, researchers were designing topology of the model, were precisely specifying how many machines need to be distributed on. And then, when you finish describing your topology of the model, you can do a more high-level data distribution when you have each four nodes is a logical copy of the graph. And now, you have four clusters. Each of the clusters doing a data distribution, they effectively training the graph on the part of the data. And then, you would have what is called parameter server. Parameter server is special designated machines, hardware machines, that is used just to save your model. Now, this is model and hybrid type of distribution. You can also do a data distribution as we discussed on the GPUs. Effectively, a year the same. Just copy a graph across all the nodes. If your nodes has enough power, you have a GPUs, and you have enough memory, just copy all of them. And now, you need to synchronize them. Synchronization, the idea is basically the same. You can do a ring. And if you're doing a ring, you need to be sure that all of your work are actually reliable, all of your worker have high connectivity bandwidth network between each other. In some cases, if you're not sure about this, you can split your ring on two. You can have two rings. Now, in this particular case, let's say, you have a fast connectivity between worker two and three. So you can create a ring for the fast synchronization of the model between them. And then, you have a green arrow that, effectively, two nodes ring. That second ring, you can start only when the first ring is over. So usually, you can technically split your cluster on subparts that has a high connectivity between each other. Now, the next one is distribution that's still around is parameter server. When, effectively, you're no longer sending the small chunk of your gradient step, you're sending the whole gradient step to the parameter server. And the parameter server is the source of truth of your model. This was one of the most widely used type of distribution in Google way back then. And there was several reasons because of this. First of all, you can utilize heavily the schema if one of the workers goes down. It's not a problem. Other worker will continue working because they no longer rely on the circle ring. They are no longer waiting until they will receive all the part of the gradient step from all the members of the rings. Therefore, you can technically use this model of distribution with a lot of frame table machines, a lot of instances that are not reliable, or if the connection is not reliable. Now, people always ask, which of these two models should they be using nowadays? And there is no silver bullet. Effectively, if you know that you have a reliable connection between instances, instances are fast. You probably should go with all reduced topology without parameter servers. That is actually why in TPU, we're using only all reduced. You cannot use parameter server with Google Cloud TPUs instances. Now, if you want to use something like preemptible instances, or you have unreliable connections, or you have slow instances, you have CPU instances, or any type of instances can go down, you probably need to use parameter server. But there is a bottom line. If you're using parameter server, often your workers might using a stale data. They might using not the latest version of the model. Therefore, you're paying time for converging. Now, these are the four distribution models across instances. You have ring, you have parameter servers, you have model distribution, you have the hybrid mode. Out of these, if we're not speaking about parameter servers at all, data distribution is the simplest one. Model distribution is a hard. Hybrid is the hardest one. Now, the question. If the hybrid distribution is the hardest one, why we still can see official TensorFlow models or other models that still using model distribution? They're still doing this, and you as a developer will probably have to work with this. Particularly, this is example from the documentation of neural machine translation model. It basically says that it's doing model distribution, and it leads to really fascinating results. You're starting your training, and you might see the GPU number one throwing out of memory exception, while GPU number two actually utilizing only 5% of memory because your model not distributed evenly. The reason can be found in the history of how the TensorFlow itself was developed. If you go back to the disbelief white paper and you look on these two pictures, you will see that model distribution is the hardest part from that point of time. So the question, why in 2012 the people were doing model distribution? The thing is in 2012 GPU were just at the beginning. A lot of people were start trying using them for the research, but they used to have small footprint of memory, just six gigabyte, and CUDA computational compatibility were not having that much of sophisticated API to implement workaround of the small footprint of memory. Speed of training were not that fast, and the several more interesting, this is actually chart from the white paper, and it's really interesting. On the white axis, you can see amount of CPU cores that they used to train a huge model. On the Y axis, you will see time to train. All of this is different CPU configuration except that purple triangle, purple triangle close to Y axis. This is actually GPU time to train. So basically in 2012, GPU was one of the slowest wave training the huge model. This is one of the conclusion that the researcher found, and I will just give you a moment to read it through, because this is so not how it today. OK, so basically by then in 2012, training on GPU was not that big of a thing as it today. Nowadays, no one using model distribution, but due to historical reasons, there are many models that you can find an example that's still doing this. And we're going to cover in the practice part how you can take model that doing this and actually use it with data distribution that way more simple. OK, yeah, this is basically what have changed in 2013 and why we now training on GPUs. So let's go to practice part. Practice part, first, good old days, how you used to do distributed training. This part of the source code from the official TensorFlow website that gives you explanation how you used to start the cluster with parameter server and a lot, a lot, a lot of source code. I basically just put it here to emphasize how horrible it was. I'm not going to even dive here what you're doing here, because you need to do a lot of stuff, and usually you would find yourself doing a lot of bugs and mistakes in starting to run this. Community have come with two simplification for doing distributed training. One was designed by Uber and has the name Horowitz.  Distributed strategy is a native concept, new concept in TensorFlow that we're going to discuss. Distributed strategy right now in the process of the development, so you can think of a distributed strategy as something that will be default distribution way in TensorFlow 2.0, but you can actually start using it right now. We'll speak why you might wait for 2.0. So let's take a small example that is not distributed, and let's speak and discuss how we can do it distributed. So here's an example, simple linear classifier. Let me walk you through all the source code that we have. So we're creating two numerical column, x1, x2. We're creating small optimizer. We're creating estimator. We have input function that just produce random values, and we finally start the training. Now let's see how you can do it in distributed way with the Horovot. So Horovot initially, when they were designing the whole idea behind the Horovot, they took, they look on the current approach of doing distribution. If you have the model that trying to utilize all the four GPUs, let's say I have an instance, you have four GPU, you have the model that utilizes all of them. Usually you would start TensorFlow, and TensorFlow somehow inside of the model knows how to split the training between four GPUs. Horovot and the people in Uber decided to train this and do it in slightly different way. They thought, okay, now you're going to spin up process for GPU. Each of the processes will have TensorFlow that will think that is the only one process on the machine, and it will think that it has only one GPU that it can utilize. So effectively you have four absolutely independent processes. Now if you have many, many processes, how are you going to start them? Usually in Horovot way you're using OpenMPI, OpenMPI to start different processes, and then Horovot under the hood going to synchronize model between all the processes of the TensorFlow. So it will be kind of transparent for the TensorFlow, and most importantly, Horovot heavily utilize NVGA-NICL and CCL for synchronizing model between instances. The real beauty of this model that technically if you want to do multi-node distribution, you don't need to change the source code.  That's it. Now coming back to example. This is original example. This is example, the same example with Horovot.  First of all, we're importing Horovot and we're calling any. This is something that just must, according to documentation. The second part, you wrap in your optimizer within Horovot optimizer. This just a wrapper. It has the same API. Your model source code should not be changed. It just should be absolutely transparent. Then this is a small if you want hack. You're saying if the rank of your particular instance is zero, then save model here, otherwise none. The reason for this, since you spin up a lot of processes, all of these processes going to train the same model. Therefore, they will have, at the end, the same model. And you don't want each of the processes trying to save it in the same folder. You technically just want only one process to save it, so you're saying, OK, if your rank is zero and the rank is unique ID of the processes, your rank is zero, then you are the one who are going to save the model. Everyone else can just ignore it. Then you're specifying config just to point your process of the TensorFlow which particular GPU it can utilize. Horovot local rank, give you the local rank on the machine. If you have four processes, rank will be from zero to three inclusively. And therefore, here, each process will utilize each own GPUs without any conflicts. And you're passing this config to your estimator. That's it. That's it. Almost, yeah, I forgot about the hook that you absolutely must apply. OK, so this is the changes. Usually, it's really small changes. You don't need to change a lot of your stuff. And this is how you're starting it. In this particular example, you're starting training with eight GPUs, eight processes, on the local machine with MPI run. And as I said, the best part and the beauty of this model that you want to do with training on many machines, you just do this. This is actually how you're starting these same source code to do all reduced distributed training between four different nodes. In this particular case, I'm going to utilize 32 GPUs. Each of the instance will have eight GPUs. That's it. So you're just scaling it to as many nodes and machine as you want. Another part that Horowitz actually gives you, it's not only the simplicity of scale horizontally, it has several more additions, really nice addition. One of them is TensorFlow Fusion. TensorFlow Fusion and the technology that allows to combine small tensors together and send over the wire only when you have the full buffer. This saves some amount of traffic. And they have a special visualizer that allows you to debug your distributed training slightly simpler. And the second part, distributed strategy. Distributed strategy is the brand new API. Right now, distributed strategy lives inside TF Contrip. And TF Contrip is something that the TensorFlow we're using, as you can think about, is an incubator. So some API never leaves Contrip and might be actually removed. Some APIs eventually gradiate it from the Contrip and move to another part of the TensorFlow when they satisfy the requirement that we want them to satisfy. In this particular example, distributed strategy is still in the Contrip. So we need to take this in account. This means that the API might change in the future. And they're going to be moved away to all the Contrips of the effectively this API will graduate with TensorFlow 2.0. So you effectively can have a sneak peek of how distribution will look in the future. Distributed strategy right now provides you four different types of synchronization. The first one, one-device strategy, the default strategy. If you just start an example, the same example that we used, it will pick one device and visualize it. That's it. Second one is the mirror strategy. Mirror strategy is effectively wrinkled reduced that we just have discussed. And let's quickly see how you can change this example, same example, to use mirror strategy. Again, same example, changes. So it's even first hard to see where exactly the changes were made because it's literally two places. One, you're creating the mirror strategy distribution instance. Two, you're passing it to your estimator. That's it. You can actually inline this. And you will have only one place where you're saying, I'm, you want to use mirror strategy. And that's it. TF effectively will, under the hood, try to do wrinkled reduced and visualize all the GPUs that it have. You no longer need to use open MPI. You can literally don't need to do a lot of other changes. Then you have collective old reduced strategy. It's the strategy for synchronizing between the instances. Under the hood, collective old reduced using JRPC. This is the biggest difference between Horovot that actually utilizing NCCL from NVIDIA and all intents of flow implementation that heavily utilize JRPC. We will look on the difference later with benchmarks. Again, same example, what need to be changed? Unfortunately, a lot. First, you need to specify config of your cluster.  This is interesting because you need to somehow figure out what the index of your current worker that you're starting. You need somehow to start worker. Then you need to actually create the instance of old reduced strategy. And the interesting part here, if you can see NUM GPUs per worker mean that all of your worker need to have the same amount of GPUs. It's actually common that your worker will be the same, but time to time, I saw that the people trying to train on strange configuration when different instances has different amount of GPUs or even type of GPUs. With this type of strategy, for now, it's not supported.  It doesn't support just train. You have to use train evaluate. So this is basically how you're doing multi GPU, or sorry, multi instance distribution with distribution strategy. Now, small conclusion. This is the comparison of the speed between different distributed strategy. Red line is a Horovot, blue line is distribution strategy. Now, as you can see, while you speak on the one instance up to eight GPUs, you probably do want to go with distributed strategy. It's slightly, slightly slower, but it's probably still within the deviation of our measurements, but it's way simpler. It's only one line that you need to change, that's it. Another important part that distribution strategy is supported by the TF team. So we can file a bug and have expectations about the time of reacting on that bug. If you're using Horovots, I don't know, I cannot speak on behalf of Uber. It's up to them to decide how soon they actually can fix the bug inside of the Horovot. If you're speaking about multi-node distribution here, for now at least, Horovot is faster. And the hypothesis why it's faster like this because they actually utilizing Nickel, while TensorFlow native distribution, utilizing GRPC under the hood, and it will take some time to TensorFlow to actually optimize GRPC communication to be on the same level as Nickel right now. So yeah, for multi-node, I would suggest to at least look on the distribution strategy because it's the future. It's what you will have to use in 2.0 because even the Horovot effectively will migrate to distribution strategy APIs. It will be just another distribution strategy. That's it. So this is comparison of different distribution strategy that you can apply. And huge thank you and questions, questions. Okay, so if no questions, you can find me outside. Thank you guys.


Hello, everybody, and welcome to another hobby-cheating video. And today, we're going to talk all about, well, 2024 plans. Let's get into it. So this is going to be potentially a slightly different video, but as 2023 has drawn to a close and 2024 is beginning, it gives us a moment to pause, reflect, and take stock. And also set some goals for the future. Now I've never been one for New Year's resolutions.  That being said, the New Year is a great time for reflection and to sit down and think about some things that we can hopefully achieve in the coming year. And I know that might sound like a resolution, but to me, the difference is it's aspirational versus punitive, I guess. I've never liked the idea that these things are so, that these New Year's resolutions become, I will never do this or I must do this. I'm instead going to say, I'm going to try to do this, and I hope to do more of this. Maybe that isn't a strong of a statement, but for me, it becomes part of a mantra of every day, because I do believe this hobby and how you paint and what you achieve is largely a matter of what you believe you could. Not what you can, but what you could. And so I'm going to put that into practice every day in 2024. So in this video, we're going to talk about some of what's happened in 2023 and how I hope to build on it in the year to come. 2023 was really, I think, a great year of growth for me. I feel better about my hobby right now, where it's at, what I've achieved than I think I have in the past any years that I remember. It's been a year where I felt like I actually was able to push myself and achieve things I wasn't able to kind of ever before. And it's ironic because this year that I feel the best about my hobby and the pieces that I've done is also the year I've performed the worst overall across the board in every miniature painting competition I've went to. I've done worse across the board in every single one since I started doing basically miniature painting competitions eight years ago or something. This is probably the worst year since that first year. And that's, I suppose, tough to reconcile that somehow I could be proud and happy of where I am and what I've achieved and yet still have that occur. And I think it's because in the end I hope what I've learned is to evaluate my own work and my pieces about how proud I am of them and how happy I am with them and what I feel they represent in my hobby journey and not what any other external person says. And I hope that the same can be true for all of you. External validation is awesome. It is truly amazing when someone else looks at your piece and says, this is really good or I like this or here's a medal, you did great. But at the same time, I think that we have to be able to validate ourselves. And I think that's been my major journey this year is when I look at my pieces and I look at where they are and I hold them up to what I painted before, I'm like, wow, I've learned so much. And of that I'm really proud and so that makes me feel good regardless of medals or awards or anything like that. So there you go. Yeah, I mean, I've had a few of the pieces scrolling here over the top as I've been talking. But really you can see how this year has been a year of experimentation with light and color and all sorts of things that I've really tried to push myself on texture and all sorts of things like that. And I want to continue doing that. So let's talk about my four hobby goals for 2024. All right, hobby goal number one is to keep pushing myself with display painting. I've gotten bit by the large figure bug and I don't plan to stop anytime soon. That does mean there will be a fair amount of display scale tutorials on this channel. But don't worry, those of you who are worried, stop worrying. Are you stopped? Have you stopped? Okay, good. I'm glad you stopped. I'll still have plenty of normal army scale stuff too. But I want to keep pushing myself. I want to keep experimenting with light and color with really painting some pieces that are a challenge to me, something that I might have looked at in the past and said, whoa, that is too much or I'm not sure I could do that and do that well. And really give it a shot and see what I can accomplish. I think this is a year where I'd like to push myself some. I want to do that because I feel like just operating outside of the edge of my comfort zone this last year has really led to growth. And that's a really important part on the hobby journey of how you grow is you don't push yourself to where you're scared, but you put yourself outside of your comfort zone just enough. And that's going to be different for everyone. But I would say it's something that I would, you know, if you're looking to grow yourself, which you don't have to be perfectly fine if you're not interested. But if you are, I think that's probably the best way to do it is on every piece, push yourself on something about that piece just outside of your comfort zone. I also want to keep experimenting with some new tools. We're going to mess with some really cool different types of tools this year and some upcoming videos you're going to see. Specifically, we're going to start messing around with some oil pencils and things like that. So look forward to that. We're also going to take a look at the Army painter, new fanatic line, the new war paints. I am very excited about those and to be sort of getting into and amongst those. So stay tuned as I continue to of course be obsessed with and try new paints, new tools, new everything, because you know I'm a fanatic for all that stuff and can't stop myself. So in this case, the goal is just, I guess, lean into what I would do anyways when it comes to trying new things. And if they work, they become part of my repertoire. And if not, throw them out. My second goal is to keep participating in competitions. I know I said that 2023 wasn't a great year for me in competitions. It was a lot of loss, a lot of loss, or not placing as high as I had previously or whatever. But that's fine. I still want to go back. I've got that fire to take my pieces to competition and see what I can accomplish. The reality is I do really enjoy painting competitions. Now I won't be participating in Golden Demon this year. My resolution from last year is that is absolutely still on the table. I have nothing prepped for it and intend not to participate and will not be participating in US Golden Demon given all of the challenges and problems I talked about with the competition last year. Now that being said, I will still be going to Gen Con's many painting competition. I will absolutely be looking forward to Nova's Capital Pallet again. The judging on that and the entries in that have only escalated in an incredible way every year. And frankly, I've got to push myself to keep the pace with just the amazing stuff that's on display there. And in 2024, I'm going to go to Monte Sansevino, already spoken to the wife and everything like that. So that's all arranged. I have a couple of friends I want to go with. But I want to head over in November to Italy and participate in this. I'm really looking forward to it. I've obviously never been. It looks like an incredible time. Obviously it's a lot of artists who I respect and admire. I love the way the competition is run as being as much of a show as it is a competition. Being as much about sharing in everybody's art as it is about walking home with medals. And I really just want to participate in that. My goal is to build a set of entries and pieces that hopefully I can take to Monte that I won't embarrass myself with. That's really the goal. But I guess we'll see. But I'm looking forward to that. It's going to be a long 11 months until that day. But my guess is it will not be enough to make me as comfortable as I want to be with all of the pieces I want to bring. My number three goal. I want a new army. I just really want to get into an army project. I know that sounds crazy given the display painting goal I just said. But the army can be something slow and over the course of the year. It doesn't have to be done fast. But it's been a little while since I've done a really cool army project that I can work on slowly and put a lot of love into. The last project I really worked on to that degree was my rat cast. And I truly did love working on my little Skaven storm cast. I loved the conversions. I loved painting them. Doing an entire army in red and steel non-metallic metal is no I suppose mean challenge. But I really do love army painting. And my hope is that somewhere in maybe the middle of the year fingers crossed maybe when we get an AOS fourth edition that Skaven end up being like redone and the launch box or something like that. Or maybe they redo the whole army like they did for necrons in eighth edition 40K. I think it's a line that I fell in love with now. 25 years ago, 19 or more than good night. Yeah, longer than that. Time goes too fast in the late 90s. We'll just say it that way when I fell in love with original Skaven. And I've always just loved to continue painting that army. I'd love to see what the sculptors can do with it, how they imagine that army in 2024 and in the mortal realms and in age of Sigmar. And I look forward to if we would get new sculpts, man, that would just become my dream project to work on for the rest of the year. And importantly, something I can share with all of you who might be interested in the same thing and also something to help me break up the display painting. For those of you who know how I like to work, I like to do a display pace, sorry, a display piece and then an army piece and a display piece and an army piece. I kind of like to alternate and it keeps everything fresh and it keeps the juices flowing and I suppose your brain from melting out of the side of your ears, which is always a nice advantage. My final goal, one, I'm guessing probably a lot of you can empathize with. And that is to play more games. So I played somewhere between 55 and 60 games of AOS this year, something like that, as well as a bunch of other ancillary games around that, including obviously play testing for the games that I write and that Adam, Uncle Adam and I put out under snarling badger. So my hope is that I'll get to play some more games this year. I've got a good buddy who just came here and move close and joined our sort of Warhammer clubs. That's another person who can come and roll some dice, looking forward to more games there. I really want to try the Old World and see what it's like. Have they, you know, I played Warhammer Fantasy for a really long time and I'm interested to see what the update is and to give it a shot. You know, I have massive Bretonian and Tomb Kings armies still on square bases, still ready to go. So they can be busted out at any moment. And I'm I'm excited to get them on the table again and see how the game plays. So I want to play lots of games in different games. I just I want more time with friends sharing that awesome communal experience that is actually playing the games. There's something that's magical about this hobby because it is both the art side that we get to push ourselves in that way to be creative. But then also we get to get together with our friends and be social and engage in this actual gameplay experience with a lot of what we do. It's a rare hobby that allows for both of those things simultaneously. And I think it's what makes miniature painting really, really special in my mind is that it has so many facets that you can choose to take part in. And honestly, I want to take part in more this year. So that's my goal is to play more games, not only of, you know, AOS, Old World, but also try some new games as well and see what it all comes down to. There you go. Those are my goals. I want to know what are your goals for 2024? Do you want to paint a new army? Are you looking to learn some new technique to paint something that's always scared you? Tell me down in the comments about your goals for 2024. I really appreciate you watching this video very much and more and deeply more and deeper than that. I appreciate you going on this journey with me. It is truly wonderful to have all of you watching week to week. All of the patrons who support this channel, I cannot thank you enough. You are the lifeblood that keeps the lights on around here. That and it's unbelievably meaningful to me. Everybody who watches, who likes, who subscribes, who comments and shares and supports the Patreon and buys the merchant does everything. I am just a guy. I really wake up every day and feel like I am the least special person in the world. And I consider myself blessed and lucky every day that I get to share my hobby in this way with all of you and go along in this journey. I am honored in a way that I really can't easily describe that I get to share this and do all of this with you every week. And as long as you'll have me, I'll keep at it for the years to come. So thank you. Hit like, subscribe, do all those things that make all those dings. If you want to support the channel, like I said, maybe you're looking to grow. Maybe your resolution is to take your next step on your hobby journey. There's a Patreon link down below focused on review and feedback. We'd love to have you as part of the community. As always, though, I thank you so much for watching this one and we'll see you next time.


All Snapchat users know that it does not auto save conversations like WhatsApp and other chat services. But are you also among those who save conversation and media on Snapchat app? If yes, then you need to be more secure of its privacy and security. So guys welcome to Sistvik, I am your host Shivani and in today's video I will tell you how to lock Snapchat app to enhance its security. So for this you need to install Locker for Snap app chat from Google Play Store. Wondering how Locker for Snap app chat works? Here is a small guide. Install a Locker for Snap app chat from Google Play Store and open it. Create a 4 digit passcode and enter it. And enter it again to confirm. Add a recovery email address and hit save. On the new popup, hit enable and give accessibility permission and that's all. This app not only boosts your app privacy but also let you lock your app apart from your device's app locker. Some of its highlights include a user friendly and interactive interface, easy setup and navigation, light application with the minimum battery usage, Locker for Snap app chat offers pin recovery option making it a better choice for the user. You can anytime recover your pin in case you forget it. You are now good to go with this smart application to have a secure chat experience. Easy wasn't it? Download it today and secure your Snapchat conversation from prying eyes using a 4 digit pin or your own fingerprint. If you like this video don't forget to like, share and subscribe to our channel and for regular updates press the bell icon.


The Fed Saharan Express kicked off with an opening ceremony in Praia Caboved. Saharan Express is one of the regional exercises for AFRICOM in which operations are centered on maritime interoperability and building partnerships. Rear Admiral Tom Wreck, director of the Maritime Partnerships for Naval Forces Europe and Africa, said this exercise is needed to help keep relationships strong. It's always important to get together with your partners and exercise at the tactical level up to the operational level. This week's event, Saharan Express will bring together nine countries, nine ships, and five maritime operations centers to test and work with the interoperability to help overcome the challenges within the region with the maritime community. Besides keeping strong ties, all nine countries are helping Caboved enhance its expertise in combating maritime issues like drug trafficking, piracy, and illegal fishes. They have a local drug problem and they need help and assistance in fighting the smugglers that bring drugs through Caboved or to Caboved, so that's their big security issue. What we can do in the U.S. is help with training and capacity building, we're very, very good at that. Saharan Express 2015 will last for eight days. Petty Officer Mandy Washington, Praia Caboved.


This is what you shall do Re-examine all you have been told in school or church or in any book And dismiss whatever insults your own soul And your very flesh shall be a great poem and have the richest fluency Not only in its words But in the silent lines of its lips and face and between the lashes of your eyes


To recap, the traditional model of how fruits and vegetables protect against cancer is that their antioxidants prevent the buildup of free radicals, also known as reactive oxygen species, ROS, which would otherwise go on to damage our cellular DNA, membranes, etc., which could lead to the transformation of healthy cells into damaged, diseased, or dying cells. But in that landmark 2003 kiwi fruit study, we learned that there's a second pathway as well. Tritoneutrients actually modulate gene expression and can increase our cellular defenses such that even if there is some damage to our DNA, our cells may recover instead of being irreparably lost. The kiwi study looked at one of those defenses, one DNA repair enzyme. But there are many, many ways our cells repair our DNA, you know, mess around when it comes to protecting our genes. So question number one, what effect does kiwi fruit consumption have on all these other defenses? And question number two, what if we branch out and test multiple fruits and veggies at the same time? You'll remember that there did not seem to be a dose response with the kiwis, right? As far as this DNA repair enzyme was concerned, you were either eating kiwis or not. It didn't really matter how many, but man cannot live on kiwis alone. What if you did a mix of fruits and veggies? Could we break through that ceiling? Now studies are expensive, particularly if the kiwi people refuse funding because you have the audacity to test other fruit. So they wanted to make this study count. So when they designed their plant portfolio, they went all out. Check it out. Green tea, rosehip juice, berries, berries, berries, berries, berries, berries, berries, pomegranate, dark blue grapes, Brussels sprouts, broccoli, red cabbage, kale, blue potatoes, dark chocolate, walnuts, rosemary, oregano. This study is making me hungry. I don't know if you noticed, but this is that same amazing research group that blessed the world with that study of thousands of different foods, so they knew what they were doing. OK, long story short, plant-based diets can prevent development of several chronic age-related diseases, blah, blah, blah. We know that. Yes, but how and what about that plateau effect? This is what they did. Kiwi groups. The antioxidants to the teeth group compared to the three-kiwi-a-day group compared to control, took blood from everyone, and then for the first time ever reported, did this microarray analysis where you can measure the effects of a plant-based diet on the expression of hundreds of different genes at a time. The first to investigate the influences of healthy diets on gene expression in whole blood. This was the study. Well, the kiwi group was able to significantly regulate not just one gene, as I showed in the 2003 study, but a total of five. Meanwhile, the berry-berry group significantly regulated five times more 25 genes. Conclusion. The observed changes in blood cell gene expression profiles suggest that the beneficial effects of a plant-based diet on human health may be mediated through optimization of defense processes.


So that the man that they claim that far is going to have the delegation they want. That's number one The second thing they ought to be concerned about is electing Ralph Yarbara and put every energy and dollar and time They can't in it not run people off from it All they're doing is making everybody not want to vote for you. I were almost including me Well, you see when I talked at Yarbara when he called me He said that Conley was deliberately trying to build up In order to deny a resolution endorsing him at the state convention resolution endorse now We ain't gonna change him one damn move. The point is if they have a general resolution endorsing everyone That's what the convention is for. I don't know whether I don't know whether they're gonna have one even endorsing me All I know is I told them this I want every delegate to come to that national convention to be a Johnson man Period We try to bump some heads together down there And I'd be glad to work on the labor end of the thing even though the UAW guys not the problem The number one that the convention pledges full and I'm a whole hearted support to you and To all the Democratic candidates who've been nominated in Texas and that that means the Yarbara people support the Conley people and vice versa I would certainly be far that I'm gonna support everybody I'm gonna support I'm gonna support everybody that's That's right, and I'm gonna support them openly and I'm gonna help raise him money And I'm gonna throw down any corporation president. I can make them happy But while I'm doing that he's gonna be off trying to start a fight And you and Dave you and Dave McDonald and And Oh a night I better just say now what we want you to do is we want that one vote in the Senate and Incidentally, we've got to have it for the cloture and now he doesn't know what he's gonna do on it And we just got a habit of allowable to lose it Hubert and I talked last night on midnight on that problem and He's going down there this Sunday and And It's a hard vote, but he can't turn back to labor and he can't come back to negris and they can't join over the Dixie crash But he hadn't He's done that all of his life, so he's he's constitutional that way. That's why I ran five time before he got elected I Do my best because he votes with me a hundred percent.  It'll be Hubert Hubert told me last night He's about them this problem that he had ever had And I had him at lunch today he and his wife for the show of Iran and had him said that they had table And I went up to him And I said now I want you to know Ralph. I'm with you money marbles and chalks and Anything anytime you want me to do call order Jenkins and I'll pick up the phone call Now you go down there and run for yourself day and night because we cannot take two Republican senators from Texas He caused the first one, you know He got in that thing and caused that now you're gonna cause another but what he wants to do here Miss Randolph they want to get out there and fight with Conway now What in the hell you want to fire the man's got 72% of the votes for I don't know Well, you see two years ago when Conway was running The State Council down there were subtly against him and I sat down with Moon and our fellas I said this doesn't make sense. You're gonna like the Republican You're gonna turn this the whole state administration over Republican Party and that doesn't make sense And so we fought to get the state convention to leave it up to each individual union We then endorse Conway Well Conway is not as liberal and progressive as I want him to be but Conway is much more a representative of the Texas sentiment than I am and And 10 times better and What we've got to do my job is to get Conway to support your arbor now You know the first thing at this election Conway called me up and said well, I've got 72% of the vote Two days later he called me up and said what have you got around the arbor down here looking the Secretary of State to office They're checking everyone in my county chairman for and I said I never heard of it Well, they said he got three men in there and they checking every certification. There's two See who my people are and what votes they got and said it looks like by God when I didn't run a man against him I didn't support McClundon and I let him peek out and just slip in But we wouldn't have anything to fight about now and he's gonna need me a lot more in November than I need him And I said well, I'll check into it and I checked into it and your arbor was down and see if he could control the convention So he's picking that fight. He can't control the convention It's gonna be it's gonna be to the one Conway So then he gonna say well, if you don't let us have our way Even though we just got a third of the votes and all low down the arbor just got 28% of We'll walk out Now Conway don't give a damn And Walter that's what's happened to us in Texas all these years shivers was a Seattle lawyer He represented a few workers at the Port Arthur. He was sympathetic to labor. He went to the Senate representing labor Conway went in there Being very sympathetic to him but when they get in some of these extreme folks start fighting with them and and They rule around them and so they drive them over and they get so extreme now Conway has a 20 degrees to the right What he wasn't he's elected? Yeah, well, I'll go to work on this thing. I'll do my best Well, your boy I forgot his name that died down there you remember Yeah, what's your name?  She's a good woman and she's for our program But she were the steel workers the all workers of the CWA Joe Burns people and then they just got damn bitter They can't be sensible Well, they If I only had 28% I'd be awfully careful to a challenge now The only thing at that leave the one job that labor's got to do in Texas is not Johnson so they all farm anyway There's enough to make rules. They all farm anyway, not the Mexicans. They all farm anyway the one job They've got to get done Just get the moderate conservative Democrats to vote for the Arbor They've got all the others they've got all the all field workers all the CWA all the steel workers all UAW Including the president But they haven't got the others that they need to win this race in November and they ought to run them off They ought to try to get them. I'd kiss their butt every minute and that's what I'm gonna do I bet I've seen 50 conservatives from that state and everyone at my say, please let your Arbor come back to help me That's right and What happens in Atlantic City is not a matter of his concern at all because I'm gonna be nominated What happens in Texas is a matter of great concern to me And I'll carry him over in Texas if he if he just won't make himself so damned obnoxious And he's a good man and he he votes right and he feels right except you've got to get him to vote for You've got to let the day McDonald and see that be here some of them know that this vote last night It looks like they've got about 24 And Hubert thinks we've got 41 that's 65 now we need 67 and we may lose one of those But we've got to get Frank Clements agreed that this fellow Walters with Kennedy would vote Far closer, but now he won't vote for him and the Arbor's doubtful. So we lost Keith offer and we lose the Arbor Why we may lose that bell and we're within one or two votes of getting it And I think if they just got to tell him that he just can't go back on the people elected He and somebody got to get me as Randolph tell him and now only one I know do that still They don't quote me on it because I don't want him to thank you, but I had that I will go 100% For your Arbor and I raise him $100,000 if I have to I'll do anything in the world To get him a little reelected and if I had any power with him I'd get him to quit fighting these other folks now I'll fight with Connelly. I've already fought with him on on the on the Arbor's First campaign, but I'll fight with him again and Connelly's gonna support the nominee and he's gonna support him And I'll try to get him to say so by resolution. I imagine Connelly thinks that What he's trying to do is hold these Dixie crats and he's conservative fellas from going over the Republican He's trying to hold them in the Democratic Party And when we don't hold them we always lose we didn't hold them in Dallas So we got out here. We didn't hold them in El Paso. So we got to go out there We got defeated and we didn't hold them in the Senate race. So we got John Tower Unless we can hold some of those conservatives if they go Republican with us while they're there beat us Right, okay, well, let me one thing. Yeah, I'm coming to that air-heart dinner That you know, yeah, and my wife has a problem that night I was just wondering whether I violate protocol if I bring my oldest daughter. Yeah, no, I'd rather I'd rather have her than you You want I've got two daughters, and I'm a daughter man You just bring her tell her that she got a special oral invitation from the president Okay, that's wonderful. Well, I got a little too. How old is she? Well, she must be 21 or 2 I've got one and I've got one that's going on be 21 in the next March Well, I'll see you. Thank you water. Say, what are you gonna want? When do you get started on your negotiation? I've made a couple passes out of emerging and reduced prices, but I haven't had much success All the way


T. F. N. N. Headline. News update. Good morning, folks. Steve Rhodes coming to you live from the shores of sunny Delray Beach, Florida. This year, a Lebanon update. We've got a sea of red out there. All the U.S. indices that we track are trading to the downside. Dow's down 67 points, a little less than two tenths percent, a quarter percent for the S&P are 13 points. NASDAQ's off 14. That's less than one tenth percent. The Russell struggling of one and a half percent. That's a 30 point move. Semi's down 49. That's one percent. Trendy's off 294 points. Nearly two percent moved to the downside. Gold's off 15 bucks. Seven tenths percent there. Quarter percent for silver or six pennies. Light's recruited us up a buck 49. Natural gas is up three pennies to the 30 year treasury. Down one and a half points. Printed out one 18 25. Let's figure out what all that means by looking at that nine panel market update chart. We begin with the E.S. mini upper left hand side. What do we know? Well, we know price is trading above the top of its daily profile. It's trading above its trend lines. It does not have a bearish reversal candle, at least yet out there. If it did, it would generate a roadsman, deminucator top. If we take a look at the spot, Volotinix, it says, Hey, be careful. You could see a further move lower as that spot. Volotinix is above its 50 day exponential moving average. The 50 day printed out at 14 20 spot Volotinix. Whoops. I'm sorry. That was not correct. The 50 day is printed out at 13 96. The spot printed out at 14 29. The NQ still trade above the top of its daily profile, and it's still trading above a little cluster of trend line support out there. It needs a bearish reversal candle to confirm a top. We take a look at the U.S. dollar index. It's consolidated with inside its daily profile. It's making its way up to resistance that resistance levels up at the 102 90 area. We get a close above that could be suggesting a move back towards its February highs. We take a look at Goldilocks got that TD nine count top price still trying to make its way back to its oscillator and change line. I don't have that print for you off the top of my head. We'll certainly take a look at that during the trader's edge show. We take a look at silver. Silver. Yes, they closed above the top of its daily profile. Second close above 24 90 today generates a profile change in trend. What I'd really be watching is the swing point that is trading into and that's a swing point from the trading session of December the 4th. As long as price can remain above 2503, it suggests to run for that high by the way that high out there is up to the 26 57 level like to be crude. Trying to break above profile resistance trying to deal with all this cluster of trend line. A resistance out there right now. It looks relatively bullish in the case of the natural gas contract. It still has its by the D point pattern. The price remains below the bottom of its daily profile. A little bit suspect in the case of 30 year Treasury looks like once go targeted slows from February 22nd. That's anywhere between 1711 1809 folks. They do for the trader's edge show if you would. But if you're starting off your day, please have a terrific one. We'll look forward to speaking to you again soon. Take care now.