What I want to do today is to talk about how actually events can be not only coordinated in time but coordinated also in space. And this will have a lot of overlap with some concepts that you have already heard from both in this lecture yesterday and something that even James mentioned today. And before we get into the specifics let me just try to make the claims that actually waves could be a much broader phenomenon in biology that we often realize. We've seen a similar movie, this is not as nice as the one that Andy was showing yesterday maybe with the two colors but what you can see here and this is taken from this paper are during somatogenesis on one of the cyclic genes that oscillates as Andy was discussing yesterday. But you can see that when you really look at it there are these clearly waves of activity that are traveling both in space and time. And this is not only specific to somatogenesis, there are many phenomena in biology that show this wave-like behavior. There is firefly synchronization in which you can really see this like that if you have a lot of fireflies in the wild and they can just synchronize with each other and they can generate this wave-like pattern. And the embryonic development is really also shows a lot of this example. Here is one in which you can have upon fertilization of a Xenopus egg and you see that there is a calcium wave that I think I should have looked at it when I start this relatively fast. And you can see where the sperment is going to be a calcium wave and actually a wave of contraction. So what is the role of this wave? And one possibility is maybe that those waves are a way to coordinate biological events over large scales. So if you want to transfer information over a tissue that are large, there are not of order of few micron diffusion. It's really not going to help you much. So maybe you need some special less mechanism. And we have already seen in Carl Philippe lecture and this is a movie from Jim Ferrell's lab in which when vertebrate embryo, this is a Xenopus egg divide under the Gokrivich division. You really get this very clear contraction of the cortex that are associated with cell division. And recently, at least more theoretically, there was a paper from Ruben Peretz-Karasko and James is one of the other in this in which they were addressing the problem of if you have a dynamical morphogen gradient and you have heard about this a lot in this course, that is talking to one of the simplest circuit that you could think could generate by stability is the so-called toggle switch. It's just if you have a double inhibition, A represses B and B represses A. You can generate essentially two stable states, one in which A is I and B is low and alternative another state in which B is I and A is low. And if this is coupled to a morphogen that is dynamic is changing, you can actually generate wave of differentiation across the tissue. And this paper also shows that the speed of the traveling wave that one guy is a function of noise and is something I will get back to. But there's nothing special about biology and about the fact that you can see waves. You can see waves also, it's just a simple chemical reaction. And one of the standard example is the Belousov-Zapotisky reaction in which if you mix Malonic acid with any bromide atoms, you get this really beautiful wave propagating.  Instead, you can really get the very beautiful dynamics. It's all transient. If I let this movie go, you'll see that this Petri dish will eventually homogenize and will all become white. But yet it displays a lot of very interesting dynamics. And you don't even need chemistry. You can just have waves in physics. And this is what happens if you put the water in your freezer and you let it cool for about you three hours. So this water is super cool. It's below zero temperature, but it's still water because there hasn't been any ice crystal forming yet. But then if you perturb it, for example, by banging on it, then ice form and it forms in a wave. You can actually do an even cooler experiment. Instead of banging on it, you can just add a very small ice crystal and you will see that it will also nucleate and initiate a wave. Sorry for the shaking. I didn't, I all got these movies from YouTube. So and again, you can see that there is a traveling wave. So can we understand where this wave come from? And of course embryonic development also shows them and I shouldn't this movie already. But what I will be talking about today, mainly in a little bit is the work that we do in our own lab in my own lab. And this is for you. So now there are there will be chemical waves that synchronize the early mitosis of Drosophila. And what you can see here is that you have waves that start at the poles and propagate. And what is also interesting about the fly embryo, and we'll get back to that is that it's actually as development progresses, this wave gets lower. And that will be one of the main factors of my talk is to show you and convince you hopefully that we understand why this way gets lower and how they are coordinated. And so that will be one last way where you actually will be able to see very clearly the slow down and now you see this very slow propagation of the wave. Okay. And this will be the last 13 division we talked about last time how they count to 13. Why do they divide 13 times? We still don't know that happens, but this will be the 13 division. So in trying to understand the wave propagates usually a very good place to start is the so-called Fischer-Kolmogorov-Petrovsky Piskanov equation. It was first proposed by Fischer, the same Fischer of genetics and statistics. In 1937, what he was interested in was what happens if you have a logistic equation. This is just an equation in which if you did not have this term, it will just explain exponential growth. But if you're thinking of an organism growing in the wild or even an allele in a population, you could think that there is a capacity. There might be a limited amount of food or something that is limiting so that this organism when there's very few of them will start by growing exponentially, but then eventually this will need to reach saturation when you get to that capacity. And this is the simplest equation that describes it. And what Fischer was interested in was what happens if I couple this simple, or one of the simplest law of growth with diffusion. So I put it a special term and it was interesting in the spreading of allele through population and it could show that this generates a wave and then the Russian mathematician actually gave a much more careful treatment of this showing what was happening. So what is the behavior of a logistic equation? Let's just start by thinking of a logistic equation in the absence of diffusion. And the way one goes about understanding these, and it's probably very obvious for the physicists, but I just want to remind it and that's why I drew this plot. So one could, in this case, one could just solve this equation, but we instead are going to do something a little more intuitive and based on geometry. Well, one does, one usually just plots the derivative and in this case the derivative of the concentration and then the points that are very important in looking at the dynamics are the point where the derivative is zero. Those are the points where eventually you'll get and the system is not going to change. However, there is two classes of points. If you do it in one dimension, where it's very easy, things get more complicated in multiple dimensions. So the point could either be unstable or be stable. So this point is unstable. What that means is that if you're sitting right at zero, you'll be there, but as soon as you perturb the system a little bit, if you have a little bit more C, then the derivative is positive and you'll just be pushed towards the state.  If you go to the right, you're pushed back. This is a stable state. So based on this, what you would expect is that if you start simulating this equation, you'll essentially end up with C equal one at some point. So in the system, this is what you will do, but so what happens if you add space to this? So what happens if now what I'm doing here is simulating that equation and I'm just starting with the initial condition that are not uniform. If I were to start with zero everywhere, this will not go anywhere. What I'm doing is right in the middle, I have a little bump. So there's a little bit of C that is describing my population of alleles. If we want to go with Fisher and then I'm just going to simulate it and see what it does. C grows up and go to one and then you get this nice wave spreading. Why is there a wave? Well, what one usually do, it's one makes the answer that there will be a wave and then one tries to see what are the conditions in which this wave could be fixed. This is a slightly more complicated exercise that it may sound. I'm just going to give you some insight though in the math, since we are at the end of the day, we are in a theoretical physics place, it's okay to actually go through the math, I think. And I think that some of you might find it interesting. So what we are going to do, what one usually does, is one, instead of saying that the concentration depend on x and t, one makes the assumption that there is a wave, that means that the concentration is such a function of x minus vt. This means that if you know what the profile is at a given space and time, then all you have to do is evolve it because it's just going to be a wave traveling. So as long as you don't need to know both x and t, you just need to know x and the speed of the wave and then you can describe the behavior. And of course we can start with the previous equation and drop all the parameter d and alpha by redefining space and time. This is called making an equation non-dimensional. So what now we do, instead of taking this equation, we can call this variable z and we can rewrite these as a function of z. And here the one prime will mean first derivative and the two prime second derivative. So now instead of having a partial differential equation, we get an ordinary differential equation that has this form. So you can very easily normalize it and you can rescale d and alpha. This is just, in one case, you can renormalize the concentration, redefine space, redefine time, so that essentially this is very general. Then if you want to get to your specific case, you just have to redefine your variable based on alpha and d. This could have been ct but then you can just divide everywhere by ct and it's just a right and normalize concentration. Okay, so now we are left with this equation and the way that we want to solve is with the boundary condition in which, so here there is one thing we need to be careful is that actually time in this definition that I gave time is running backward, right? So because there is a minus sign, so instead of having something that starts from zero and goes to one, we are going to actually have something that starts from one and goes to zero, which means that we need to solve, one needs to solve this equation with the boundary condition that the time minus infinity, the solution is one and at time infinity now is zero. And the way that one does it, if you have two-dimensional equation is the, I mean a second order differential equation is that you can map it into two equations by defining another variable u, which is the derivative of c, and then you can rewrite u prime. And now what one needs to do again is to find the fixed point and this is really easy because a solution of u, you just, it's a solution with u equals zero and see either zero or one will solve this equation. And what we are looking for is a solution in which now the origin is a stable fixed point because what we want is that, because our system is traveling back in time, is that it's going to end up here and that point is stable. So the way that one does this, again most of you, the physicists will notice, but the way that one does this, if you have this two ordinary differential equation is that you find a steady state and you do a small perturbation around the steady state very similar to what I was explaining is like taking your system and just moving it a little bit and seeing how it responds. That response is just given by essentially taking the derivative of this function and computing them at steady state. What that means is that essentially these are not function, these are just numbers. So essentially we have mapped a problem into a problem of ordinary differential equation with constant coefficients in which there is just a matrix A and then one could just use geometry and it's very simple if you know things like the determinant of this matrix or its trace or the trace of the matrix you can really easily see if there are unstable nodes, spirals, and what we want is a stable node. What that means is a point where the system is just going to go that is stuck because sometimes it's possible that system will evolve into something by spiraling and that will give us some non-physical solution. So when we put, but one can actually in this case the questions are so simple that one can actually just compute what the eigenvalues are and so what the eigenvalue means is that if you add that system I just showed you will mean that the perturbation in C and perturbation in U will evolve will evolve with, will evolve accordingly or this could be, I mean this one will need to compute them also the eigenvector but essentially the perturbation in U and C with some or some linear combination of them will evolve in time according to them to the eigenvalue but because we want this fluctuation to go to zero all we are looking for are cases in which bodies lambda are negative and the only way to get that for zero is essentially that the velocity is larger than two so every speed that is larger than two is a possible solution of Fisher equation and then one could do actually a lot more complicated math that I don't really want to get into but actually to show that the really stable solution is only one with a speed of v equal to so and we'll get actually this is not as simple as it sounds but we'll get back to it in a second so if one where to do a phase diagram of plotting is from a book where what I call C is called U and so this could be C and C prime what you see there are solution or in this phase diagram you get solution in which you start from here and end up there in that is what we are looking for because our system is running backward in time and in term of profile you will get something similar to what I showed in the simulation you get a profile and and this profile is traveling will be traveling over time so a simple logistic equation with diffusion could generate waves the speed of the waves will be two times the diffusion coefficient times alpha where alpha is the parameter that was in front of the logistic equation and from from the point of view of what this means for the physics or the geometry what you have is you have something that is starting from an unstable state and is traveling into a stable state however this is not the system what we are often interested in in biology and James and but James and I have touched in our previous lecture about these is the fact that often systems in biology are are bi-stable so would a bi-stable system with diffusion also generate waves and so what we want to consider is a state if you want in which a stable state in bi invades a metastable state so one could just write a generic reaction diffusion equation for the physicists at Ginsburg lambda equation and then one wants to simulate this with a potential that has this shape this is pretty much what we'll describe the physics example I was showing you before you have super cool water so you could think that this is your the state where you have ice and this is the stable state but you are trapped in this state because you haven't got a fluctuation yet that has brought you over the barrier but as soon as the fluctuation appears or introduce a perturbation the system is going to evolve from water into ice or from this low state to the ice state but what are the characteristic of this system that does it really have a wave and what are the property of the wave so we repeat the same trick we say there must be a wave and we rewrite our equation and now we end up with something like this and now instead of analyzing this in you're using the same 2d trick I'm going to explain you another way to do this which I think gives a very intuitive interpretation of what are the physical property of a chemical wave that's spread to a by stable potential if you for a second assume that this derivative of time derivative and this was a describing a position the second derivative will be an acceleration and this will be a speed so what you can think of is that this is very similar to just Newtonian mechanics in which you have a particle of mass d that is moving in a force field minus rc with a friction coefficient right because if you have minus v times c prime and this is you can think of this as speed so you will have that v will be a friction so what you really need to solve in this case and the way to think about this is that you have the potential that I gave you before and now you flip that potential because you have a minus sign in front of the force and to find the speed of the wave you could think that in this potential you roll down a ball from these and the ball goes down and has to come up and stop here however there is multiple solution that will do that because one possibility is that the ball just rolled down and the friction coefficient is so that it stops here and that friction coefficient will be the speed we are looking for but there could be another solution in which the ball rolls down goes over comes here then comes back and stops on his way back that's another solution and then there could be another solution in which these rolls goes here comes back here and goes back there and there will be another solution so there is really an infinity of solution and one could show with a little bit more advanced math we're actually gonna take a couple of minutes to go over that this is the stable solution and by this the reason why I want to show you this is that it gives you an intuitive interpretation of where the speed of a wave is coming from and we'll get back to this when we actually talk about why the speed of the wave in the embryo were slowing down so this is an important concept for us and that's why I'm taking some trouble of going through the math so this may be a little more technical for the known physicists but what one does one repeats the linear stability analysis for the differential equation with space and time and one end up with this sort of equation and then you make since this equation is linear in time you can just rewrite it as something that only depends exponentially on time times something that is a function of the space coordinate and then essentially if you do these answers you end up with a Schrodinger equation and as you do that then you can just plot this energy potential and when you do that what you realize is that everything that has a positive energy will quickly decay and as such all the solution that have a positive energy will be will essentially be unstable and so if you think there will be few solutions of speed of low energy and then all the speed of all the solution that have a high energy will be high in this case will be a low speed they are all unstable and then one could go through a bit more trouble and actually show that that this early solution so the solution in which in this potential the ball is just rolling down and stopping right here that this is the stable solution so that the highest possible friction coefficient or the highest possible speed will be what will control the speed of the wave and this gives us a nice geometrical interpretation of what is happening because if you have a if you had a best able system in which now here instead of plotting the potential this will be the derivative of the first what you what you see is that depending if you jump when the barrier is very very high then you will gonna get a very low speed but as the as this barrier becomes lower and lower and this which means that this energy barrier goes down essentially then you need a much higher friction to have the ball to stop that will mean that the speed is high so by simply thinking about how much in the unstable region you need to be to jump to the stable region you can get an intuition of why you get a wave to slow down yeah yeah we right and we think that so the waves that we have that I showed you they essentially propagate along the anterior posterior axis they are on a two dimensional manifold but if you think about the effect of curvature given what if given the the diffusion coefficient and the speed of the wave you can ignore curvature so it's essentially a one dimensional wave so we don't have to worry about this right so what I'm trying to say I guess is that if you have a if you have a one dimensional system and you can write the reaction term as a potential term the geometrical property of that potential are gonna tell you what the speed is it's gonna be only one speed that is possible in such a system that is stable and you can in principle derive that by doing the simple exercise of solving newton and dynamics of what is the friction coefficient is going to make a ball roll down and stops right here but there would be one speed and it's going to be a property of the geometry of the potential that's I guess the take home message right so so it's right I mean what determines the speed is really going to be the whole shape of the potential so it's not is not as trivial there are some limits in which you can sort of convince yourself that maybe it's just proportional to this difference in in energy but for the more general case it's going to be a function of the detail of the system we right but that will not be a solution right so so remember we have a system that goes so what the so you have something that starts from here and jumps to there right this is the physical condition is your water becoming ice once you do all this trick what you do is that you flip the potential because you had a minus sign in the equation but you also have time running backwards so what you're looking is for a solution of a ball that goes down here and comes back and stops right here that's the the friction coefficient that gives you that one can show you using this this you know shredding mapping this into a quantum mechanic example one can show that that's the only stable solution so that's just the way you're gonna get i mean that's just not so there's only one way that is possible is gonna be stable and it's gonna be geometrically you can compute it by doing this little newtonian mechanic exercise so so that the take home message here is that essentially if you have a a best able system that generates a wave and you the best way to think about how the speed of the wave or the physical property of the wave are controlled is to map these into a a problem in which you have a potential and thinking about the geometrical properties of that potential actually the speed of this wave is relative would be relatively insensitive to noise although i'm gonna show you how there is some if you make the potential dependent on time then you get noise sensitivity which is which will be our case for the waves in the in the embryo and finally if you read a lot of biological literature where people talk about the wave they use the question i showed you before the fisher equation and they tell you that the speed of the wave you get is two times the diffusion coefficient times alpha where these alpha then they will say it's the time scale of the positive feedback that control your biological system that's wrong so if you really want to compute the speed of a wave in a best able system you need to do a bit more work what why would you you so so the i'm describing waves that are so this is not an excitable system this is just a best able system plus diffusion that generates a wave and i'll show you that this is what's relevant for the control on mitosis because really it's the the entry into mitosis which is a best able switch there's control in the wave and you'll see that what closes the oscillation is just it has nothing to do with the way but i'll show you at the end okay so now let's go back to the embryo and let's look at what they do again and so so now we have waves and again you can see that these are essentially one-dimensional waves they only propagate along the anterior posterior axis there is very little effect of curvature they're clearly well organized waves that have to synchronize the cell division in the embryo and this is the main biological function is that you want to divide as fast as you can but also in a manner that is synchronous through the through the entire tissue and while this is a syncytium it will be too big for diffusion to work if you plug reasonable number for the diffusion of protein and you compute how long you will take a molecule to start from here and diffuse to the middle it will take like two hours and development is over in about that time scale so you need a specialized mechanism and it's this collective mechanism so we are not the first one we are not the first one to think about this trigger wave they were first proposed by Tyson and Novak in a theoretical paper and then some evidence for it came in a synthetic system in actually a Xenopus ag extract in a work from Jim Ferrella and they had the idea that they if you have a bi-stable system coupled to diffusion it will generate some waves chemical waves and then they show that if you look at mitosis in the in they have an extract so essentially what you do you take an embryo you crush it and you spin down the cytoplasm you pick up the cytoplasm and you put in a teflon tube and you can reconstitute some nuclei and that will undergo mitosis several mitosis and they could show that if you look at the progression of these mitosis they progress in a similar wave-like pattern and they show that you could perturb this wave-like pattern by perturbing cdk1 and they interpret that meaning that there were waves of cdk1 I'll show you that what we did was actually directly visualizing those waves which I think is better and but this mechanism was brought into question by Andrea Leo's group at UPenn because what she noticed was exactly the the same thing that I was drawing your attention when you were looking at this and it's the fact that the mitotic wave gets lower and what they also and this is shown here if you have you look at the speed of the wave as a function of cell cycle there is a clear decay in the speed but what they also noticed was that the distance between one nucleus to the next as you get more and more nuclei is also getting slower is getting smaller sorry and so they thought maybe there is actually they are not talking through chemistry they are talking mechanically so that if there is a fixed amount of time to propagate the mechanical signal from one nucleus to the next then from one nucleus to the next as you get more and more nuclei or you reduce their their distance then what you will what you should have is that it takes longer and I'll argue that this mechanism and this idea that is mechanical is actually wrong so so the first thing we wanted to show and this is I think the first thing you should show is that these they are really this is really an active mechanism in which the nuclei are talking to each other another possibility is that every nucleus in the embryo was just at its own clock but they just had very very accurate clock and everybody just knew exactly when he needed to divide you could still have a wave if there was a gradient in the timing around this clock ticked because if I knew that I had to divide that 12 and he knew that he had to divide that 12 or 5 then you could still make an apparent wave one way to think about this is actually and it's our favorite example in the lab is the Mexican wave at the stadium so there are two ways one way in which you could have a wave going around the soccer stadium one is that everybody gets up when the next guy gets up that way if you imagine that now you can't see where your next guy is because I put a barrier then what will happen is that you don't know when to get up another possibility is that I tell everybody in the stadium at what time exactly they need to get up and if I do that in a gradient everybody will get up at the right time and so the prediction there is that if I put a barrier when they are talking to each other the wave will start going but then it will stop the next guy on this side of the barrier cannot see that this guy's got up so he doesn't know when he needs to get up but if they are on that they have their own clock then everybody's getting up on time so if I put a barrier in the embryo then the wave should go right through so if there was just a timing mechanism or what is called a kinematic mechanism or a phase wave it will travel right through the barrier a trigger wave will be absorbed by the barrier so we have Andy and James have talked about mechanism to think about if cells are talking to each other by either taking them out or by genetically manipulating we do something a little cruder because our time scale are really fast they are not transcribing so we just go in with a razor blade and ligate the embryo it works it's impressive that it works but it does and this is what you see if you do that experiment actually if Victoria sitting right there did this experiment I don't think I could do them I'm not as skilled I'm an experimentalist as she is and you see that one half of the embryo divide and then the other half divide and actually if you if I let this move it go you'll see that this half does an extra division and this one doesn't so they are clearly talking to each other if you put a barrier in the embryo you can totally desynchronize it so now this wave will start and this is something you will never see a wave that will stop and a barrier like this unless there is a barrier there so they are clearly talking to each other it's a nothing mechanism well it's a theoretical possibility so I would agree it's an unlikely one but we need to prove I'll show you that you can use this trick though to time exactly when the synchronization is happening in the cell cycle so I'll show you that actually if you do inside this same experiment in certain phase of the cell cycle the wave will go right through the barrier so this will actually be the punch line of the top you all said it so I think if I understand you correctly you are asking why does it start at the pole why do you get a wave starting here and a wave starting there and they don't start randomly the short answer is we don't know so we observe that we know there is waves they are actively coupled there is a strong preference to start at the pole it's not an absolute fact but probably 90% of the wave we look they tend to start at the poles we don't really understand why I don't think there is a signal then this is way earlier before all this gradient act and definitely there is deviation so my favorite interpretation is just something geometric so the embryo is maybe more pointed at the end and maybe just there is a slightly higher concentration the enzymes that regulate the cell cycle or maybe there is a little fewer nuclei at the tip and I showed you last time that the ratio between DNA content and cytoplasmic content can influence the cell cycle but we don't know we haven't done and it's very difficult to do an experiment that conclusively show why the poles are special but the wave is real James right right so I guess what you're asking is how well ligated do they need to be to be separated do you have you looked at that so by is about so how far do you need to usually go down to almost 10-15 micron to the other side the best estimate so so we really have to go really deep because this is about if you look at the height of an embryo it's probably about 150 microns so you have to really push it down all the way to about like all the having at 10 to 15 micron constitution at which point I think we're applying so much pressure that it's very difficult for things or very slow for things to diffuse through sure yeah the thing is I mean it's not so easy to ligate in different geometry I mean like this is a small object they're really delicate I mean yeah it would be great if you could go in and just block it in a circle and see how they travel around or generate this graph or do it in two places and see if the waves start interfering is yeah yeah yeah no I don't I don't think we have to I mean this is like very simplistic at this point okay so sure so okay so this is an important point they everybody's trying to oscillate so the cell cycle is going on in every nucleus independently the role of the wave is just to make sure that if you're a little delayed compared to your next guy just gives you a little kick so that you go on time so it's a synchronization mechanism but everybody's trying to go through the cell cycle so this is not like an axon like an action potential traveling down action where you start like the excitation in the neuron and then you just you there is a signal propagating and if you cut there will be nothing going through if you I mean technically it's essentially impossible you will need to raise a blade with within like this is all the alpha millimeter long and you will need to put your answer but it's so close to each other and really control them accurately it will be very difficult but if one could do that experiment this nuclei will still divide I think they would just you left triple three different area out of sync but this is an important part though everybody's trying to go through the cell cycle everybody's making city k1 activity and is trying it this is just making sure that they stay in schedule and what this experiment show you is that if you don't if you don't let them talk to each other that can fall significantly out of schedule and that will let we think influence cast relation and a lot of other processes because the fly embryo developed so quickly that you really want to get to the point where you make your genes all together so that everybody then synchronously start this more for genetic processes so that's the interpretation you also have a question so are you saying that they were in one place? those droplets they make that sea bottle they could make those droplet they're really small so I don't think that they've looked as spatial but the droplet are I think of such a size so what he's talking about are experiments in which people go in with the needles suck out the nuclei and put them in a droplet on a cover sleeve but those droplets are probably about 30 or 40 micron in size and diffusion is even enough you don't need so the problem with diffusion is that it goes like the square root of time so it's very easy to diffuse small distance but when you want to diffuse large distance it becomes very difficult that's why you need this specialized mechanism okay so what did we do and I think this was our major contribution through this was to actually find a way to to look at CDK1 activity which as Jim Ferrell proposed and if you work on cell cycle and you remember Monday lecture this is the master regulator of the cell cycle so why don't we look if there are waves in CDK1 activity but what we wanted to do was not to just look at mitosis as a proxy of activity we wanted to measure the chemical activity and we were very lucky because John Pines and Cambridge and made this great sensor which worked very beautifully for us in the embryo and this is a threat sensor I explained it last time already but the idea is simply that CDK1 is a kinase what the kinase does it puts phosphates on a peptide if when it puts a phosphate on this peptide this peptide changes conformation these two molecules get closer as they get closer it is possible that a depot moment excitation from this molecule is transferred to that molecule so as you excited this molecule with blue light you don't get blue fluorescence out but the energy goes to this and you get yellow fluorescence instead of blue fluorescence when they are far apart this is great that is a great molecular rulers because the typical distance for this thing need to be about few nanometer close to each other so if you design this well it's very likely that they will only threat when they are in this conformation making threat sensor though is much harder than I made it sound not should be very rarely work but it worked beautifully for us thankfully and we get this really beautiful oscillation as a function of the cell cycle and if you mutate the ceiling where there is phosphorylated as you would expect you get no activity the same is true if you knock down cdk1 I'm not showing you all the controls with that we did but it works and so what we can do is really not only measure this activity very accurate in time we can also do it relatively accurate in space definitely accurate enough to see the waves so what we know the way that we normally plot it we plot the activity with time on the x-axis and spatial coordinate on the y-axis and then the way we go about looking if there are waves or we meaning Victoria does is she takes an embryo computationally divides it into several slides and we can do this because the waves mainly seem to travel along the anterior posterior axis so we can map this into a one-dimensional problem in space and she looks at the activity in one region where the wave starts and then you look at the activity in another region down and what you get is what you would expect if there was a wave you get one profile in one region and about the same profile but shifted over time so this is a wave that is traveling from here to there and that is what you're seeing and then we do simple computational tricks in which we do cross correlation and we found how much was these lies the activity here delayed with respect to the activity there we find the maximum of these and then we plot this delay that we estimate as a function of the distance and you get more or less linear profile at least in a first good approximation which are indicative of that this wave travel more or less a constant speed and because we have time versus distance the inverse of this slope will actually give you speed so ice slope means low speed low slope ice speed and you can do this in a embryo for various cell cycle and this is just another way to show you again that the wave gets lower so cycle 10 and 11 you have very slow slope fast waves and then a cycle 12 the wave is a little slower and cycle 13 is significantly slower you do this for like 50 cell cycles various cell cycle various embryo and you end up with plot like this so you now have the speed of the wave a cycle 10 and 11 is high and then cycle 12 and 13 is significantly lower if you have what which mutant or if you block CDK1 it doesn't divide yeah it gets stuck I mean it's absolutely required for division oh yeah I'll tell you in a second that they haven't told you yet what this is it's actually theoretical prediction is even a big word but what we what we did was not only so far I've just told you about CDK1 activity but if you if CDK1 activity really was the and the wave was what would draw my ptosis what you would expect is that as you see CDK1 wave traveling you also see my ptosis traveling right so what we what we do in our experiment we not only measure CDK1 we have a third fluorescent protein on which we visualize instance and we look at how instance change shape because that's a proxy of my ptosis and when the nuclei split and the theoretical prediction is that these two waves should have the same speed right if really a CDK1 wave drawn a methodic wave so I don't know that you can call it theoretical it's just a prediction is that if this wave is driving that wave that should be in a one to one relationship and this is exactly so this would be the identity line and this is the best fit line so you can see that it's true these two waves travels at the same speed yeah the the color the different color at different cell cycles so you can do this at cycle 10, 11, 12 and 13 so what you see is a cycle 13 all the points are down here so both the CDK1 wave and the mytotic wave are very slow and then at cycle 12 they're a little faster and then cycle 10 and 11 they're much faster and you can see also this this noise and look at that to them here oh what these colors are so that's the mean that's the standard deviation and that's important you can ask Victoria later she made the plot yes i will get there it's uh i need to bother you with a bit more molecules before i'll get there unfortunately but i promise so i'll try to answer that and we can talk more later if you're not satisfied yeah right oh i don't know you measure a speed getting number of oh right so i'll show you i'll show some experiment that really arguing is the mechanical signal and then if you're not satisfied we can talk again okay so what i'll show you is that you can perturb CDK1 activity and i can exactly predict what the speed of the wave will be and that's what the speed of the wave is and those predict those manipulation i don't think you know it's like kinase that specifically regulates CDK1 so they should not have any and they certainly they have no effect on the nuclear no inter nuclear distance and i'll show you that there's still the speed of the wave changes so i but i need a bit of time no really so it tends to start at the pole it doesn't normally always start at the same pole at the same time but we haven't really seen a bias starting at the anterior before the posterior no okay so one does when one doesn't know anything in physics or yeah so you can you can probably see waves in a lot of phenomena and i'm gonna argue that the master clock is the cell cycle and all the rest follows and i'll let's see if i don't convince you okay yes you you will locationally see that but you can also convince yourself that curvature you can know you can ignore curvature in this case just the curvature effect done last long essentially there are on a very small scale oh you never you'll never see i mean if anything you will go in every direction right yeah yeah so it's activity moving nobody's moving here there's no flow this is just activities like one molecule activating the next molecule activating the next molecule yeah well that most of the nuclei and the cytoplasm is on the surface and the yolk is inside CDK1 is in the cytoplasm and nuclei mainly and there may be some in the yolk i'm not sure actually but we are we are it's very hard to go inside because those are 40 micron tall cells and we confocal microscopy we don't have a solution and there's nothing interesting happening in the yolk we think from the cells are not as interesting okay so well one does in physics one when you don't know anything you do dimensional analysis and it's like okay how do you get the speed well you get the speed by the square root of diffusion over some reaction rate and so maybe the reason why the wave is getting slower is because CDK1 is all of a sudden diffusing slower and you can kill that idea very quickly because you do a frappe experiment on CDK1 and it diffuses at the same rate as cycle 10, 11, 12 and 13 so what a frappe experiment is is an experiment in which and because I'm not sure that everybody knows but imagine you have your embryo and then you have the GFP molecule in this case with a YFP molecule and you remember from Thomas lecture another lecture people explain you what that is hopefully so the YFP will be everywhere but then what you do is that you go into let's say a box and you just kill all the GFP with your laser and then what is going to happen over time is that GFP molecule from here are going to be diffusing in and from looking so if you were to plot the intensity in this area you will start high and then you go in with your laser you destroy all the fluorescence your fluorescence will drop but then molecule diffuses in and so this will go back up right so from from feeding this curve with the appropriate curve from theory of diffusion you can essentially estimate the diffusion coefficient or you can just look at the curves and if you just look at the curves I don't think you have to work hard to convince you that they the diffusion is not changing so something is changing about the chemistry or about the reaction in the CDK1 system what could it be so because we can measure activity we can try to ask the to answer that question so we go back and we look at our plots of activity and the first thing you might guess so maybe the activity which is CDK1 is going up around mitosis is changing but that again you can very easily convince yourself that that's not the case because these loops really don't change however if you thought you saw let's say cycle 13 that is getting significantly slower now what you see is that the activity is not monophasic anymore you do sort of get this biphasic behavior but and interestingly you all remember from the other lecture that the cell cycle in the early ambience made up two phase there is DNA replication and there is mitosis so during DNA replication CDK1 activity goes up slowly and then it goes up fast in mitosis so we reason maybe this this slow down that is causing the slow down of the wave and that rate is indeed getting slower and seems to be somewhat predictive so that if we plot speed versus that activation rate we actually do get that we can sort of predict the speed then we actually get a slow test not that far from from a square root and we'll get back to why this system kept why we can predict to the waves okay so if now I'm correct that is the rate in s phase that is controlling the speed of the wave I should be able to do an experiment in which I change their rate and and I should change the speed of the wave so that's exactly what we thought you did remember I already talked to you about on Monday about the fact that as you get more and more nuclear inside the embryo they have this mechanism to prevent that you enter mitosis before you're done replicating your DNA it's called the DNA replication checkpoint so as long as as soon as there is a problem with your DNA replication machinery it activates a signaling cascade and what this does this puts an alt on CDK1 it tells so if there is damage or stress activates this molecule called check one and that activates a repressor it needs the activator and CDK1 activity it's progressing slower or activating slower so the prediction then will be that if we we mutate this this kinase or the major effector we want we should lose this biophasic behavior and this is exactly what you observe you now don't have this would be the cycle 13 and you see that there is no biophasic behavior and because there is not such a biophasic behavior and slowdown of S phase actually the cell cycle are faster and they keep on dividing they do two extra divisions and the same is true for we want so now I've lost this low activation the prediction should be that the wave should not slow down and this is exactly what we saw so this is a movie of check one check two you just saw first very fast wave go through and you'll see a second one is still extremely fast and then there will be a last one which is again really fast they can split the DNA but you can still characterize the wave see yeah you do oh as low down here right so cycle 12 is 12 so normal cycle are about 10 minutes cycle 12 is 12 minutes and cycle 13 is 18 to 20 minutes so the slowdown gets pretty significantly longer as cycle so as cycle 12 there is a slowdown we can measure it we can see it use low space very subtle yeah you're right it's much more subtle is just because there is enough DNA to activate the checkpoint but the checkpoint is not that active and I'll show you that into a set we have also have a check one sensor so now if I put the numbers here you'll see that all the speed in a plot of CDK one activity to speed all the speed in check one check two and we want no lie and all the rates are high and importantly you get the same relationship so the data appear fall fall on the wild type curve so this goes back to your question these embryos the nuclear nuclear distance was changing because you are still getting more nuclei so if that was the mechanism that is playing the slowdown of the wave this wave should have got slower but they didn't and they didn't because CDK one activity did not slow down so we think that this is a strong argument against any mechanical signal because we can perturb the wave in a quantitative predictable manner by just changing CDK one signal which I think is a strong argument we also have a check one sensor and this is just a sensor when it gets phosphorylated there is a peptide that when it's phosphorylated it goes out of the nucleus and when it's dephosphorylated it goes into the nucleus so by measuring how much of this is outside of the nucleus so how much of it is in the nucleus you can estimate check one activity and this is what you see you see that you have very low activity at cycle 11 a little higher at cycle 12 much higher activity at cycle 13 but you also see something else which is probably the most remarkable feature is the right one minute before the nuclear envelope breaks and that's why this curve stops which is the earliest mitotic event this activity just plummets so check one is high but as soon as you enter mitosis boom we go down and it goes down in a wave-like pattern which and I probably did not put the data to show you but this wave-like pattern is exactly the same wave-like pattern that you see in mitosis maybe for the sake of time I can skip this so what our data argue is that there is somehow CDK1 that drives mitosis is also communicating back and repressing check one and so now to try to understand this a little more theoretically and to link it back to the potential what we did was essentially write a simple chemical kinetics model for this it's just actually I'll show the question in a second but what the model is doing is just assuming that there is a given dynamics for check one and all that is happening from one cycle to the next is that the initial condition of how much check one activity you have is different so cycle 10 you start with little check one activity and then cycle 11 you have a little more a little more cycle 12 and then a little more cycle 13 and then makes you go from something that almost accumulates linearly to something that becomes more and more bi-phasic and this model generates waves and gives you the right scaling so the model works now let's try to understand why the model works and so this is what the model is for the question so the first one described check one activity write the inhibitor up here and it's just what it's just saying is that this activity diffuses and is repressed because this is a negative sign here is repressed by by CDK one activity so this is just described this repression or this feedback and then there is an equation for active CDK one which is diffusing and it's activated by check by this positive feedback and is repressed or is activated by CDC 25 and is repressed by V1 and these feedbacks are all incorporated in this function R plus and R minus and I can tell you what those are if you want later and we also have nice in the system but we also need an equation for all the total amount of CDK one so this is a still a bit of a complicated system we really want something a bit more intuitive but the way that this is written and the dynamics of that we saw in which check one really seem to follow CDK one suggest that we can just make an assumption and just solve this equation a steady state and get some estimate of what F is and plug it in here and also one can convince itself if you put the value for diffusion and the time scale that and what the curvature effect are that really that this C term is going to be very uniform there is really very little reason to believe that C is diffusing much and this much these are this homogeneity in the total amount of CDK one so you can ignore these at this C drop a function of X just having a function of time and put a simple linear increase put that in there and so you end up with a simple equation and now we are essentially now I'm back to the same Gisborne-Landau equation I wrote before and this is what the first field look like to answer your question so this is a cycle 11 and this is a cycle 13 so at cycle 11 and this is how it looks at time t equals zero so there is a very very small region of stability at the low state and then there is a very quickly only an high state and the system very quickly actually transition from this low state to this high state and if you think in term of the potential there is really very very little energy barrier to jump over but at cycle 13 as you increase this inhibitory activity upstream of everything now the force field changes completely and at the beginning you actually only have a stable state of low activity and then as time progresses the system becomes by stable and as I showed you when we do our experiment and we don't think it's measurement noise there is a lot of variability so what we think is happening is there is an important feature here that I did not describe when we went through the math of the wave is that this potential is not fixed with time the reason is that you are synthesized molecule and as you synthesize molecule this term is a function of time essentially and so the potential is changing so at the beginning there is really no high stable state but then this potential is changing with time and at some point if you think of these as being the amount of active CDK1 it is possible just that there is a fluctuation that brings you from here to there at that particular point you jump from the low state to the high state and you initiate a wave that travels everywhere but if you do an experiment the next day you're not going to jump right exactly at the same point you're going to jump a little earlier or a little later if you jump a little earlier you see a slightly different potential and I told you that if you know the geometry of the potential so the wave travels really fast compared to the time at which this is changing at least in first approximation so you could do an adiabatic approximation in which you say whatever the potential is at the time you jump that's what is going to control the speed of the wave then that explains why you have noise because this is a noise trigger system as soon as this the energy barrier becomes compatible with noise if you want to jump but you're not going to jump always at the same place and actually you can take this model and simulate it and as you increase and increase noise as you would expect you will get the way the jump to happen earlier and earlier and you get slower and slower speeds of the wave so that implies that actually that the noise controls both the time of the jump and as a consequence the speed of the wave and actually because there is noise in the system and you might have already noticed how waves are not perfect yeah right well I don't know I mean I don't know that you can change the the speed of the wave by temperature and you can make everything but you change everything right and you change everything proportionally the cell cycle gets a little longer the rates get a little slower and it looks like the wave is a little slower it's hard to you don't make such a big changes because this all goes like I mean the changing on the rates are not that high and you could push you and if you really I mean there is a limit temperature in which you can work and still are things are still healthy so I don't know there is a great test I mean that I guess a better one would be to do some manipulation to change noise but yeah yeah well it's a nice in the activity of cdk1 which is and what is driving it we don't know but there will be some noise could be you know but there's noise in chemical reaction and there's noise in the concentration there's there's going to be fluctuation in the system but we don't really know what it's driving the what the source the final source of noise is we don't know actually okay so this is going back to to the potential the this is the this is the the simple geometric interpretation of why the wave will get slower if you have low cdk1 you have a very small energy barrier to jump so it's very easy for the system to trigger a wave and this wave will travel very very fast because if you were to invert this potential you need a friction if you did the exercise or rolling a ball down and having to stop but as this barrier as you get to cdk1 now the potential is changing such a way that you get a slow speed so other than explaining why the wave is getting slower this model made actually probably the most exciting thing that came out of the model is that it made an important prediction and the prediction was that if the system is really by stable as soon as you jump from one state to the next from the low state to the high state then the wave will just go so the prediction is that as soon as you finish or as soon as you turn off cdk1 or as soon as you finish s phase now you have started the wave so at this point the wave should just travel so if we do repeat that experiment of mechanically ligating the embryo but we do it as soon as they've completed s phase we should see that the waves goes right through the barrier and this was really a tricky experiment to do but we thought it succeeded my great happiness so the idea is the following is that if you ligate in s phase they are still talking to each other and therefore this two half should become decoupled but as soon as s phase is completed the wave is already if you wonder what the wave is doing is synchronized near the replication and my thought is just everybody's on his own clock so as soon as they're completed s phase now it should switch to this scenario in which everybody is being programmed and now everybody has been told at which time they should divide they should not care anymore so if I put a barrier here you should see the wave to go right through and let me show you what happened and I think you will believe that that's the case so if you ligate early in cycle 13 and you see what happens this half divides the wave stops there and then this other half divides and for this nuclei to divide they have to wait for the second wave what happens if you ligate an embryo that is just enter mitosis the wave will start and will go right through the embryo right through the barrier and we have done this many times at least five times and you always see the same thing so if you look at the activity and you ligate in s phase then you get a wave that will start at one pole will travel and be absorbed and then to synchronize the other for half of the embryo you need to wait for the other wave coming from the other side so these two half can get desynchronized by few minutes but if you put the way the barrier keep saying the way sorry if you put the barrier as soon as they've entered mitosis it goes right through because desynchronization has already happened now everybody is just following his own clock and yeah we made it at cycle 13 because this is the cycle that has the longest s phase and give us room to work with right so so I think we have probably in some of our like the first ligation experiment that showed actually in the allegation experiment we ligate way earlier you ligate like a cycle eight but if you there's two ways in which you can ligate there's a way a way in which you put the razor blade down and leave it there for 10 minutes and then if you remove the razor blade the two half will stay fused that's a permanent barrier and if you put that embryo down and you look at cycle 12 you'll see that they become slightly asynchronized the other way to the problem with that is that as you are ligating they are dividing so you don't you cannot have this time resolution that we need so then another possibility is to build a new apparatus in which you put the blade and stick it under the microscope right away because this is all happening on Monday or minutes so I think that cycle 12 I will say it's definitely a trigger wave cycle 10 and 11 I still think that they're still so organized that I think they are a wave but I will guess that if you put a barrier you will not I mean it's everything is going so fast that you probably won't desynchronize them so much so but clearly cycle 12 and 13 really need this active mechanism and and that is also through if you like it early you'll see that the delay accumulate and they really become more they really become very significant prominent as cycle 12 and 13 so it's a good point that I still think there is a wave in the earlier cycle but it's not as important as later on if because that is how it's supposed to be not 9 or the left or down oh right so so this was like it is probably around here right around six I will say yes so you will assume this is time from when the experiments start the cell cycle are very reproducible they always divide if you do the experiment around 20 to 23 degrees cycle 13 is 18 minutes plus or minus 30 seconds so if you want if you want to guess where the zero of so this will be 18 and these will be also be 18 so so probably this movie was started you are right about six minutes or these embryo was like it about six minutes before that embryo was like it so the zero is the beginning of the experiment when the the apparatus because you you don't right what the way that Victoria does the experiment is that she watches an embryo and as soon as she can see that they are about to enter mitosis she like gets it very quickly like 20 seconds and then you you imagine you don't I mean I don't know we probably did not record the data we don't not plot it before they got there but okay so this is it I'm finishing a little earlier just the the acknowledgement probably most of you have met Victoria she's sitting right there she's the one that did all the experiments and the theory was done in collaboration with Massimo Vergassola the UCSD and his former postdoc Anna Melbinger and Massimo is the one who taught me all the math I know about that wave propaganda


Welcome to another edition of RCE. I'm your host Brock Palin. I have again Jeff Squires from Cisco Systems and the OpenEMPI project. Jeff, welcome to the show. Hey Brock, how's it going? Today we got on someone who I've actually known for a while. It would be interesting stuff to hear his views on clustering and whatnot. Yes, we have Josh Bernstein and he's from Penguin Computing. We're going to talk about the Skilled Cluster Kit. We'll actually let him describe what that is. I've never actually used Skilled, so this will be informative for me. Josh, can you give us a little bit about yourself and where you're from and what your background is? Sure, so I'm a senior software engineer with Penguin Computing. I've been in the clustering industry for more years than I care to remember at this point. My background comes from NASA. We worked at the high performance computing lab. Mission credits include things like the high-rise camera, Mars Reconnaissance Orbiter, and those beautiful panoramics that you see on Mars that the little rovers are running around with. Those are some images that came down from the group that I worked with as well. Hey, little known fact, some distant relative of mine, Steve Squires, was one of the heads of the teams over there running all those Mars rovers. I never would have made the connection, Jeff. Well, there you go. A little trivia fact for all of you today. Okay, so Josh, can you actually tell us how you describe Skilled and what it does? Sure. Skilled is a skilled clusterware, as it's known, is a cluster management solution. The idea being that it sits at a layer above the operating system that's designed to handle the maintenance and administrative tasks involved with the day-to-day operation of a cluster. So tasks include things like provisioning the nodes, you know, getting the nodes enrolled in the cluster. All the way up through, things like middleware, like making sure OpenMPI is compiled properly or optimally and working and tested and so on, and interfacing with schedulers, things like Torq, Moab, and even SGE. And the goal of Skilled Clusterware from a design perspective is to keep things simple. And by keeping things simple, the other goal is to sort of leverage existing Linux experience. We don't want to be in a point where we're asking people to, you know, relearn a new set of commands or learn sort of new concepts or new syntax for configuration files. We just want to leverage existing Linux knowledge in order to carry that over to operating cluster. That's actually interesting because I thought Skilled was just this, like, unified kernel thing you guys did in the back. I didn't realize Skilled was this entire package of everything else. Yeah, so part of Skilled's background or sort of Skilled's history is the technology you're talking about called Bproc. And Bproc was a project that came out of Donald Becker's group at NASA Goddard a number of years ago that essentially creates a unified process space. The idea being that I can type PS on the head node and see processes running on the remote machines. And further just being able to do that but also being able to maintain all of POSIX semantics. So if I issue a kill on the head node to a process and that process turns out it's running on a compute node, Bproc is then responsible for storing and forwarding that signal out to the remote nodes. But that's just a component of Skilled. All the other processes or all the other problems involved in building a cluster also have to be solved. Provisioning is another example. There's well-known problems with the Pixi TFTP implementation. A naive Pixi TFTP server could cause nodes not to boot up 100% of the time. And then after provisioning there are things like security, name service resolution. We've actually implemented a cluster aware cluster specific name service for things like user names, group names, host names, et cetera, et cetera. And the goal there was really to, I guess as name services as an example, the goal was to take the complexity of managing cluster where you normally have to move files around or set up another DNS server or so on, was really to credit a much more simplistic service that handled or solved the problems specifically on a cluster. And so that's another component of Skilled. And then on top of that is the middleware. We custom-build and test and do a lot of work with OpenMPI. We also ship Mpitch and Mvapage to give our users a choice. We also ship Torq, and Torq is sort of, quote, working out of the box, if you will. You don't have to compile it. You don't have to worry about adding, you know, worrying about host lists or nodes lists. We wanted to really make the experience of building and managing a cluster as easy as possible. So hopefully that gives you a point of comparison. B proc is really just a component in the whole package. Oh, okay. So bear with me, because the majority of the stuff I looked up must be B proc then. Because that's actually a really interesting tool. So that came out in NASA. And actually, doesn't that guy work for your company now? That's right. That's right. Donald Becker is our CTO. What happened with Don was after, you know, Don wanted to basically build a system that was capable of performing, like the large SMPs, the craze at the time, convex machines at the time, that rivaled those machines in performance for a fraction of the cost. And of course, in doing so, the first thing you're going to do when you take a commodity set of machines and plug them all in is you're going to need network drivers. So Donald Becker is most well known for writing a large number of the network drivers that are popular on Linux systems today. But after Don left NASA, he came and founded a skilled software, which was then later acquired by Penguin Computing. Now, a little known fact actually, I'll go along with Jeff's little theme earlier. Skilled, the name of the product goes back to the Beowulf legend, where we all talk about Beowulf clusters. And in the story of Beowulf, the character Skilled was actually Beowulf's father. So that's sort of where the name of the product comes from. So Skilled, and well, the B proc part specifically, does not actually, is it something kind of like a, you know, VSMP? Like, will it actually make a shared memory system out of all these compute nodes? Or is it still a, to do a parallel application, I still need to use distributed memory programming like MPI? No, that's a very good question, actually. The software creates a unified process space, but does not unify the memory space. And that was a really strong decision or a really conscious decision we made, even in Skilled's early stages, where unifying an address space is extremely difficult to do. Unifying a memory space is extremely difficult to do well and have it work 100% of the time. And with the popularity of things like MPI and other distributed memory paradigms, it just didn't seem necessary. And unifying the memory space has a whole other number of problems like latency. And also, you know, when an application crashes, does it crash legitimately or did it crash because there's something wrong with that unified address space? So we consciously chose to stay away from that. So as Penguin, you guys are basically cluster resellers, right?  You don't just sell a pile of boxes and tell a customer good luck. You're actually, you know, your intent is to sell a whole solution. And your solution is generally delivered in the flavor of Skilled. Would that be an accurate representation? Yeah, I think that's a good distinction. I think that Penguin is essentially a value reseller or a solutions provider, if you will.  Some companies go so far as to rack and cable it for you and ship it to you. But that's only really, you know, the hardware is almost an afterthought, right? You have to be able to provide the hardware all the way up through the customer's application stack. And so part of what Penguin does is, of course, provide hardware, provide Skilled as a cluster management solution and provide a scheduling interface. But then also, we have application expertise with, you know, end user applications. So if a customer calls us and say, you know, we're running Ansys and Abacus and Fluent and we don't really know what kind of hardware we should buy, or we think we should buy this hardware, what do you think? And we have PhDs and other experts in various fields on staff to assist customers with buying a solution that meets their needs, not just something they want. Oftentimes customers, you know, call us and say, hi, I want 100 nodes with 12 gigs of memory per node.  What are you really doing with the hardware? And our goal is to really provide an end-to-end solution among anything else. So Josh, tell us a little bit more about the underlying technology. Most of the people who listen to this podcast are probably aware of the various cluster resellers out there. What makes skilled cool? If I was a system administrator or network administrator, what are the cool things about your solution that make it better? Tell us a little bit about your technology. Well, I think that one of the biggest advances or one of the biggest values in our technology, again, is to, it's designed around the idea of keeping things simple and also leveraging existing administrator knowledge. So Brock earlier mentioned BPROC, the idea that as an administrator, I can log in and I say, shoot, how do I see the processes on the cluster? Well, I run PS. I don't have to, you know, memorize some other funky commands or remember to SSH to 12 nodes to see what's going on on the cluster. The goal here is really simplicity and to take the burden of provisioning a whole operating system out to a node out of the equation. For example, our provisioning model is very unique. I like to call it diskless administrative. This is the idea that the compute nodes themselves don't require disks in them in order to be part of the cluster. We send out only the small kernel, the small int rd and build up sort of this, you know, very, very minimal running image on the compute nodes. And what happens to the course of execution is that when a process lands on the node that needs a particular library, we have a library caching mechanism where the node will say, hey, this program needs libfoo. I don't have it. So instead of relying on NFS or instead of relying on, you know, some other sort of file transport, the compute node goes back to the head node and says, hey, I need libfoo and the compute node serves the library up and then it is cached for later use. The beautiful thing about this is that from an administrative perspective, you really can install and maintain the cluster from a single system. If you, as the administrator, then go back and upgrade libfoo, you don't have to go push libfoo or the libraries out to all the cluster nodes. They sort of just end up there or migrate out there through the course of execution. And the goal here with this technology is really to maintain consistency. There are other cluster management utilities that try to maintain consistency through synchronization, you know, you add a username and you forget to run that magical command that synchronizes everything together or worse you remember to run the command but, you know, it fails to synchronize and the key with skill is that synchronization doesn't guarantee consistency. And so our consistency model is built into the design of the software rather than leveraged out there. So Josh, with that library caching thing is kind of interesting. Can I use that to kind of upgrade stuff on the fly? Like say, you know, libc or maybe libc is a bad example. Say something that's a little more like less used, like some library you're using. Could I actually drop a new one on the head node and would it kind of invalidate the cache on already running nodes and I could kind of push new ones out there for new applications or do I have to kind of restart everything? Well, no, that's a good question. Let's say you're going to upgrade, let's pick a Python library. Let's pick SciPy is a very fun little scientific library.  All right, let's pick one that you like, Brock. How about FFTW? Okay, yeah, FFTW is good with me. The idea here is that you can upgrade FFTW on the head node and then when you think about what happens when an application is actually launched, the runtime linker takes a look at the library that the application is linked against and, you know, figures out where in the address space you're linked in.  Now, when the application migrates out to the compute node, if the compute node has a cached copy, the compute node recognizes that this new program is linked against a newer library and thus it must invalidate its cache and fetch the updated library. So through this mechanism, you're able to upgrade and change and install new libraries all on the head node and then those libraries are sort of deployed or pushed out to the compute nodes on demand, right? If an application never needs FFTW on a particular node, it never ends up there. The distribution or you can think of the OS distribution being built on the fly. So that's actually pretty cool and very, very genuinely useful speaking from the perspective that I have to maintain my own development cluster at Cisco because I'm not high enough on the food chain to merit some sysadmins to help me out. But how does this work into your all maintenance? I mean, do you guys have a very customized linker and a very customized kernel?  That's a good question too, Jeff. You know, one approach to being able to solve this would be to ship custom versions of all the crazy commands that you could think of. You could ship a custom PS and a custom, you know, cat and whatnot. But it turns out, you know, that that's not really desirable. So the distribution itself is a hundred percent, you know, red hat standard. The only thing that we do change is the kernel. The kernel is sort of patched so we can have hooks in a certain places in the kernel to do this sort of thing and guarantee the ability to maintain, you know, signal delivery and so on and so forth. It's very, very similar to what VMware does, actually. VMware or any other virtualization that relies on sort of a runtime patching of the kernel goes about the same approach. In terms of our policy, when Red Hat issues a new kernel, for example, or a new kernel update, our policy is to have a update or a patch to our users within 30 days of the Red Hat announcement. And it turns out that that's actually a very smart thing to do because sometimes Red Hat releases a patch that might break an Infiniband driver or ruins the performance of a gigabit network driver. And so that 30 days gives us the time to do the engineering work, which is actually very little work, but most of that time goes into testing and queuing and makes sure that that patch doesn't actually break anything. So it sounds like you guys focus quite a bit on performance. As you mentioned earlier, you have, you know, application specialists and you are very concerned about performance of network drivers and things like that. I mean, that's very refreshing to hear from a cluster integrator. So how do you guys actually focus on HPC or do you sell clusters to other groups?  No, our core focus from a business sense is all for high performance computing. The interesting thing is that people generally think about HPC in terms of scientific applications, but we equally sell to, you know, web analytics companies or let's say insurance agencies doing risk analysis and certain areas of markets that I think that people don't classically think of as scientific computing. But generally everything we do is geared towards performance and geared towards, you know, the scientific or the HPC market. And maybe with, you know, as the years have progressed, the HPC market has brought in to include, you know, maybe a little enterprise shop here or an insurance or a mathematics agency over here. We also service those customers as well. So what does this, how does this B proc scale with the number of nodes? This unified process space seems nice with, you know, 64 cores, maybe 128. With a, you know, 10,000 core system, if top is showing 10,000 things consuming 100% CPU, how do you actually manage to sort this huge list and manage it if it's just regular top? How do you see people actually using this? You know, that's a good question. A lot of what happens when you build clusters at that scale is that large cluster is broken down amongst smaller clusters or partitions as I like to think. You begin to think about, okay, well, maybe I'm going to take, if I have, say, 10,000 cores, I'm going to take divided into, say, 10,000 node chunks and sort of abstract that away for the administrators to deal with the process space and so on. And from that perspective, the view of the cluster from the user's perspective is really done through the scheduler. So you leverage a scheduler there to bring those subclusters together. And I think this is done with or without BPROC. Although I will say that some of the larger clusters in the national labs are using BPROC to run, you know, 9,000 cores on. It really is just a personal preference, I think, from the administrator's perspective. I've seen it done both ways. I personally like to build, you know, at that scale, build sort of smaller components and scale out from there. So two questions. What is the largest BPROC managed system as well as the largest skilled managed system that may be broken up into multiple BPROC systems? And then also, what's the failure mode for BPROC if a node kind of goes away in this unified process space? In the case, I'll answer your last question first. In the case where a node fails, a node fails. That's, you know, the node went away, a node went away. That doesn't disrupt the unified process space whatsoever. In terms of the largest BPROC cluster is probably either at Livermore or Los Alamos. It's probably in the 9,000 core range, as far as I know. And then in terms of the largest skilled cluster, there's a cluster that went out to Georgia Tech that will be announced before supercomputing, actually. That will be in the 415 node range, roughly 10,000 cores, and that'll be a top 25 machine. And so is Penguin Computing basically doing all the development of BPROC going forward and it's a product that's only available through you guys, or do you resell it sometimes? BPROC, you know, is technically an open source project. I think there's still the old BPROC source forage project lying around. So we've never actually been approached about people asking for the open source version of BPROC, but we'd be happy to provide it if somebody asked for it. We sell skilled directly to end customers, but we've also sold it for, you know, resold it through another solutions provider, or sold it to a customer that already has hardware. There's nothing about the Penguin hardware that makes skilled work, skilled works on any x86-64 machine. So just out of curiosity, you know, I'm a little bit familiar with the background of this question, so it's a little loaded question here. You have this kind of a different model for Skilled, that, you know, they really are different machines, but they kind of sort of look like one machine. My question is, do applications need to integrate with this model, or do they just kind of run seamlessly, or if they did want to adapt to it, what could they do, and, you know, what would be neat and interesting to be able to exploit on a BPROC kind of architecture? Well, I think that, you know, first of all, applications run out of the box. There's nothing special that an application needs to do in order to run on a skilled cluster. Our goal, again, is to provide a solution, and if we had, you know, cases where applications needed special customizations, there wouldn't be much of a market for our product, but I think a lot of applications tend to expect that, you know, everything is running on a remote node, so I have to do everything on a remote node, and therefore I have to always double-check my consistency, or I always have to check to make sure that file systems are available and so on. One of the biggest things that Skilled provides is not only a unified process space, but a unified location for looking up various configurations or various status info. For example, the head node maintains a shared memory region that keeps track of the compute nodes, their CPU utilization, their memory utilization, and so on. So if an application really wanted to optimize itself when it wants to figure out what's the load on node 4, instead of logging into node 4 and asking, you know, node 4 for the information, simply ask the head node. And in fact, what we've done with Skilled is we ship ganglia, which is a well-known sort of cluster usage or performance graphing system. And ganglia is really, really bad if you think about it in its classic architecture where you're required to run sort of a special ganglia daemon on each node, and thus any time you have a user that clicks the refresh button in ganglia, I think it says get fresh data actually these days. Ganglia goes out and spams the network with all this XML data, this, you know, XML request out, XML response back, and then tries to package all up and graph it. Our version of ganglia that we ship, when you click get fresh data, all it does is just poke the shared memory region on the head node to pull out the status information. And that way we've alleviated the sort of management overhead on the Ethernet network associated with managing a cluster. The same thing happens with schedulers or, you know, other graphing applications. Nagios is another example that could take advantage of this sort of model. So I think there's a lot of efficiencies to be had by integrating with a system like Bproc for applications such as ganglia. Okay, now what about other types of applications? You know, does Bproc offer hooks, for example, for process migration, or can a system administrator just transparently say, you know what, I want to take nodes 38 through 42 offline and let's move everything that's on there off onto other nodes. Can they be snapshot it up and moved elsewhere? Yeah, what happens with Bproc, that's a good point. Bproc has this process migration sort of undertone where processes can be picked up and migrated from node A to node B. In terms of an application, and this is something popular actually with our customers, is they'll write sort of a checkpoint, a checkpointing that'll catch a SIG user one or catch a signal that will actually write its status file, close its open file descriptors, and pick up and move to another node and then continue running. And so there's a checkpoint migrate functionality available. And users and even programmers and developers can access our API for doing this sort of thing. It's really powerful for an administrator and a user where large jobs that run for a month on end can't be interrupted. So these sort of checkpoint migrate functionality can be implemented in the application. And then, yeah, administrators can say, I'm closing nodes 10 through 20 to upgrade the memory and all the applications checkpoint and migrate off to another node. Which is, I think, a pretty cool feature and I think it's pretty unique to our system. Okay, so what's coming into futures? Are there any new features that you guys are planning soon for skilled or Bproc? So for supercomputing, we're looking forward to showing off our skilled hybrid. And skilled hybrid is essentially a mechanism to manage provision and operate on a mixture of different operating systems. So I could say submit a job that says, I need eight slests or SUSE nodes and skilled will go out and find eight nodes, boot them or install them into SUSE, run the job and then, you know, reboot the hardware back to sort of a traditional skilled state. A lot of other people are trying to do this, but they're doing it with virtual machines. And from a performance perspective, I feel and I think we at Penguin feel that that's just not good enough sometimes. And so the goal of skilled hybrid is to be able to do this deployment and management on bare metal and also integrate it with all the middleware. So for example, if an application wants to run on three Red Hat nodes and then five SUSE nodes, well, the open MPI implementation that we ship will be able to start jobs up with their respective manners depending on the OS at hand. So there's a whole extra step of integration, you know, provisioning is just one of the problems. And skill hybrid, the goal is to not only solve the provisioning problem, but also sort of the day-to-day runtime nature of the problem as well. Okay, so actually that's quite fascinating. We've heard other people talk about using virtual machines. You know, that's typically the most common way you hear people talk about VMs and HPC is just for provisioning and like, oh, my job needs this particular OS and whatnot. So you're doing it just through your tiny NIDRD and tiny kernel.  That's right, that's right. We've been able to do the virtual machine deployment, you know, with VMware and Zen for a number of years now. We've kind of gotten stuck with the fact that, you know, the technology doesn't move forward. For example, this node has, say, a GP-GPU attached to it and an InfiniBand controller. How do I access that through a virtual machine? Well, it turns out it's really not that easy. Or if you can, the performance is abysmal. So our goal is to, you know, get away from the virtualization. We kind of see that as like last year's fun or two years ago. You know, that was cool. But now we want to be able to maintain this heterogeneity on bare metal. And if you can do it on bare metal, then you can take advantage of all the other technologies that have evolved around HPC.  That's awesome.  But, you know, check out the ping latencies between two nodes on Amazon. It's atrocious.  Some people say, yeah, yeah, it's only 10%, maybe or 15%. But that can go a long way for some applications. All right. So how about this, though? If I have, you know, Red Hat running on one node and Susie running on another node, what happens to your unified process space and things? Does that still work or have you effectively divided up into a subcluster or what happens there? The unified process space at that level sort of falls out for those special nodes. You lose that value add with, you no longer get B proc on that remote node or that special Susie node, let's say, but you will get things like, you know, the centralized management, the centralized monitoring interface. We will make attempts and we're working on it now where we're working on sort of leveraging that library caching mechanism to these remote nodes where it makes sense. Surely if I have a fully installed node here, I don't need to cache things like LibM or LibP threads, but maybe, you know, I need a different version of FFTW on the Susie node versus the Red Hat node. And so we want to try to handle, you know, that level of integration. Cool. Now, did you tie this all in with your, the version of Torq that you're shipping as well so I can just, you know, Q sub dash Susie and the right magic happens from a user perspective? That's the goal, yes. The version of Torq we ship isn't really, there's nothing really magical. We don't necessarily, you know, ship a specialized version of Torq, although, you know, the Torq needs some patches and some things that we found work well most of the time. And so we roll those patches as well as we give them to the community, but we roll them in for our customers as well. But the goal is to have it all integrated with Torq and Moab, and also our new graphical framework is called Skilled IMF, which is a sort of a WED 2.0 point and click interface to managing and moving nodes in and out of hybrid mode or back to native mode and killing jobs and starting jobs. And it's just sort of a web-based management framework for the user and the administrator. Okay, let me jump back a little bit in the interview here. There was something you said earlier that I wanted to ask about and now seems as good a time as any. You said you do sell to, you know, non-HPC kind of clusters and things that are, well, maybe non-traditional HPC, you know, where people are actually using clusters but not necessarily for science. You mentioned web analytics, for example. How do these customers use your clusters? Are they just using them basically as a pile of boxes and it has a nice administration front-end on it?   Well, I think, you know, just like any other question in HPC, the answer is it depends. We see, you know, some users are using the Skilled cluster as sort of a, you know, just a magic box with a bunch of cores on it and they're just firing a lot of serial jobs to it. Some of the web analytics companies these days have written very sophisticated MPI applications that leverage our value of InfiniBand and integrated MPI stacks and so on. So, you know, it just depends really on what the customer is looking for. I think in all cases, though, the customers appreciate that the part that they don't want to worry about the administration side of things is sort of taken care of with Skilled. You know, the provisioning of a node and getting users on the node is sort of an afterthought.  You don't have to worry about what version of, you know, this network driver they're installing because they're getting crappy performance. You know, that's all something that we offer to all the customers, regardless of what type of application or what type of use case the customer may have. So, Josh, you guys recently launched this new thing, too, that I read some headlines about this. Penguin on demand. Well, tell me about it rather than me assuming because I admit I didn't read them all very carefully. So, are you guys launching into the cloud buzzwords or, you know, what is Penguin on demand and how is it cool? Well, Penguin on demand is really designed to be HPC as a service. You know, the hardware is architected and designed to basically be a cluster that we host for you. And this differs from like the Amazon model or, you know, any of these other cloud providers where, you know, you ask for eight nodes and the nodes could be God knows where. The Penguin on demand service is designed to be very high touch. In other words, our engineers work with the end customers to get their applications compiled and built and running on pod, but also takes advantage of things like, you know, high memory per core. Our initial rollout had 32 gigabytes of memory per node and also include things like Infiniband. You can't get Infiniband on Amazon or any of the other cloud providers and also offer things like high performance file systems. We've leveraged our fabulous relationship with Panassas for them to provide a large number of, a large amount of space really in a parallel sense, you know, running on a parallel file system. So we call this HPC on demand or HPC as a service. Whereas I think Amazon and the other guys traditionally think of it as hardware as a service and we really wanted to shy away from that and really provide a solution for customers rather than just bare metal. We'll have a demo. We'll be talking more about Penguin on demand at our booth at supercomputing. I encourage everybody to come by. I think we are booth number 9-11 this year. I will be on hand to answer questions. We'll have fuzzy penguins to hand out and it should be a good time. It's all about the chach keys at supercomputing. It's very important to have good keys.  And we have people that come back year after year looking for these penguins. I think it's got to be the best swag I've seen at the show in a long time. Excellent. Hey Josh, so I'm going to go ahead and wrap this up now. But this was interesting, the B-Proc thing, the unified process space is kind of cool and just being able to say kill anything or view basically memory usage across everything is quite neat and to not have to learn a new set of commands. Yeah, and I'll tell you there's a, we offer a free 45-day evaluation of skilled clusterware the whole shebang you can download and install and sort of play with it on your own cluster. And we even offer technical support to the customers that download and want to try it as well. So if it's something that's interesting, you know, feel free to go out and give it a shot. You can check it out at penguincomputing.com. Okay, no problem. Thanks a lot. Appreciate your time. Thanks for coming, Josh. Yeah, and thank you guys for having me.