So, good afternoon everybody. Welcome back. It is time to start with the second part of the first lecture by Deepak Da. Okay. So, let us start. So, in the first part of this today's lecture, I had discussed the classical mechanical tongs gas and the gas with nearest neighbor coupling which is called the Taghashi gas. And now we would like to discuss what happens to this problem if you introduce quantum mechanics. So, I call it quantum mechanical tongs gas. So, the Hamiltonian is equal to this kinetic energy V of x. So, the key point which is the difference from the previous problem is that now we cannot separate away the momenta. Momenta and positions are coupled by quantum mechanics by the uncertainty principle or you know you cannot I cannot just do this problem. I have to put p square. Okay. So, we will try to do this. We will take the same old system in which you have a total system of length L and you will have n particles which are looking like this of width sigma. And now each of these is a quantum mechanical particle. And we want to study this system. I guess in the beginning we just want to find the eigenstates of energy. Diagonalize the Hamiltonian. Okay. Once we have diagonalized the Hamiltonian, we will sum up over all the energy states that will give the partition function. And then if you have energy left, you can try to calculate the correlation functions of this system. Okay. So, this problem is actually quite hard. And let us discuss n equal to 1 case. That is actually rather easy and I can do this one here. So, there is a single particle and it moves in a potential. The potential is provided by its interaction with the walls. So, actually it can only move from sigma by 2 to L minus sigma by 2. Okay. So, there is a wave function psi of x minus d square h cross is set equal to 1 m psi equal to e psi and e psi of x. And x lies between sigma by 2 and L minus sigma by 2. And psi of x equal to 0 at boundaries of this interval. So, that is well studied and well known. All the answers are known. It is given in sheaf. The nth energy level has energy n squared h cross squared by 2 m. I think there is a pi squared. And then there is a L minus sigma squared. Okay. And the corresponding wave function equal to sine. We will also call it k n pi x minus sigma by 2 normalization k n squared h cross squared by 2 m equal to n squared h cross squared equal to n squared. Okay. Pi. Pi should be absorbed in the k n. I think this looks good. Okay. So, one particle problem has been solved. That is rather straight forward. And now we will try to do the n equal to 2 k's which as you will see is already quite complicated. Okay. So, n equal to 2. What is my state space? The same as what we discussed last time. It looks like this. The forbidden region x 1. This is x 2. This is allowed. Then there is all these are forbidden outside. Okay. Again, there is a wave function psi of x 1 x 2. And it satisfies minus t squared by dx 1 squared 2 square by dx 2 square 2 m sorry e times psi x 1 x 2. And the wave function has to vanish inside these regions where it is forbidden. And so it should also vanish at the boundaries of the regions. Okay. Bigger psi 1 x 2 equal to 0 at boundaries. So, can I solve this problem? So, the first thing to notice is that the solution here will always be degenerate because the wave function can be here or here. But I can certainly construct a wave function which is psi 1 of x 1 x 2 which is nonzero in top triangle and 0 in the lower triangle. And of course, psi 2 can be done with the way you know opposite. These two are linearly clearly linearly independent solutions because they do not even have any overlap. Okay. So, a general solution can be a linear combination of these ones. I should solve the problem in one triangle. Then I am kind of happy. Okay. So, give or take a little bit the triangle looks like this. And I want to solve the del squared phi Laplace's equation in this triangle with the wave boundary condition 0 at the boundary. Okay. Now, it turns out that this is sort of a generally a hard problem. There are a lot of people who spend a lot of time studying the Billiard's problem which is you take some funny boundary shape and you discuss the solutions of the Laplace's equation on with that boundary. Okay. All the different Eigen functions. And they show very complicated and difficult behavior. We are not worried about the most general boundary condition.  For the sake of my discussion just now it is useful to consider m 1 m 2 here. m 1 m 2 are two masses of two different particles. Right. In classical statistical mechanics m 1 m 2 did not show up in the tungsten gas problem because they came in the part of the momentum integral. And it was you know it was a multiplicative factor which was thrown out in the configurational integral m 1 m 2 were not there. In the quantum mechanical problem m 1 m 2 have to be there because they determine a coefficient between these two. You cannot get rid of them if they are not equal that problem is not the same as if they are equal. You can scale m 2 you can scale x 2 to make m 1 m 2 equal. But then the triangle becomes a right angle triangle but not an isosceles triangle. Now it turns out that this problem when the triangle is unequal in size cannot be solved in closed form easily. Even this simple problem has not been solved in closed form. Of course there are solutions they exist you give numerically you can determine them. But there is no simple expression for E n as a function of n and the corresponding wave function psi n. They cannot be written in terms of are not explicitly known. So what is meant by explicitly known these things this is this is an important point. We have defined the problem it is well defined psi n is the nth eigen function of this particular shape. So that is the definition it is well defined. Why is that not good enough for everybody? No some people are much happier if psi n can be written as an elliptic function of x 1 times some other function of x 2 and some complicated integral thereof. So that is called an explicit representation of psi. But just saying that psi is the nth eigen function of this shape is that not an explicit representation of psi. I think it is equally good in fact it is not known that you can write this for general shape even as as simple as shape as a square. It turns out that if there is these two sides a and b are rational multiples of each other then in some lucky cases you can determine the solution. But some of the solution is not all of them you can write down an eigen function which will actually work. We will write down some of them and but they do not exhaust all the eigen functions of the system. So that is a problem. But the case where a equal to b is a lucky case and there we can get actually all the eigen functions exactly. So that is what we do next sorry there is a explicitly known for all n for all a by b. So we take the case a equal to b and then I construct a solution. So the answer comes through some trick which of course is hard to realize but it is easy to teach like I mean if I were to discover it on my own it will be very hard I may not get it for 5 years. But if I tell you now then in 5 minutes you know the answer. So the trick is the following we want to solve the problem in this triangle we do not do that we solve it in the full square which is obtained by reflecting the triangle above its diagonal. And then choose the functions which are odd under reflection in this all the functions of psi which I can write down there are finite set of eigen functions which actually have the symmetry that they are odd under this thing ok. So then I work with them that is the answer. So a equal to b the solution is psi nn is equal to sin m pi x1 over a sin n pi x2 over a this is the solution on the full square. But I just make an anti-symmetric combination n pi x1 over a sin m pi x2 over a and I can normalize it ok. If I look at just the first term this is an eigen function of del squared x1 this is an eigen function of del squared x2 the product is an eigen function of del squared x1 plus x2 the eigen value is m squared plus n squared up to multiplicative constants the same thing is true for this one with the same eigen value ok. So if I make this combination then of course the linear combination also is an eigen function with the same eigen value. But this one has the additional property that if x1 equal to x2 it has to vanish. So on this line sorry I just flipped my directions on the line x1 equal to x to the wave function vanishes ok. Then I can go back to my original problem by introducing the shifts you know by sigma and so on that is not a problem that I will not write down the solutions here. Is that clear? I have once I have solved the problem on this equilateral sorry right triangle isocellist right triangle then I solved the problem for two particles with equal masses ok. So that is the solution. But this is odd under symmetry. So I will write here for x1 x2 belonging to top triangle and the corresponding eigen value is emn is equal to n squared plus n squared times some constant. I can write down these constants but I am deliberately not writing them because it is not necessary for understanding what is going on. If you get written notes of these lectures which I will actually not write. But I could have written then all the factors of h cross and m and pi will be there. You can provide them yourself ok. So we are discussing the fact that this solution psi 1 as written here here and psi 2 are both of them valid quantum mechanical solutions. If you start with the wave function which is nonzero here some wave function and evolve it in time using Schrodinger's equation. It will remain in that triangle it cannot escape out of there and the solution in the second will remain 0 ok. So that is just saying that you can exchange. So there are these sectors x1 less than x2 is a dynamical sector in which the particle stays. If you put it initially there it remains there throughout its life and if you choose the other one then the system will stay in that sector throughout and these two are not connected to each other. So if you want you can ignore one of them. If you want I can say that oh the particles are identical then I cannot say which one is one which one is two indistinguishable particles. Then I can say by convention in this case it is correct that psi x1 x2 will be equal to psi x2 x1 we may choose. So under exchange we can since in this case the two sectors do not communicate with each other. I am allowed to do whatever I like and in particular I can choose them to be psi x1 x2 equal to psi x2 x1 then the wave function is symmetric under exchange of particles and these particles are called Bosch ones vice versa. If the particles are Bosch particles then the wave function has to be symmetric. All the other hand I can choose psi x1 x2 equal to minus sign and then I will say these particles are fermions. And the particles perhaps in this case do not care and they will remain the same whether you call them Bosch ones or fermions because according to the interaction given they actually cannot exchange places. So what happens when they exchange places you know will the wave function change sign or not change sign is a known question that is an important point to realize. A more detailed way of saying this is that in one dimension the Bosch ones and fermions are the same thing there is no distinction between them if they are hardcore. So now we come to the case n equal to 3 and luckily that is rather easy. So 4 and less than x2. So now it is actually fairly straightforward. The equation I will write will be d squared x1 squared minus d squared x2 squared minus d squared x3 squared sorry minus psi x1 x2 x3 xn equal to e times psi x1 x2 xn. The phase space configuration space will be a generalized pyramid you know you will have directions x1 x2 x3 x4 each of them is positive and there is a condition x1 plus x2 plus x3 ok.  Non-trivial to think of this in the three dimensional space but I encourage you to try to do it then you will see that this is the configuration space and then there are other copies of this which you can put x2 less than x1 will give you the same figure but rotated and put in space you know and if you put 6 of them together I get you get the whole cube. The cube can be decomposed into 6 such figures and we solve for one of them and put the wave function 0 at the boundary and the way to do it is to put psi of x1 x2 xn equal to determinant of sine given x1 sine k2 x2 sine kn xn with kixi pi kl equal to ni pi. Put all ni distinct otherwise you would not get a non-trivial wave function and that is like the fermion condition ok. So that all the ni should be distinct and then this provides the solution. And if you want Boson wave function psi for me is equal to free particle wave function. Sorry, yeah it is a square matrix n by n. Sir what? Ah it is sin k 1 x 2 sorry I beg your pardon I wrote it wrong this is k 1 x 1 k 1 x n thank you k 2 x 1 sin k 2 x 2 this is the so called slater determinant sin k 2 x n like that. So, sin k i x j would be the generic term yes that is correct you are right. So, in going from here to here the change from x to delta was made ok. And so psi bose is equal to mod value of psi for me. So, if you go to a different sector the Boson wave function will be symmetrical ok. So, now, so what have we learnt with this exercise? We can determine all the Eigen functions of the quantum mechanical tongs case with n particles if the masses are equal we just did it. Now, what is the partition function? Partition function is the same as the partition function of an ideal firming gas which you have studied in your BSE because the energy levels are the same. Each allowed energy level of an ideal firming gas gives you an energy level for this system. So, did I work out the partition function of an ideal firming gas? No, did not. Usually what people do is they say that we will work in the grand canonical ensemble and calculate the grand partition function that is easier to do. And then if you really want to do it you know you take the nth coefficient by doing some contour integral, but it is not done or it again the final answer cannot always be written in closed form for the partition function. For the grand partition function there is a nicer simple product form representation ok, but that is not my concern now. So, this issue about different masses behaving differently than equal masses is sort of an interesting question. And there have been lot of studies in literature you take a one dimensional line put particles with different masses m 1 m 2 m randomly selected you know you can take two masses, but each mass can take two possible values sorry you can take particles where each particle has one of the two possible masses and you make a line and then give them some initial velocity and let them collide and see what happens. Turns out all kinds of crazy things happen and people keep on studying this for long time and they cannot figure out. All the masses are equal then the behavior is kind of very simple. When two collisions occur the velocities are exchanged and so that problem can be studied or understood. When the masses are unequal that does not work and so all this stuff will not work if the masses are unequal ok. So, I do not know what happens to the tombs gas with unequal masses. Somebody might guess that no, no, no all these equivalences to free fermions etcetera are going to be valid still, but that is the hope it does not seem to explicitly work out ok. So, I only indicate the fact that a that different masses makes the problem much more complicated in quantum mechanics and secondly the connection between the equilibrium stat mac and these dynamical simulations in which particles collide and see what happens at long times is also an issue because you know whether equilibrium is reached or not reached becomes a question. If you have funny boundary conditions like this part is at a higher temperature then this part then all kinds of extra stuff happens which I will not discuss, but yes yeah no say it is correct I do not know what symmetry is broken yeah the masses are not equal. So, the symmetry between two particles is broken yeah ok. So, you know I do not know what is the consequence of which one which comes first the symmetry breaking comes first what the name you invent a symmetry when it is there when it is not there then you first imagine some symmetry and then it is broken. When I stated the problem to begin with there was no symmetry then it is not there no. So, I think it displays a mindset by saying some symmetry is broken because it was not there ok. So, now I still have some time and so what should I do in the remaining time. What we will do is to calculate some correlation functions of the tongs gas yes yes yes yeah no. So, the problem was well defined there will be the original problem was a equal to b you know that was how the problem was posed, but the coefficient d squared by dx 1 squared has a different coefficient than d squared by dx 2 squared which you can rescale to convert to a problem in which the sides are unequal length, but the Laplace operator is you know with d squared by dx 1 squared plus d squared by dx 2 squared with equal coefficients and I am saying that that problem cannot be solved completely in closed form already even that stupid simple problem with two particles right triangle what is the wave function I cannot do it in closed form ok. So, now so incidentally once I have said that the partition function is the same as that of free ideal gas I can take the thermodynamic limit I can see how the specific heat varies at low temperature for ideal Fermi gas in one dimension anybody remembers the answer linearly yeah yeah yeah. So, in all these systems the Fermi systems there is a Fermi liquid theory and this stuff will satisfy the Fermi liquid theory assumptions and everything will vary as predicted calculated by Fermi ok alright so let me again define the problem. So, we have the system of size l there are n particles of size sigma and what I like to calculate is the mutual correlation between these positions of these particles. So, there is this stuff which is called the two point correlation function rho 2 r 1 r 2 equal to r 1 x 1 x 2 x 1 dx 2 is equal to probability of finding particle t equal between x 1 and x 1 plus dx 1 and 1 particle between maybe r 1 r 2 was better x and y because we have used x 1 x 2 as the coordinates of the particles and these are not the coordinates of the particle x plus dx and y and y plus dy. So, I take this line I pick a point x and pick a point y and make a tiny window around this of with dx dy and I guess you know what is the probability that the center of one of the particles will lie in this window and another here and I do not care where other particles are ok. The key point is that I am only asking what the probability of finding some one particle here and one particle here and I do not care what happens to the rest ok. So, these I hope to compute, but without calculating it can I sort of guess what this function does rho 2 of x y will be equal to rho 2 of x minus y away from boundaries or large n ok. So, I take the limit of system size becoming very large and the density remains finite then I ask this question then the answer only depends on the distance between x and y it does not depend on the precise position of where I take x and y because we will have translational invariance in the system. Maybe I have I have not established I have not shown this, but is this convincing enough for all of you. This argument that the two point correlation function will only depend on the distance between the points. So, long as the points are away from the boundary is a good behaviour condition which we expect will hold unless something awful happens and in my system that awful thing is not happening it is a good system. So, this will hold. So, this is a function of single variable and what is that can I plot this function rho 2 as a function of x minus y mod space yeah ok space. Well, let us write x minus y. So, at 0 there is a little bit of a problem because same particle may come twice no. So, let us not worry about 0, but what is the probability that very close to 0 here what is rho 2 when x minus y is very tiny let us say sigma by 10 0 precisely all the way up to distance equal to sigma this is going to be 0 and of course, the same thing on this side then it will have some non-zero value it will do something when x minus y is very large. We expect this to become equal to rho squared which is the mean you know each these particles will be independent and it tends to rho squared which is the mean density square again I have not proved this one. But is this obvious or you do not need a proof you will prove it yourself going after going home or some such thing or we do not need a proof we believe it anyway ok. So, very good known. So, now I want to sort of be able to guess the general structure of this without doing the calculation and then maybe one will actually do the calculation later. So, it turns out we will show this eventually, but you know even if you do not follow the details of the algebra later this point should be clear we will find that rho has a very interesting structure it is 0 up to here then it shoots up to a very large ish value and then it kind of goes down like this and then shoots up and goes down like this and then it goes down maybe and tends to an asymptotic value where why does this bump occur precisely at sigma because you know if you have put n particles. So, the mean density is rho, but you have removed particles cannot be here. So, the n minus 1 particles have to be in the rest. So, the some bumps have to occur, but suppose there is a particle here then again in very close to it other than particle cannot occur and. So, there is a dip and. So, this argument very qualitative works even in 3 dimensional heart spheres. So, the correlation function in 3D will have roughly this behavior not be exactly this, but of this sort which is that there is a region where it is 0 then there is a peak which is called the first shell and then there is a dip and there is a second shell and there is a dip and there is a third shell, fourth shell and after four shells you cannot see much because it becomes the this function is different from 0, but not very different from. So, the correlation function is different from rho square, but not very different it kind of asymptotes to a constant value. Now, this phenomenology can be qualitatively understood in as many words as I said just now, but the advantage of the Tung's guess is that we can actually calculate this function rho 2 exactly in this problem which you cannot do in most other problems in Stake Mac ok. Even for the 2D Ising model rho 2 cannot be calculated not in this way because you know Ising model can only be solved at h equal to 0 and we need we need to work with fine variable density and stuff like this. So, it will be an exaggeration, but let me make it. It will be only a finite number of cases are known where you can actually calculate the correlation function exactly this rho 2 and so, this is one of them. I am sure you can come up to me and say there is another one, yeah there are some more, but this is one of the few cases which can be done and it can be done in 15 minutes. So, so that is an extra advantage ok. So, I will calculate the correlation function for this rho 2 in the next few minutes 15 yes. So, there is the mean spacing between particles. So, they actually correspond only to sigma, but the healing length corresponds to rho and the rate at which these things die down like this or like this they correspond to the mean density ok. So, I want to calculate this function probability there is a particle between a plus d r even. So, how is this different from this? I have shifted the origin that is one thing I have done and this is a conditional probability. So, there is it is divides by rho when r goes to infinity this function will go to rho while the other function went to rho squared ok is this point clear ok. So, now equal to what this is what we want to determine the key to the solution is the fact that the variables delta i are nearly uncorrelated variables and in the thermodynamic limit they become uncorrelated variables ok. Had another way in the constant pressure ensemble which is we introduced last time they are independent variables. So, I will study this problem in the constant pressure ensemble where life is easier. So, in constant pressure ensemble delta i are independent i i d ok. Now, what is the distribution of delta i? Because they are independent variables in our problem they it was just I forgot the answer somebody can help me with this what is the distribution of delta i? It is a so probability of delta i I guess looks like this it is it is shifted exponential it is 0 up to something and after that there is an exponential minus beta p delta i which is in the distribution function. So, it is e to the power minus beta p delta i minus sigma that is the function and there is some value here and there is some normalization I will not do ok. So, so this quantity is equal to well suppose there is in our problem there is one particle here and I am asking what the probability that the second particle is here then the answer is 0 you cannot have it. So, I guess my r should be bigger than sigma otherwise the answer is 0 suppose r is bigger than sigma, but it is less than 2 sigma then they cannot be any other particle between them then delta i is equal to r that is the probably it must be there is nothing else that can happen ok. So, this probability is equal to probability that delta 1 equal to r the full answer good, but this answer is valid only if r lies between sigma into sigma suppose r is a little bit bigger bigger than 2 sigma then it is possible that delta 1 equal to r, but it is possible delta 1 is not bigger than r n, but there is one more particle in there ok. So, I should add to this plus probability that delta 1 plus delta 2 equal to r minus sigma probability that delta 1 plus delta 2 plus delta 3 is equal to r minus 2 sigma and terms like that ok. So, the first one is very simple that was this result which we drew it is a something like this. What is this thing delta 1 and delta 2 both are exponentially distributed distributions what is the distribution of delta 1 plus delta 2. So, that is easy that is if x and y have distribution which is p of x is equal to e to the power minus x then p of x plus y is equal to x e to the power minus x is x plus y equal to alpha is equal to alpha e to the power minus alpha because you take two exponentials and you have to integrate along the diagonal that gives you one factor x, sorry x and y this is the function is e to the power minus x minus y and the positive quadrant then I want to determine the probability that x plus y equal to a given number then I have to integrate along this line, but along this line this function is a constant. So, the length of the line comes which gives you a factor proportional to the value this is x plus y equal to s. So, there is an extra s which comes here and that gives me the term. So, this looks like s e to the power minus s and this term by the same argument now there are three terms, but they have to add up to r minus 2 sigma then I have to do this in three dimensions. Same thing, but now the cut is two dimensional is proportional to s square. So, the answer will be s square e to the power minus s give or takes a little bit.   So, I do not want to do it here if I concentrated on fixing all the factors in all these formulas then the key point would not have been conveyed ok. So, I will stop here.


Morning everyone. Welcome to this early session. So we are going to talk about today, and now I'll talk about building embedded Debian images with ESA. So our talk will give motivation for this work, obviously, and introduce what ESA is. We have a short demo part as well. And we would like to talk about where we are moving to, what needs to be done, and primarily we're interested in having a discussion around our approach, and ideas, and then comments on this. So first of all, who are we to here? So my name is Bojan. I work at Iberia 3 in the high in Munich, Germany. I am a software developer for Linux and other systems. We do drivers, applications, virtualization. And for fun, I am a Linux user since 1 to 13, and Debian user since 4.0. And I like low-level stuff. Yeah, my name is Jan Giska. I'm working now for 10 years at Siemens Cop Technology. They are doing in-house consultancy and development for meta Linux. So kind of proud to say that I haven't written any significant line of non-free software in just 10 years. Most involved in abstract activities regarding kernel, virtualization, real-time. And these days also about base systems, distributions. For fun, I'm doing Linux now for 20 years. Start the hard way, low-level stuff as well. And yeah, I also enjoyed hacking on this hardware, those two hardware things. So Linux is being used in industrial scenarios, embedded as the scenarios for quite a while. So I don't want to go into all these details, but we are, as companies, Siemens, we're active in all these areas, and also as a community here of industrial users, we are interested in all these kinds of devices. They are running various Linux versions, various Linux variants, and, well, for example, on the left column, the device in the middle, that's actually also running Debian. So these embedded devices, what makes them special compared to, well, never your desktop system or the server, the cloud? Well, first of all, there's a variety of hardware, even more than we have on standard systems. But that alone doesn't make it special. It's also about the customization that we apply on these systems, for example, to customize the boot processes for various reasons. We have special kernels still, although we are all happy to use upstream kernels, but there's always some need to adjust things. Optimizations are being done oftenly during to optimize the size or to reduce the number of packages for these devices or to optimize the speed of certain things further. And that leads sometimes also to deviating packages, so you can't take identically what is from upstream distributes available. And last but not least, hardware is always special and you have to do some interesting tricks to make it work in practice, so there's also need for performance. What also makes it special in our domain is the long-living, the long life of the system that we deploy. Most of the devices we support start with 10 years of support.  If you're building a power plant, you don't throw it away after a couple of years because the software is no longer up to date. Of course, the devices themselves, they change over the lifetime of these long-living systems, but the core you want to keep for a long time and yeah, that means special measures to maintain them at all. At the same time, if you look at the better device today, they're becoming more and more the same like we have in the enterprise world as well. There's a desicion of hardware going on. So for us, for example, it's mostly about x86 and ARM these days, modern ARM architectures. The ARM becomes frequently more powerful, so it's no longer an issue. The optimization need goes down a little bit. You have more CPU power, you have more RAM, you have more storage available. At the same time, we've been asked to introduce more standard features into the very devices as well. Well, connectivity is a big topic, IoT. That comes, of course, with the need for even more security, more hardening on that. We have a high demand for new modern languages on these devices, and last but not least, you have all these nice little visualization and container features these days that should be also available just like the same on the better devices. So now if you have a better device, the question for us is always what to get to a bootable image. It's about generating an image that runs in the end on the device and breaks it up. You're not installing, we are getting for this image. So the standard answer today to get to a bootable image is to build yourself. I mean, from scratch. This is what these two projects are doing, and there are way more than that, who basically create their own distribution sort of source on demand with all the customizations included. But the question is, do we really need to do 100% from scratch generation just because we need 1% of customization, maybe? And if you look at the software stack, that is something like that, maybe it's 10%, maybe it's even less than 1% depending on the device. But actually, you are paying quite a bit if you're just doing it for this customization. It's completely rebuilt. Why don't you just get in for it? So why specifically getting it? First of all, it's a well-tested major binary package feed for us, which is very interesting, and it has broad hardware support, even though we need less hardware these days, but still. It scales up and down, means we can apply the same thing on the high end devices, like on the lower devices. It's long-term maintained, which is very important. It's still not that long-term maintained that we need it, but it's on the right way. Which also is very interesting for us because we are then shipping those devices. We become a distributor of free software. And therefore the licenses that have to be, that are included in the packages, need to be very well-documented. And that's something which is also high importance for Debian. So we have the same page on this. And yeah, we have history already using Debian for products, so it's just about extending it a bit more. And that's where our requirements come in. How do we want to build these images for our devices? As I said, we need ready-to-run images. There is basically no installation. This is just about getting the image that has to be flashed as is on the device at the end. They need to be reproducible, at least on the higher image level. But of course, we are also very excited about learning the great progress on binary reproducible, the package side as well. And we're not just shipping individual devices. We are often shipping also a large variety of deviations of these devices. So there's a lot of commodity in how we describe the images for our systems. And that needs to be accompanied by the tool that we need for the image generation that we can layer things. Like it's seen on the right side and that there are common features that many devices get and there are some specializations and you don't want to describe one for each and every device, everything from scratch. And ILE, as quite a few of us will still use also the out-of-source tree approach that's industry standard. There should be a smooth way for developers, for engineers to move between both worlds. So we want to share the concept and the artifacts ideally with the out-of-source tree systems. Yeah, that's where our integrated system for automated root file system generation comes into play, so what is that? First of all, either uses the well-known, basically, embedded automation tool called BitBag. This tool combines the build of one where it's still needed and there are some cases of packages of new packages, custom packages. The bootstraping of the system and the customizations in one thing. It supports that layering concept by being able to append, repend, overwrite, replace descriptions of your configuration depending on where you are on the layout stack. It also parallelizes the build of multiple targets. Wow, which is important if you don't just build for one device. So either, why is that strange acronym on the page side, either is also a well-known place for people who have been evacueing back in Munich and the river side, so either is a river in Munich, so that's where the name comes from. So how does it work? We first of all have the upstream depth repositories, and from that, this is for the case of an ARM device. We create a built-in root environment. So while in this environment, we then can generate our custom packages, usually the so-called business logic, so your own proprietary software on top of that. That is done in the second step, and you get a normal Damian package on top of that. But of course, nothing which you can upload. Furthermore, you then create the rule file system itself. That's normal means of LibuStrip or multiple hour case right now. And you also add further artifacts like the bootloader or the kernel, either out of sources or out of existing Damian packages. And you install your custom packages on top of that, and you have your bootable system for the better device. Currently, we primarily focus on native compilation because that is the standard way of doing things. And well, we don't want to deviate much from the standard because it creates also interesting effects, but of course, it's interesting that Damian is moving towards cross-building again, so that will be beneficial for us as well. Right now, builds are being done within the free user space environment. Then of course, it's not very fast if you have to build a lot, but it's well, we have powerful machines, it's still okay. And that built-in root we have up there is also our SDK for developers to write applications on so Dev Shell SDK combination. So with that, I would hand over to Aran to give some demo and further insights. Okay, thank you. We need to switch the device unfortunately. So first of all, a question. Who has already worked with Yachto? Oh, yeah, okay, quite a number. Let me do a short introduction. Nothing seen yet. Yes, now it's better. Or a short tour of how ISAR works. So I'm currently in the ISAR directory and we see here a major director structure like BitBake is the tool that executes our customization scripts. Basically Debian provides built infrastructure and also artifacts that come out of it and all the tools that needed to generate the file system and the target image. However, we need some glue tool for scripting and we would like to do this in a maintainable and structured way. So we have started with playing shell scripts but as you see, we needed to handle product lines with multiple products that share code but have also differences from the very beginning and these shell scripts become, of course, quickly un-maintainable. So BitBake, what BitBake does for us, it provides needs for us to structure individual tasks in our recipes which are then executed in certain order according to their dependencies. So what we see here is also a meta directory. It's ISAR core which provides the core functionality used by all projects and meta ISAR is a template that you can copy to your project and customize and develop your own product. So to give you an idea how it looks like, so we have here the tasks that are packed in so-called recipes and they have the extension in BitBake they have the extension bb for BitBake and we can see here four recipes. Basically what we have is buildtrude is the recipe that creates a build change route for us in which we want to build our own packages. Hello is a sample package for our sample product and ISAR image base specifies how to build the complete image. So to give you a feeling how a recipe could look like, we can look at the buildtrude. So what we see is that a number of variable assignments, some of them have some predefined meaning like description or license product package version and so on some have more local meaning that they're used later in this or other recipes. So here we see for example the build change route pre-install variable that will be used later to install certain packages in the build change route. Also it allows to define so-called tasks. They look pretty much like shell functions and they can be implemented either in shell or in Python. And the rest of the recipe looks very much like shell but strictly speaking it is not shell, it is bit-backed image. So what we see here basically the do-build is the standard task of a recipe that is defined by bit-back and what it does after some preparations it bootstraps a build environment for building our own packages. So regarding packages, we have our hello package and what we see here we have the source URI, I changed it to local file in case that network doesn't work and we see that it doesn't define any actual tasks and this is intentional because we want to build several different packages and building them is actually all we're calling DPKG build package. So this whole logic is hidden in the DPKG class which is then shared by all package recipes. Classes in bit-bake have the extension bit-bin class and we can have a look at the DPKG one and we see here a number of tasks that implement certain things like fetching is cloning of the source tree into a local file download directory and pack puts these sources, cloning these sources into the build truth. Of course build truth has to already exist in at this moment of time and this is specified in the depends variable and the whole thing is then to build the package which happens in the do-build task and build SH script is basically more or less change to the package directory and execute the deep package build package. So to build actually an image, I have to prepare the execution environment to set path and other variables named by bit-bake and specify a build directory. So now it creates a build directory with some defaults. They can be customized or in our case we can use it as is and we can then create our base image for a D64 architecture with stretch. So what we, okay, this was already built previously so I will remove the build directory and start from scratch. Okay, I can take this later and who is interested, I can show the demo after the presentation. What we can see here is we can generate the tree of the tasks to be executed and we can see actually the sequence how this is done. So what we see here, we have build short creation and target image creation that can be done in parallel and after that we have further tasks that are done after that. Slides, yes. To give you an idea of how we structure this into different layers, basically you would like to have repetitive stuff in one place and don't copy paste it to every project. And this picture on the right that you have seen in the previous slides would be implemented in the following way that we layer the respective recipes in directories like Meta Platform which carries the platform packages Meta board for specific board support and concrete products like device A, A prime and B are each in its own directory. And in this way we can combine hardware-dependent parts and hardware-independent parts with product-specific parts and get our final images as a combination of those. These directors, they can be in one repository or in separate repositories. For example, if you have some library or hardware, PSP vendor, they could have their own repository that you could pull manually or using a tool like repo or CAS. And this is the way how we handle the variability. We have the machine layer, distro layer and application layer. And for example, MetaEaser includes a sample implementation for Raspberry Pi and we can see the settings in the respective configuration file. What is here interesting is perhaps the image type. It specifies the partition layout and sizes and so on for this particular device. And the distribution, so we can have products that use different Debian versions. And this is reflected in the distribution configuration file. And of course, any product is using its own applications. This is handled at the image level, so we provide our packages and install them into the image and we can specify it with image install, for example. So as we have seen, we have a directory, we have the application recipe and this is basically where we build our package and this application recipe is later included in image install. This is the way how you add your own applications in EZAR. You basically build the package in the recipe and list it in the image install. So in this way, we can actually provide any kind of customizations like update kernel, update the bootloader, provide some, for example, create additional users and so on. So the goal here is to have a state of the system that is completely described by the list of packages. Of course, these customizations can also be done in an ad hoc manner. This is a trade-off that everyone can do to his taste. My approach is always to package things in an appropriate package so that the package list and version alone specifies what is the intended state of the system and if some files are overwritten, then I can compare them with the package and see how the actual system deviates from what I had intended. And regarding ad hoc customizations, multi-strap that we are currently using is providing a number of hook scripts and setup and configuration scripts and we do this, it's possible to do this in setup script. So a couple of words regarding future work. We would like to do this in more deviant way. So currently what we do is building packages and copy them to their file system and install this with DPKG. Of course, this has the problem that we have to install all runtime dependencies of those packages in advance or later in some way. So the right answer to this is using apt and this is what we have in our development branch. Basically we build our custom packages into an app repository and install this from there. So this is expected to be soon merged into the master. What is next? Internally we use Debian source packages described by .dsc files. This is not yet released. We have to move this from our internal repository to the public one and we rely on sudo for doing debutstrap and other tasks. This works well, however this creates problems in environments where the servers are managed. So we are working on removing this requirement. We have tried to, yeah, I see. Nail shaken his head. Let's discuss this afterwards. But as far as I know, we have a prototype implementation of that. We did try fakehood and it didn't work well for us. So it would be interesting to discuss this. Okay, so Nail says please don't do this and he knows what he's talking about. So I'm glad to have a discussion about this afterwards. So as you have also seen, in the recipes we specify dependencies and if my package build depends on some other library or package in a general way, then I have to specify this in BitBake recipe. However, this is of course a duplication because Debian Control and other files already provide this information. So my ideal solution to this would be to have a BitBake backend that could directly support Debian meta information. Whether this is possible we have to see or to look for other solutions where if there are any suggestions, I will be also glad to brainstorm on that. What we do every day is providing customizations and the current solution to provide a customized version of a package is to clone this package into our own Git repository or create a branch out of that, make our tool line change, create a tag and then use this in a recipe. So this is quite of an overhead for this small task and what we are aiming at is easy source package package patching so that we can provide a source package that is quickly modified and archived in the app repository. And one of the last things is version pinning. So Debian basically provides version pinning and this is good. Our question is here how we can do this in a way that we don't fork the complete distribution and do this in more or less maintainable way. So now. Okay, to summarize I will see a lineup for the questions. So it's very important for us to emphasize that either although you may see it as an embedded BitBake thing, it's Debian, there's a ship there on top but in the core the majority is Debian. It's very important for us. We see it, we're not completely married to this approach and all details, we see it as a tool to increase the sharing of concepts and implementation between communities, between embedded communities and Debian communities. Things tasks like image generation is immediately done independent web images or web binaries for the images come from, optimizations, customizations, these things are to a certain degree shareable and other templates for that. We will, we already look into aspect where we can contribute to Debian where we have common requirements we fulfilled specific on packaging improvements, topics like extra C-flex or cross building is of common interest for us. We will possibly also improve the tools we use. We see deficits currently we don't have a hot list for this but it may come. In the end we are interested in, well if we are using it we want to strengthen also the community to attract more embedded developers to Debian, that's a huge potential right now because a lot of people are not very happy about how these out of source free image generation for when it works and that actually also is visible if you talk to some of our suppliers, the hardware suppliers, they all provide the ecosystem for Yocto and Co. But quite a few of them are also looking to alternatives like Debian and possibly this could be a tool, a mechanism and yeah to open it up for them as well. So with that thank you for your attention and I'm looking forward for the question and your opinions on this. Where do I start? I did all this about seven years ago right the way up from BitBake, right the way through the fake route and that's where the monster came from. The monster was actually part of a system that predates this and which we burned it in fire with a lot of celebration when we finally replaced it. Everything you're actually doing with BitBake can and should be done with standard demo tools. You get build package and that can work with S-Build. So strip directly from your Git source tree or the Debian directory, whichever type or branch you're on get a bit of a build package or get your package from there without pre-building everything. You can use the bootstrap once, you may want to put it on the stretch one for Jesse, one for Buster, one for Sid, leave them alone, stick a lot of falsehoods and don't want them when you need them and then you do the rest of the work on top of that just by running out of the sandwich route. That also allows you to have multiple repositories that you don't need to do multiple stuff all the time. The problem I see with this is, again, what we had in our own version of the same system was that you're actually running multiple stuff too often. You're rebuilding stuff all the time and that harks back to the BitBake methodology. BitBake is built on that kind of principle of rebuilding a lot all the time. Getting rid of pseudo is a red herring. I know there are issues with certain servers when you're trying to do that, but actually what you need to be thinking about is using virtualization on those servers now that you've got a virtual environment and a real boot that is disposable. If something goes wrong, you don't blow away the main system. That's the big problem with multi-strap, that's the big problem with the system we had that tried to run as a low reuse and then use in pseudo. When things go wrong, trust me, they will with this kind of system. When things go wrong, the first thing that happens is that something like multi-strap blows away the et cetera directory on the system it's running on. And the reason it picks et cetera first is because that's what Devin Stoller first creates. So it's the lowest I know. And that's the worst possible way to lose a system. Suddenly you've got all the configuration and trust me, I've rebuilt several systems from that and it's a lot of work. So stick all the building in virtualization. Then you don't need to worry about getting rid of pseudo because you've got a virtual boot, a boot in a container, a boot in a 4VM. And everything does work really nicely inside of it. Basically what I'm recommending is that you follow the way that Devin builds our own packages. So you build natively. Don't worry about cross compilation except when you actually have to rebuild a kernel yourself and this kind of thing. You build in a virtual environment that is safe that we throw away and we try. You build once and you use a repository tool like RepRet Pro or there's numerous ways of building a Devin archive. Don't rely on flat file systems or just your own directories. Use tools, database, a way of handling or proper directories and the release files and sign in the release files and building suites and pools and disks and all this kind of thing that could want all of it on the chain. And you can do all of that even with typed flow down in embedded code. You can get all that working on the chain. We've done that. It just needs a little bit of scripted to type existing bits together. And then at the end you can build your image just by putting these together from, so you build the boot date of the truth once. You add the packages using apt but just by tuning in and specifying the apt source you want and you can have exactly what you need from whichever repository you've got everything organized in, even a local one if you're making a local changes. And then once you've got that you just lump that into an image and you're way enough. So version pinning them becomes trivial because you've got a virtual environment and you've got all the tools to do that inside the tool. The biggest worry I've got for things like ISR is you tend to end up in a situation where you've ruled out the possibility of getting security updates from the visual source and you have to go around and rebuild everything from scratch. So that's a big cost. And it's a worry that that kind of thing because you're potentially customizing one step maybe too far with some of the core packages. It's a worry that I've got a lot more than enough of exactly what you're doing with the lower level packages. But if you take the standard degree strap and then add your own stuff on top you will always be able to add security updates as they come from Debian. And the kinds of long life machines you're actually putting it on. Long term security over the year upgrades that kind of stuff is going to become critical. We can't afford to avoid or just ignore the security problems or the need to keep these in the lake no matter where they are they need one of the updates. So that's, I realize that's a lot and it may sound as if it's pulling everything apart but actually you can still use a lot of the glue that you've got there just by switching out bit by bit putting in something that's virtualized and running S-Build or any other way of building that's from packages from GIMP. Like on 2000. Okay, so thank you for your comments. Let me comment on this. There were quite a number of topics. So the easier one is the virtual machine. So we have this requirement and we will basically we are considering what we do in this direction. This is less a problem because currently change route solves our problem but virtual machine has its advantages definitely. Regarding GIMP build package. Well, we already use DPKG build package because not all of our sources are directly in Git some are packaged in source packages and on tarp. That is why we use this generic tool but this already is done in the Debian way. Regarding rebuilding, so this app branch that I'm talking about, the development branch it already contains the fix for the problem of rebuilding. So our intention is definitely not to do it the Yacht away and rebuild everything every time. It happened to be like this because we used stock Debian and our applications were modified from build to build anyway so we didn't lose any time. However, right now when we are moving more into the infrastructure thing and want to provide some more infrastructure packages to other departments that are going to using this we look that we pack it into the app repository. So this works already in the development branch. This means we build packages, use rep-repro to create the pool and all needed packages, files, et cetera and then install this with app, with whole dependencies and most up-to-date packages. That's a fair way along the half months but the actual big thing, don't worry, means in this case it is actually easier to do it with the experience. Yeah, well, we use BitBake not to do to replace what is already in Debian. This is just a means of structuring the customizations. So we have many customizations and how we started was a huge shell script. This is of course unmaintenable and you see here basically three recipes that provide build system, base system and application. What you don't see here in the real project, this part remains pretty small because Debian handles it perfectly, I would say. This was the reason why we chose Debian in this regard. However, on top of that we have 20, 30, 50 recipes that do certain customizations. So this is seen as a glue tool that allows to structure this because metal airing is something proven in use in other communities and we want to reuse this. We don't want to invent a new tool for that because two years ago Rico already presented a multitude of tools that are available for this. So we are using one of the existing tools and it does its job efficiently. So it can parallelize tasks if they can be parallelized and so on. Yeah, so the recipes type of thing. If you follow through and put all of these into after packages, all your little customizations become dedicated packages, then the recipe just becomes a list of package names and you can extend that further so the list is actually coalesced into a set of what Debian is still called tasks and these are simple meta packages and you can have a meta package with the product name and that has all the dependencies, it can specify versions, you pass that to apt and apt will do all of the dependency resolution for you. So that's something worth thinking about is actually moving the recipes into packages themselves, sticking those packages into the repository and then saying, right, unpack Debian Strap, to root into the root, apt install product A1, done. Yes, yes. This is what we are into. So thank you for your feedback. I think it was really helpful. So coming back to you wanting to get rid of your current use of Pseudo, I kind of agree with Neil that fake-rooted friends are a trap, having tried them for like, flat-pack OS3 things and it's like they work except when they don't. I've had quite good success with somewhat counter-intertively using the virtualization backends for auto-package test, which provides like a fairly abstract interface for doing interactive things in an environment where you're root over there. And that actually works quite well as a way of having like the business logic run as your user normally and then having the bits of the work that need root happen in a virtual machine or a container or whatever. So that might be something that's worth considering. Yeah, thank you. Thank you again everyone for the feedback. I think we've asked them to follow up this question off-site. Thank you.


Yeah, so I'm going to talk about managing fires and leadership's role in incident management. So specifically, won't necessarily touch on what you should do as an individual contributor during incident management, but what an incident commander or leader can do in those situations. So this is me, now you can all see. I am from Connecticut where I work at Cage Data. We do DevOps consulting, helping organizations that want to make a transition but don't know how to do it. We help navigate that change, but I'm also organizing DevOps CT and DevOps Days Hartford. So if you know people on that side of the country, please let them know. And I'm crazy on Twitter because I don't like when people can spell things correctly the first time I make it difficult. So yeah, so little fires are not really the big instance we're talking about. Those, you can handle with tooling. These are the kinds of things that go off in the middle of the night, someone responds to it, and it's usually an individual that can fix it. We've got tools like the context about what's going on. That's really important to fix these small incidents. If you understand just why you're getting this alert or what it means for your company, you can analyze it and make a good decision on how to act. Runbooks are critical, just sort of very simple. If I have a small fire, here's how I put it out, and if I follow these steps, it will go away. So these are critical tools for managing small incidents, making sure they get taken care of quickly. If we've got them clear, we can a lot of times build automation engines out of that as well. So whether it's chat ops or some other automated script that kicks off, some incident happens, some computer can analyze that, respond and fix it without having to wake someone up. That's always good stuff. But eventually, there's some sort of escalation. This is where we also want to have this documented. I believe this escalation process was from brush fires. This is the actual escalation process of what it looks like. It starts out small. There's a fire, but we've got it under control. Don't worry about it. Okay, now we need to look at it, and then now we've got an emergency you need to probably evacuate. You need to take action. So the same is true of our incidents, right?  Maybe it's nothing, it's handled by automation. Maybe we need to get a couple people involved to look at it. Eventually it's going to escalate. They happen. Crises require a team and leadership. If there's a large fire, they don't have one fireman go in. They don't have a few without someone who's an incident commander actually managing that fire and directing people and helping them tell where to go. We want to talk about what are the good qualities of leadership in that. What's their actual role during that incident? In case no one believes that I actually had up my slides the day of, there you go. Transformational leadership. These are great qualities to have. These are really good. These are kind of like peacetime leadership qualities. It's sometimes easy to say, well, yeah, but if there's an incident, how do we manage it? There we go. My job's done. Thanks, Scott. No, but seriously, these are actually really good qualities to talk about. I'm going to try and make them a little bit more practical. Kind of the key points around this are that B column. We have a servant leader who's adaptive, creative, and a global communicator is kind of major in any sort of incident management, being able to share information. Before we actually get to a crisis, you want to know who the leader is. Because if you're in the middle of a crisis and you now have to figure out who the leader is for this incident, you have two crises. Because now you've got an argument on top of it. That's critically important, understanding who the actual points of reference are, who should be making decisions, and not having a question about it. There might be some democratic process for passing that incident commander along. It's going to depend on your organization. But it's crucial that if I say, hey, we've got a major problem and I want to escalate this to a major incident or a major crisis, who's going to take over that control? Kind of the key of that is that whoever is the incident commander should be authorized to make really big business decisions sometimes. Sometimes really big, but always business decisions to clarify that. If your major production database is down, they need to say, hey, can we spend twice as much for at least the next two weeks to light up another copy of it somewhere and analyze it? We're having errors. Can we bring down the production system right now to look at it or reset it? These are decisions that are going to affect bottom line for the business. If they can't make them quickly, just delaying, trying to get approvals, trying to talk to people is potentially going to blow up that incident even worse. If I've got a small error that I can stop everything fixed and bring it back up fine, but it might balloon to corrupted data, I want to be able to bring it down while it's still safe. Part of that also isn't that all this power should rest with one person. You want to try and spread that out as much as possible throughout your team. So if I'm an incident commander, I still want to break up authority over one task to different people. So if I say, hey, I've got big picture, but I want one person to be in charge of database recovery. And they should be able to make those decisions about what database recovery looks like, because now I'm sort of delegating that choice. Choices can happen faster. We can iterate faster on that. And that's kind of, right, this is how we sort of troubleshoot stuff 101. If I have a big problem, I need to start with the smallest components and work through them. It's just how that works, right? First check DNS because it's always DNS, and then work at the next smallest component up the line until you figure out how it fits together. And it might be a connection that's broken, but it could be just this one small piece that you only find out by working incrementally, which is my next slide. Take it incrementally. I mean, that's exactly how you build Legos, right? So you start small and you eventually make something that's more interesting from all these component parts. That's how we build systems, that's how we need to troubleshoot systems. And if we can put people and teams in charge of individual groups to make all those decisions about checking it out, we can kind of, instead of building step 1, 2, 3, 4, maybe we can work on 1, 10, 15, and kind of assemble these all at once. So it just speeds up our actual recovery time. This is a good point too. So this is actually, all of this comes from a story. And this is a great quote from Rahm Emanuel. And the story this comes from is something that we did at KG Data. We had an outage with a client who, making a transition, they had old infrastructure, single server kind of running all of these things, running directory services, running domain control, running file storage, running emails all in one place. It's aging. It's not under warranty, which is terrifying. Software is probably end of life within a week, which is terrifying. So we've got an improvement plan in place and we're working on it, but it's not quite there. You've got to schedule downtime to sort of extract these monolith services because it's Microsoft. So what happens if the server goes down? We just get those alerts, not reporting. So, well, that's not good. We happened to have someone nearby and said, hey, could you go to the client and actually check on this? Like, is this actually down? Because we've got a major issue if it is. He's like, yeah, it's down.  He said, no, it's looping. It's not coming up. It's crashing continually over and over again. We said, great. So we actually, we went in, we took care of it. We spread it out. We had a high availability structure we were building and everything worked really well. But what happened was at a critical point in this, I had a decision to make. We could either work for 16 hours and recover the data and come back to how it was, which would be a recovery. Or I could take 16 hours, push through my migration that I had planned, and come up with something better. So this crisis, this downtime, gave me an opportunity to actually improve their services across the board because I had this great moment to make use of or I could focus my efforts on improvement rather than just getting back to the status quo. So having that leadership, having someone who understands where you're at, where you can go and what you have in place is critical. I mean, I would much rather go through a crisis and come out better on the other side than come out back where I was and now have to work to improve it on top of it. So don't let a good crisis go to waste. So that's part of it too, right? This is what it comes. We could either work on what will make it work and that's not great, right? I can still use the toilet right now, but I'm not really happy about it. So that's part of it and it's good to know that because sometimes you have to implement these changes, right, like if my toilet's backed up and I really need to use it, I would rather have this than nothing, but it's important to know that's not the resting point. My toilet isn't fixed yet. The other differential is what'll make it whole, right? What'll actually restore it back to what it was beforehand or better? What'll actually make it complete again? So that's a distinction that's also there to make in making those decisions in a major crisis about, hey, we need to do this to fix it, but here's our plan afterwards to actually bring it back to a whole service again. So communication was a big thing with this as well and there's a great quote. If you know John Hodgman, you may know him as PC from the Mac and PC commercials or a correspondent from The Daily Show. He's now a fake internet judge where he dispenses fake internet justice, but he's got this great quote. He says, brevity is the soul of wit, but specificity is the soul of narrative and I sort of adapted that because as humans, we communicate in stories. It's what we've been doing for millennia so even when we're trying to coordinate efforts, we're building this narrative of what we're telling people to do, what we're going to do, it's simple stuff. These are the next three steps I'm going to take. That's still a story. So being specific about it and understanding that communication is key to being clear about, hey, I'm going to run a restoration on this system, then I'm going to run these tests, then I'm going to do that. Now I don't have to question what's going on. I don't have to double effort because I said, hey, you're restoring that, did you test it? We just don't have to do all that if we're specific about our communication. And this is how we do it as well. We make work actually visible, so bring it out in the open. Instead of working on something, this is kind of like committing to master, right? Instead of having your own section of work that you're working on and then bringing it back when it's all done, we want to actually work publicly, especially when we're trying to coordinate as a team and get it all completed quickly. Fortunately, we've got really awesome tools for all of this, right? These are things that just exist. Trello is still free, I think, last time I checked, which is an awesome like Kanban board where you can just put everyone's work. Here's what we're working on. You can actually assign it to people, great visibility, or if we have actual chat ops running. If we're actually running our work inside of a chat window, like it becomes really apparent who's working on what. So that's really big stuff that a leader has to coordinate. Sorry, I talked too fast and then I run out of. So yeah, you're setting that example, right? So you're that clear point for communication. It's not always, this isn't all there is, right? If you have just post-it notes and a wall that you can stick them on, like you've got a good way to make your work visible. And sometimes you have to do a conference call and chatting about everything isn't practical, but the key is to make sure everyone's in communication. This is actually the other side of it. We've had a similar incident where we have an outage with the client and there wasn't good communication. The people who were in leadership were actually off-site at the time. So we weren't kind of looped in on what was happening. It didn't get escalated cleanly, but what it came down to was a communication issue. We just didn't know what was going on. So our fears and the reality were just not lining up and our retrospectives after brought that up and this became incredibly important for us for what we value as a company. So getting to servant leadership. Leaders are handling those human interaction problems on this. That isn't to say that your incident commanders can't be technical or can't contribute. It's just that your role as a commander of that situation is to coordinate all of the group efforts. It's not necessarily to come up with the fix that makes you the hero. It's to actually coordinate everybody. Machines are great because they run on electricity and they can run all the time, but as meat bags, we run on carbon and water. So make sure that issue is handled as a leader. This sounds really dumb, but if you're making a lot of critical decisions, maybe late at night, when someone says, hey, what do you want to eat? That's just sort of like throwing a wrench and everything else. Like, oh, now I have to decide about that too. I'm trying to make critical business decisions, but now I also have to figure out what I want to eat so that I can stay focused. Honestly, for us, I've been there on both sides of it and just having someone say, hey, I'm going to get food here is that good is way better than saying, where do you want to get food or having it just show up? Like, here is pizza, now eat. Like, great, everyone's going to eat it, have an array of things that everyone can eat and we don't have to worry about it. It allows the engineers to focus on what they need to do without having to make more decisions on top of their problem. And sorry for the sports metaphor at a tech conference, but you need to manage your people as well. So it's just like in sports, you don't put your best people in all of the time for every game, for the whole game, right? We need to make sure there are people that can fill in. We swarm an incident when it happens, right? Everyone wants to get involved, everyone wants to help out. We've all been there like, hey, how can I help? How can I help? How can I help? But if everyone's giving 100% right from the start, you have no one left at hour 10, hour 12, hour 24. Who's going to jump in at that point? So it's important to actually manage the people. Make sure you have people available so that if you say, hey, I need you to go home, someone else can come in. I think especially personality type-wise, a lot of individuals, I'll own that, I at least, have a tendency to work on a problem until it's fixed. That might mean that I'm not making good decisions if it's taking too long to fix it. So having that knowledge of your team and how they work and who those people are and saying, hey, I know you really want this fixed, but I really need you to go home and rest right now because I want you to be better in a few hours, not okay right now. So it's key to just make people aware of themselves. There's this, the sun keeps coming, right? It's gonna happen no matter what, whether or not the problem's fixed, tomorrow is gonna be here and time's gonna keep happening, so I need people that are there to take over. So the other part about communication, right? There's a communication internally, but there's also that public communication back out of the organization or that response team. Where the leader might not be the person communicating, view as a leader can control that message and that phrasing, you kinda control what comes in and out, the input and output there. So the main thing is you wanna make sure you or someone who's delegated is communicating quickly. If there's an incident, I wanna know that it's happening. I think it was, maybe it was last pass or something like that, I was trying to log in for like two hours and couldn't get in because they had an outage with their login service, but it was not communicated inside of their application. Extremely frustrating two hours of my life, I do not get back now. Like please communicate quickly and in a way that reaches people quickly as well. And we wanna communicate often. I love like status pages are wonderful so that when I'm trying to find out if a service I'm waiting for, because you have customers, whether they're internal to your business or external, that are waiting for you to come back up. Communicate often about what's going on, just knowing that it's gonna take me another 16 hours allows me to plan my day or if it's gonna take five minutes, I can just say, oh, I can wait five minutes. Knowing the status of things is important. And I think a key here is to be open and honest about the problems as well. When we have failures, it's really easy to react and say, oh, I don't want people to know that we screwed up. But I think we've got two companies almost a year apart that deleted production databases. Yep, that was us. One of our engineers just deleted it. And that was what they said happened. Like that was awesome. They were just totally honest about it and said, that's what occurred. So Gliffy did it in GitLab what a month ago at that point. But what's happened, what's interesting is that was rewarded. No one said how dare he. Everyone like camaraderie here. Everyone's been there.  Like I hope all the best for you. We want this to improve. And so everyone rallies around that honesty.  So that's a major interesting thing. This is kind of the summation of all of it, right? So as a leader, your job isn't necessarily to fix the problem. Your job is to empower your team through the communication tools. Your job is to remove the barriers. You're trying to remove everything that's preventing them from being the best they can do. So as a leader in crisis, you want to empower your team and make sure they can do the best job bringing everything back up and running as they can. That's my summation of it. Thank you.


What are folks down here that are looking forward to having you home? Thanks very much for this friend's song, the three-step blur, which is sad and blue. That was, we can't say sad and blue yesterday when we quit the MIRC crew and blue like the Earthies all the time. But we feel ready this morning and we're ready for a very nice new day. Kelly, we're checking and we're looking over your shoulder now. Houston, we have separation. Got it, got it. That's the picture. The picture is great. Can you see it? I can see it from the outside. The engines are working. Bye-bye. Good bye, good bye. Where are you from? From the city of Derovsky. It's the same place as the train station. The train station. It's really a train station. Are you turning? No, but it's better. Are you going to jump? Yes, yes, yes. Catch. Tietz office here looking over my shoulder at the view he's going to be looking at in about four months he's coming to pick Mike up and everybody's real proud of the super job you folks did. It's just been a letter perfect, great vehicle and great crew and we're looking forward to coming back in four months and coming to take advantage of all the things that Mike's learned up there. Thanks Chris. And for the whole crew, Frank Carbertson here as well over the shoulder and sending along his congratulations. From Frank too, it's been a great flight and all the work he did for phase one is really paid off. We're on board with you. Of course as you see and we have the reflection of a full moon down over Florida, it is a gorgeous night in Florida, our peninsula in view as we head out over to the Atlantic. Back with you now, we just had Teedra's hand over, did you call? Beautiful night tonight, but all of the eastern U.S. we've still got the tip of Florida Peninsula in sight while we're over in New York, we see New York, Washington, all the way over to St. Louis down to Atlanta. I need to stand out just nice and clear, it's gorgeous out there. You're our window. Up here singing in the late green winds, folks.