para usar o atome do microscópio para extractar as propietas físicas de vírus e aqui você temo como esta máquina está funcionando, você temo o cantelebre, no cantelebre do Iberiano, você temo um típico mágico. Of course, isto não é para a escala, vemos unha escala na escala later on e con este sistema podemos manipular, podemos provar, podemos manejar para tomar imagens de estas vírusas individuas e também podemos extractar unas propietas mecánicas. Então o típico hoje, nesta escala eu vou fazer a introducción brief, então eu vou... Eu vou dizer un pouco dos experimentos que nós performamos no nosso lab sobre as mecánicas de vírus humanas, nós tentaremos explicar como as mecánicas de toda a escala e o coro de vírus humanas nos da uns perspectivos, as intenciones sobre a interplanera biofísica para que se perfilie o vírus biológico de estas vírusas. Então nós conectamos a o ponto 2, vamos explicar como nós monitoramos o rellénio de genome, como nós veamos o vírus humana do Andrés, ok? No ponto no nº 4 nós vamos cambiar o típico un pouco e tentaremos explicar explíxan que son o rolo mecánico de proteínas de sementín na sessas sistemas. E finalmente, eu tentarei coxer. Então, o vírus, o típico dos títulos que usem x-ray e eletro-microscopia e con estas duas técnicas é possível coxer bonitos estruxas A loa de then, con muy detención, con muy particulares detencias sobre as proteínas, e actualmente, o náxido de estas técnicas é que, of course, nós entendemos a loa de estruxas e podemos estabilizar relaciones entre o estruxas e a facción. However, beyond the struxas and the facción, nós temos also properties. E, en este caso, nós temos unas propietas físicas, como, por instancia, thermal stability, mecánicas, estratix, vibración, etc. E todos estruxas nós podemos clasificar them as physical virology, que son as propietas físicas de viroces. E, por mesurar estas propietas, podemos tentar acomplir e completar este esquín between the stretcher, the function and also the properties. Por instancia, podemos mesurar, podemos palpar a single virus para extratar a presión, a presurización que nós tenemos dentro, e também podemos mesurar as estratix properties de viroces. E, atomiformeroscop is a single molecule technique que allows mesurar the properties of viroces and to get images of viroces in physiological conditions, so we can see functional protein cells and in the second term also it's possible to make manipulation and real time experiments. In order to make these experiments in liquid, we need to submerge both the tip of the cantilever and the surface in liquid. So, when we put our viroces on top of the surface, we just seek each one of them in order to perform the experiments. In this image you can see a real comparison between one of these AFN cantilevers. This is the shape of one cantilevers, which is about 50 microns of width and about 200 microns of length. And each of these small dots you see here are fluorescence-label viroces. So here you can see in a glance how is the comparison of size between the cantilever, the AFM and all individual viroces. Actually, atomic for microscopy can be understood as a blind person with a white cane, which is probing the environment and is trying to get a virus and to see if there's a chair, a virus or whatever. So, if you get the picture of this guy here, you get the idea about atomic force microscopy. The typical approach we use for imaging viruses is jumping mode. In jumping mode we are trying to see this guy here, this virus, by going back and forth with the AFN cantilever in such a way that when we are down, we are in contact with the surface or with the virus, we measure how much force and we control the force we apply on the virus and then we release the surface and once we are out of contact, with the surface we make the lateral displacement of the cantilever. In this way we avoid dragging forces acting on the viruses that can destroy them. Here you have an example of one human aviravarous particle where it is resting on top of a triangular facet. Here you can see a pentameric stretcher, just the penton, and also you can see the exon. Actually we apply a filter, we can distinguish, we can resolve much better the details. Here you see the penton here, surrounding 1, 1, 2, 1, 2, 3, 4, 5 capsomers. Actually you can see the individual proteins which are on the capsomers too. On the other hand also if you make a zoom here, also you can see the trimmers of the exons, so here you can see each trimer and each trimer is conforming in this case one capsomer of the human aviravarous particle. So the main tool that we are using in order to extract the mechanical properties of viruses, we are using the atomic force microscopy cantilever as a hammer, well, it's just a rough comparison, but somehow we are poking on top of the virus with the cantilever, with the tip, in such a way that first you deform the caps a little bit and after a while you break it, you use surpass the breaking force. Actually here in this cartoon you see a better explanation what I said before. Here you see the stage number one where the tip and virus are out of contact which corresponds to this deformation which actually is zero, there is no force applied to the virus. Then in point number two the cantilever starts touching the virus at this point and you see actually it's bent a little bit and this bending is detected in this sort of linear deformation that is provoked by the deformation of the virus. We see the combined spring constant between the cantilever and the virus. The third part of the experiment once you go above the breaking force you break the particle and sometimes the particle is going away, sometimes the tip is penetrating inside the particle, well, this depends on each experiment, but the important thing is that you see the third part as there is a sudden decrease in the deformation force which is going on. If you keep pushing at the very end, eventually you can find the slope one, the infinity slope, sorry, which means that you are now deforming the cantilever on top of the surface. Here you have an example, in this case this is a p22 bacteriophage particle, this is the particle before and this is the particle after. So here we perform one of these single indentation assays and then after you break the particle you see you can see a little hole right there. You can control how big is this hole by controlling the maximum force you applied to the experiment. Let me make here a small digression about these experiments in order to focus in these cracks. When you make a crack in one of these viruses usually this crack remains forever. Well, at least forever means that you stay hours and hours scanning there and the crack stays there. However, there are certain protein cells such as for instance the ball particles which are protein cells that don't have a viral origin and this particle which is also an automatic size is possible to make one of these holes as you can see here and then after waiting a few minutes you keep scanning the surface and at some point the hole disappear. This property of healing that some of the protein cells do have has been so far little investigated and I think that this would be a nice place to go on with this kind of research just to see the determinants of how these little particles can recover the cells. Let me go back a little bit to the other slide because from this curve we can extract just let you know we can extract two main mechanical parameters such as the spring constant of the virus which is given by fitting a linear regression to this data here and also the breaking force which is telling us about how much force do you need to break the particle. So now let's go to the second topic which is about the mechanics of human adenovirus particles. First as a brief introduction to human adenovirus this virus has a size of about 100 nanometers and one of the particularities of this virus is that it packs double standard DNA together with histone like proteins. So it's not the same as the virus we saw this afternoon ok, where we have a third of a cheese packing a spool double standard DNA in a spool without the help of any protein here these virus need to have proteins inside and actually 50% of the molecular weight of the core are the histone like proteins and the other 50% of the mass of the core is given by the double standard DNA. So it's absolutely condensed by these proteins. In the process of these virus first is recognizing the wall of the cell there is an endocytosis of this particle and this endocytosis there is a stepwise uncoating of the genome. So it means that in this case the particle is losing some parts such as for instance pentons, fibres, exons etc etc. So after a while the kinesins are transporting the viruses to the nuclear pore. Once in the nuclear pore what you have there is a semi-disrupted particle. So it's a particle which is not intact anymore is the genome is already can go outside. One is reaching this place the nuclear pore there is a diffusion process of the double standard DNA through the nuclear pore to the nucleus. Once the DNA is there of course then we have the replication, transcription, assembly and packing. Another important issue about these viruses is that of course here we have an immature particle. So there is a protease that is cleaning sand of the proteins inside this core in order to make a mature particle. So we are going to concentrate actually in these two parts. We are going to concentrate in this assembly of the particle and how this assembly is affected by the fact of having an immature particle or having an immature particle. So this is the maturación process just a cartoon. Here we have the proteins is cleaning sand of the proteins here and at some point this is affecting to the physics to the physical stage of the physical state of the double standard DNA that we have inside. So we will see then two consequences. The first question we are asking here is if the DNA is modulating the mechanical properties of the viral particle and the second question is what is the interplay between these physical properties and the various functions. I will try to give you some insights about this. How can we understand this implication? So we are going to focus in this assembly and in the DNA diffusion. So in order to start experiments with AFM it's always very important to control the symmetry or the absorption symmetry of particles. So here you have a two-fold symmetry axis particle actually you see here two triangles so the particle is resting on the reach between two triangular facets. Here you have a five-fold symmetry axis particle but you can see actually the pentone right there. And finally here you have the three-fold symmetry particle where you see a triangular facet. We are in order to make the mechanical experiments here we are going to focus only in this particle because it's more stable than the other two particles. This particle is more stable because it's resting on top of the whole facet. So it's better to make mechanical experiments. So the first thing we can do is for Easter to make to try to see how much is the spring constant of what is the mechanical evolution not only along maturation but also in comparison with empty particles. So here we have an empty particle, no DNA inside. Here we have a immature particle and here we have a mature particle. And as you see roughly the spring constant we could say that is going up. So why is this one? We can apply a simple model by considering that the shell is contributing with the spring constant and that the genome or the DNA is contributing with the spring constant. So in this case both the spring constant are adding. This is a very simple model but if this model is correct the results I just showed you in the other slide imply that the spring constant of the stiffness of the core of the mature particle is bigger than the spring constant of the core of the immature particle. So this will be the implication. But is this implication true or not? Can we try to dig more a little bit in this question? Well, the problem is that we have the combination and we are assuming something about the core which is inside but we don't see the core. We don't know what is going on inside. So something we can do is we can manipulate the protein shell to open the shell, crack it and access to the core inside. So in this way we can also take a look to the mechanical properties of the core itself. So in order to do that we make these experiments we start the sequential indentation of this particle so in total this particle has been indented ten times. So we pocketed ten times. So here are all the curves. So you see that at the beginning when you don't break it you have sort of like a linear deformation but after a while in curve number three you break it and actually you see there is a hole already there. And through this hole it's possible to explore the mechanics to put the tip through the hole and explore the mechanics of the curve. And actually you see that the behavior of the curves is quite different at the beginning because here you see that it's completely... it's earthy and it's non-linear. Also happens something similar with the immature particle. You start making sequential indentation and at some point you open a crack and again you get this kind of curves which compare very well with that. So what is the difference between both? Well we can try to adjust the deformation curves of the core by using the dimitriadis model. The dimitriadis model is taking the hertz model, the hertz model of the formation and it's adapting to the fact that usually when you do the biological sample it has a finite... just one thickness. I mean it's not infinite. So by using this it's possible to get information about the job models of the course. I will tell you don't pay too much attention to the absolute numbers. The numbers are not important here. The important thing here is the comparison between both numbers because both numbers have been obtained using the same approach. So the important thing here is the comparison. Here you see that the job models of the mature particle is very low, it's reaching about 0.3 megapascals and in the case of the mature particle they are reaching about 1.2 megapascals. Actually you can see already by eye that the curves are quite different, the green and the blue. So the blue are from the immature particle and the blue are from the mature particle. And this then is in contradiction with the thing that we saw before because before we saw that the spring constant of the inert particle in the case of what time was bigger than the spring constant of the inert particle of the mature particle. So this is a guess thing that probably we have some persuasion inside. In order to accomplish for these two things. You see that the curve behaves completely opposite to the behavior of the spring constant of the inert particle. In order to see if we have persuasion or not we can try to play with counter ions in this case with spermidine. By using spermidine our goal is to condense the genome and try to make it harder after condensation. So this is what we do here. Here we make a new experiment. Here now we have the red curves. The red curves are provided by the deformation of cores that have been condensed with spermidine. An actually you see a very interesting behavior that consists that the value of the jump modulus is going from 0.3 megapascals about 0.8, 0.7 megapascals. But still it's not reaching the condensation state of the immature particle. So this is telling us that we are able with the spermidine to condense the genome of the mature particle but still the condensation is not good enough to reach the values of the immature particle. Well, in order to estimate the pressure we have inside there are two approaches we use. The first approach is just to use this analytical formula that you can find in this nice publication. It's a formula that is giving you how much is the persuasion of a spherical shell as you measure the spring constant with and without pressure. If you make the numbers you get about 3 megapascals. The other approach is consider the virus, I mean the DNA because it's condensed with histone proteins. So it's not the same thing that bacteriophages. So here you can consider as an unbranch polymer and without going into details you also can obtain the value of about 3 megapascals when you adjust the numbers to the expected values of the persistent length of one genome which is condensed with histones that you can find in the literature. OK, so what is the model we propose? What is going on here? In principle, here we have these condensing proteins. For instance you have protein 7 which is this sphere where the genome is wrapping around. But also you have protein mu which is this green one that actually the stretch is acting like a staple which is grabbing two DNA strands together OK, and are holding together two DNA strands. However, when you have the maturación process the protein is cleaving this protein and then after cleavage what happens is that this protein is not holding the DNA strands together any more. So the DNA strands are free to fill the reparsion the restorative reparsion between them and this is exerting certain preservation against the world. So what is the biological implication of this preservation in maturación? Well, we know already from a few of these references for instance by Birklaq and also by in our lab that the weakest points on a shell are the pentons, OK, because they are under stress. So in this process of uncoating what's happening is that there is changing several conditions of the environment surrounding in this case the adenovirus. For instance the pH or other ions. So what we propose is that this change is weakening even more the pentons and this precipitation is helping to pop off the pentons from the particle and helping for disruption that has to take place in order to have the final infection, OK. Well, this was with the uncoating about what is going on with the diffusion of DNA. Is helping that for that or not? So this is the next question we are going to try to address. For this experiment we are using a different mechanical approach. We are going to use fatigue. Fatigue means that instead of making a single indentation and say where we completely break the particle we are going to gently touch and manipulate and massage the particle in order to make it lose some of the parts. So we can do that with the jumping mode. Actually with jumping mode you can touch the particle thousands of times for getting images. This is what you have to do. So at the same time that you apply a low force in this case we are applying 100 piconewton that I tell you is very low compared with the 3 nanonewton for nanonewton you have to apply to break the particle. So this force is really low compared with the breaking force. So when you do that you can obtain, you can monitor how is this assembly process induced by mechanical stress and this is actually what you see in these movies. Here you have the assembly process of a mature particle. If you keep an eye on that you can see that the first parts that are lost are first the pentons and after a while the particle collapses completely. The same thing happens when you are monitoring the mature particle. Again you see that there are some cracks, some voids that are opening right at the pentons and then after a while also collapse. So now we want to analyze the data that we can obtain from these movies. So in these curves we are monitoring the size of the particle as a function of the time. So the time is telling us how many images are we taking or one of these particles. And here we are seeing the height of the size of the particle. Each curve is a single virus. So here we have 1, 2, 3, 4, 5, 6, 7 viruses. Here we have something similar in this case with the mature particles. Each curve is a single virus and again we have something about 7 viruses. Well, so what is going on in the mature particle? In the mature particle you see that first the particle is keeping the size quite well. But at some point after a few images the particle collapses and actually the tip is getting very low heights inside the virus. You see are actually very low heights and all of them are random, okay? However, in the mature particle the things change. You see that first the particle is keeping the size but after a while the size is going down a fixed value for each virus. You see it's about 25 nanometers. For each virus the same. So here is a pattern. Where is coming from this pattern? We think that this pattern is coming from the fact that you are removing only the shell of the virus but you are not able of digging inside the core. So this is like taking an orange. You take an orange and you peel it, okay? You can remove the wall of the orange and the orange remains intact inside. So here is something similar. You remove the wall but here because the genome is condensed, okay? Then you cannot manipulate the core and then the size you see here is the size of the core plus the size of the wall there. And actually what we propose is that the mature particle the DNA escapes as soon as you break the cage. However, in the immature particle you break the cage but the genome remains condensed. However, again the question arise here is can you really visualize the genome on coating? It's possible to see the DNA. Well, with AFM you can see already some gas. For instance, this is the final state of one mature particle where you see that the core has been divided into pieces, okay? And actually we put a yellow color like that one to the height of two nanometers. And actually two nanometers is the size of the diameter of the DNA. So we could assume and we could think that the yellow color here is telling us where is the DNA. Also you see these black pieces and the black pieces are the capsules that have been removed during the experiment. Here in the case of the mature particle it happens something similar. However, now you don't see so much yellow color you don't see so much DNA, you must say. And also the core remains intact in one single piece not in two pieces right here, okay?  So still the question remains, can we see the genome? In order to address this question we are going to use fluorescence microscopy combined with atomic force microscopy. Both at the same time, both simultaneously. And we are going to use a fluorophore which is called Jojo-1. Jojo-1 is an intercalator of DNA that is absorbing in the blue color and is emitting in green color. But it is only emitting in green color as soon as it binds to the DNA. If it does not bind, there is no emission in green, okay? But if there is emission in green means that it is intercalated right there, okay? In this place. So what is the experiment we propose? We propose to break the particle in the presence of Jojo-1 molecules. So these green balls here are the Jojo-1 which are expecting, which they want to access. They want to access to the DNA but the DNA is in the cage. So we don't have any light. What do we have to see light? What we have to do, we have to break the cage, okay? If we break the cage then the Jojo is going to be able of intercalated to the DNA and then we expect to see some light coming out. So basically the experiment we propose this is an artistic rendition just to show the experiment we want to poke on the virus. We want to see the light coming from the DNA. And actually this is what we see here. This is a bright field image. This is the cantilever and the tip is somewhere there. So I ask you to pay attention to the center of the circle, of the orange circle where we are going to locate the virus and we are going to poke it, we are going to destroy it. Now we change to total internal reflection, fluorescence and we change the cantilever disappears and we see only what is going to happen at the surface with total internal reflection. So we start the experiment at the beginning as you can see there is no light coming there. Now we go there, we poke the virus, we break it and you see there is a spot which is going up there that is arising. So this light there is coming from the DNA that was packet in one of these particles. So here I can show you the whole experiment in the better way this is the inter particle with atomic form microscopy. This is the background noise of the fluorescence, no light at all. Then this is the particle after you break it and this is the light that you have after you break it. So you see that when you break it you have light, you don't break it, don't light. So the judge is telling us the DNA is there, is being released. Actually here you see the combined curve of the orange one is the force curve, the indentation curve which is telling you that the particle is breaking at this point. And the green curve is telling you the behavior of the light. So first you have background noise and as soon as you break it the light is going up and up because you are releasing in this case the genome. So if you make some statistics you can compare how it released the genome from the mature particle which is the green curve and the mature particle which is the blue curve. And actually this axis is the time of the experiment so once at the beginning you see that there is no light because we don't break it. After you break it the light is going up very fast for both kind of particles but at some point you see that the mature particle is growing faster, is growing faster than the light coming from the mature particle. So this is telling you again that the genome released by the mature particle releases better or let's say the other way. The yoyo one is for the yoyo one is more easy to find the released genome in the case of the mature particle and in the case of the mature particle. And we think that this has to do something with the condensation of the DNA. Also the other parameter you can measure is the width of one of these spots. If you measure the width you can see that again the width of the wild type particle is larger than the width of the mature particle. Again it's telling you that the yoyo access much more easy to the genome of the mature particle and the genome of the mature particle. Well, in this experiment we are playing something similar but in this case we are making fatigue. We are just breaking the particle step by step and you see that as soon as you break the particle there is some light already coming out from the virus. So it means that the yoyo is interacting somehow with the DNA. We don't know still, we don't know yet if the genome, if the yoyo one is going inside through the pentons, through the cracks we are inducing or if the DNA is going outside and then is finding the yoyo. We are still trying to make experiments to figure out this issue. So, well, the conclusion here is very easy. First the mature particle spreads more the genome and in this case the mature particle emits less photos. So it's very general response that drops very well with the mechanics that we saw in the other part of the talk. Now let's go to the last part of the talk which is about the mechanical role of cementing proteins and the cementing other decorative proteins are an alternative strategy to strengthen the various capsids during maturación. I mean some particles when they mature they change the interaction between the capsomers, they change the contact area between the bonds in the capsomers but some particles they have the evolution, they provide with different mechanics and the different mechanics is to strengthen the particle by using cementing proteins. So these proteins are covering the external part of the virus. So we already were doing some work quite a few years ago on landafage and the landafage here you see in blue color is the co-protein and in orange color you see the cementing protein. And in this case we figure out two things. The first thing is that there was a change of the mechanical properties induced by the decoration of the cementing protein so here you see the breaking force and the spring constant and this is for the undercorreted particle, you see about this value. However now you put the correct proteins and what happens is that both the spring constant is going up to this value and also the breaking force is going up. So the particle is getting stiffer and the particle is getting also less fragile because you can apply more force without breaking the particle. This is the case with landafage. Now also something similar happens with fatigue. When you play experiments with fatigue we saw that when you have the particles that are decorated with the cementing protein they need more loading cycles in order to be destroyed that the case of the particles that were undercorreted. So now the question is can we use this philosophy to recover for instance weekend protein shells? Let's assume that you have a weekend protein shell with some defects and you want to recover it because you know that it's very weak. Can we use the approach of putting cementing proteins like a general rule? Like in this case you are putting these bolts in order to fix these panels to the wall. So the wall here is very weak and can fall down. But here once you put the bolts you don't have this problem. So we are going to use to in order to address this question the P22 bacteriophage capsid. With these capsid it's very nice because you have a large variety of structures. For instance here you have the expanded particle but by heating the expanded particle at 65 degrees during 20 minutes you can remove the pentons. So you can actually remove the pentons and you can create a new structure which is called with a wall. Which is the same structure now before but in this case without pentons. Now, another interesting thing you can do here is that you can use the cementing proteins of the L bacteriophage also in this bacteriophage in P22 bacteriophage you can use them to cover not only to interact not only with the expanded particle but also with the with a wall. So as you can see here we have four kinds of particles. We have the expanded particle without cementing proteins we have the expanded particle with cementing proteins you have the with a wall without cementing proteins and you have the with a wall with cementing proteins. So we expect that with all this variety of particles we have here we will be able of addressing how is the mechanics or how can we manage to manipulate the mechanics of the particles with the cementing protein. So the first glance that you can see here is is the partial collapse of the particle on the surface. Actually I was discussing we bought an about the launch about this issue. So here you see for images this is a 5-fold symmetry axis oriented expanded particle this is a decorated expanded particle this is a with a wall and actually the with a wall you can see the three these three pentons are missing because it's a with a wall and here again even if it is decorated but the three pentons are also missing. So what happens when you put one vira particle on top of the surface? If this is the vira particle you put on the surface and there is a partial collapse of the particle on the surface. So if you pay attention to the size of the particles when they are absorbed on the surface you can already learn something about the resistance or the mechanics of the particles. So let's see that. So here in this graph I am showing you the size of the average size of the height of expanded particle and expanded particle plus the collision protein, okay. So I finish. So here you see there is an increment in the size so it means that the quality particle collapse less than the under quality particle. Something similar is happening with the with a wall. The with a wall is collapsing a lot. You see it's reaching collapses of about 80%. So it's getting on only the 80% of the original size. However, as soon as you put the cementing protein you are recovering almost 88% of the original size. So this is telling you already how is going to be the mechanical behavior. Also you can play, you can check the statistics the chemical resistance against for instance SDS. SDS is a protein in the natural, okay. So you put SDS for expanded and with a wall particles you see that the expanded particles are resisting much better than the with a wall. However when you put the cementing proteins and you compare the with a wall with cementing proteins here you see that the black curve is remaining almost constant. So the cementing proteins are making the particles more resistant. Well I just changed the last slide here. Here we calculate how much energy you have to apply to break one of these particles. You calculate this energy as actually this area below the curve and you see here you can compare the energy in KBTs. You can compare that the with a wall you have to apply very low energy about 700 KBTs. But as soon as you put the correction proteins you have to apply an energy of about 1500 KBTs in order to break the particle. And something similar is happening also with expanded but this more is exaggerated in the case of with a wall. So with this I finish my talk and I want to thank to mainly to my students which are taking part of my work here. And actually I tell you that for instance Manuel and Paco they have two posters that they will be very happy to show in you. And also I want to thank to my collaborators Melitú Carlos Catalano, Carmen San Martín, David Reguera, Iván Escab, Núria Verdaguer, Antre Bordaglas. Thank you very much for the attention.


John Lund, Guy Madison, Colleen Townsend, star on Family Theater. Mutual Broadcasting System in cooperation with Family Theater Incorporated presents The Fourth Strike, starring Guy Madison and Colleen Townsend. Reapportions are transcribed. John Lund is your host. Every week Family Theater makes a promise to its listeners. A promise that God will help and protect the home where daily family prayer is practiced. A promise that the family that prays together stays together. I know that's a pretty big statement, but it's true. Family prayer may not bring wealth or security or health or freedom from all worry, but it will unite your family in a bond of love and understanding. It will certainly bring to your family the strength and sense of God's presence, which will help it overcome every difficulty. That's what Family Theater promises you each week. The same promise that was contained long ago in the words of our Lord. Ask and you shall receive. John Lund speaks again after our Family Theater play The Fourth Strike, starring Guy Madison and Colleen Townsend. There you go, Mr. Bloomfield. Pretty little place, isn't it? Looks all right. What does Bloomfield like? Oh, enough people so as you don't get lonely and not enough to feel crowded. That's the farmer's trust and savings over there and Jeff Norton's drug store. Frame building here is the Bloomfield Weekly Sentinel. The weekly? Is that the only paper in town? Well, it's not for the size of the place. Say, what's your name, anyway? It's Graham, Al Graham. Well, good luck to you, Graham. Hope you like it here. Bloomfield, meet Al Graham. Bloomfield, elevation 3,752 feet, population 4,025. Watch us grow. Well, Bloomfield, even if nobody seems to be watching, your population has just gone up one. No, Mr. Graham, I'm afraid not. I'd like to put you on, but this is a slack time of year. And I only need one fella at the fountain. Mr. Norton, I didn't know jobs were so hard to get. Just the unskilled jobs that anybody can do. And now I could use another pharmacist to spell me or supposing you were a teacher. The high school is pretty shorthanded. Now, I guarantee I'm no pharmacist, but school teaching. Huh, I used to go over big with the kids. What's that? Oh, sounds like my back window again. If those darn kids... Oh, I was right. How am I expected to keep a window in there? This rock here. It's the second time this month. I warned that Chris Saunders and his gang the next time they made trouble. Oh, you know how kids are. I used to be a fast boy with a rock myself. Oh, I realize it's not malicious. But this sort of mischief can't go on either. I suppose I'd better call... Hey, wait a minute.  Where does Saunders and his gang usually hang out? You can generally find them loafing around that vacant lot in the back of Emerson's grocery store. I hope you'll excuse me, Grimm, for mentioning this. But this isn't exactly your business. You're absolutely right. But I think I better get me a business if I'm going to find a job. Hey, Joey, you think old man Norton will call a cop this time? My dad will never believe it was an accident, Chris. Oh, what do we care about old Norton's window anyhow? Hey, here comes a guy, a big guy. Yeah, I never saw him before. Don't anybody say anything, okay? Hi there, fellas. Ice afternoon, isn't it? I wonder why they call Bloomfield a friendly town. You don't live in Bloomfield.  Well, I'm glad to know you can talk anyway. I'm looking for Chris Saunders. What do you want him for? A couple of minutes ago, Chris Saunders tossed a rock through the back window of Norton's drug store. Probably an accident. Yeah. I've been measuring out the distance as Saunders had to throw. He must have quite an arm. A pitching arm, I mean. I wanted to meet him. But you fellas are too small, so I'll... Hey, now, wait a minute, mister. I'm Chris Saunders. I'll bet Mr. Norton sent him.  I just came on my own. How come? Maybe I'm a scout from the New York Yankees. Yeah, yeah, with those clothes. I know.  It took a lot of traveling to get to Bloomfield. Look, you kids ever play any baseball after school? Yeah, sure. We play sometimes in a lot here. Why don't you ever get up a team, a real team? Why? Who'd we play? Well, there must be a lot of other high schools in the county. Or you could play them. Maybe win a championship.  Besides, all we got are women teachers. They never heard of baseball. What have you had a coach? A man coach. No, no. Hey, what are you selling, mister? Not a thing. I might give you something, but I'm not selling. I've learned better. Yeah, well, we have plenty of fun just fooling around, a fuzz. Okay, okay. Don't mean to butt in. I just take to see all that talent going to waste. That's all. Well, that kid over there, the one with the long legs, make a good shortstop. Maybe an outfielder if he can throw. And you, Chris. Me? Yeah. Did you think I was kidding about that arm of yours? What about my arm, mister? Call me Al. You know, you may have the makings of a pitcher. Just might. Maybe you couldn't hold up in a real game. But, well, you know, Bob Feller started out throwing rocks at the wall of a barn. Yeah? It's your pity you're not interested in baseball. Hey, Al, what could I do? I got long arms, left handed. There's only one spot for you. First base. Oh, wait a minute, wait a minute. I'd have to see you work out first before I could say any more. Too bad we can't get together, but I guess if Chris doesn't care for it. Oh, Chris, why not? Yeah, sure, why not? Let's try it, Chris. Okay, you guys, okay. But if I don't like it. Sure. If you don't like it, you can quit anytime.  Hey, you're on. How about her on? Hey, it's during your five o'clock. I'm gonna miss supper. Yeah, me too. Well, I'll see you tomorrow, Al, huh? Yeah, okay. Oh, pardon me. Sorry I didn't see you before. I've been standing by this billboard for the last 10 minutes listening. Oh, it's very flattering to draw such a pretty crowd. I didn't mean it quite that way. I simply want to know why this big act about baseball. It wasn't an act. I'm in it.  Yeah, that's right. We haven't been introduced. I'm Mal Graham. And I'm supposed to give you a name and exchange, right? Well, I'm Kathy Saunders. Saunders? In relation to- Certainly. And being Chris's sister just makes me that much more curious. What do you intend doing with those boys, Mr. Graham? You walking down this way, Kathy? Until I get an answer, Mr. Graham. I'll talk slow. I like boys, Kathy. I like baseball. I like to eat, and my habit of eating requires a job. So I thought maybe the high school could use an athletic instructor. A coach. Well, yes, they could. But why didn't you go to Mr. Jackson and apply for the job the regular way? He's the principal. It'll be faster and sure if I show him a few samples. I'll say, Kathy, according to that billboard back there, the lady's auxiliary is giving a down Saturday night. You're very observant. And what have you plotted out as your next move? Go see Mr. Jackson. Not a chance. Mr. Jackson will come to see me. How about that Saturday night, Kathy? Do you know your great failing? Lack of confidence. And I don't think Mr. Jackson will chase after you. Okay, it's a bet. When I get the job, you go to the dance with me. Why? Who told you I couldn't resist a dare? Well, I've seen your corsage. Al, I'm sure you can find out where I live if you really work on it. But if you should want me during the day, call the Sentinel office. The newspaper? You, a reporter? I'm the one and only... Well, what's the matter, Al? You look almost as if... Nothing, Kathy. Just a surprise. You see, I never cared much for reporters. But I guess I'll learn to like a lot of new things. Don't feel bad if you don't strike them all off them. Oh, yeah, but it burns me up when I can't... Hey. Hey, look who's getting out of that car. It's the principal, old man Jackson. And just in the nick of time, too. Today, Saturday. Well, what do you mean, Al? Hello there, Mr. Graham. Shall we look, Chris? Coming up. What can I do for you, Mr. Jackson? Oh, you know me. I've been hearing things about how you make use of this school playground after hours. What sort of things? Several of the boys' parents have called my attention to your baseball team. They think it's a splendid thing. A curb to juvenile delinquency, Mr. Graham. Call me Al. We just come out here for a good time, me included. Well, Al, it's something which Bloomfield needs badly, has needed for some time. Now, I've been understaffed with the school, you understand? We do need a younger man on the staff, a person who could organize a complete full-time athletic program. Now, I was wondering if I would take the job. Well, in a word, yes. Could you come into my office Monday to arrange such a contract? Naturally, Al. I don't know your plans, but I was told that if I asked you by Saturday, well... Who told you that, Mr. Jackson? Well, it was the Saunders. Her brother plays on your team. And I've been wondering all day about what I was going to do tonight. Okay, Mr. Jackson, it's a date. Bye. Let's sit out here on the verand... Kathy, my dancing has gotten out of condition. I expect you're used to better partners. I'm only a little country girl, you know. Me? I like the country. I like the scenery.  Chris thinks you're simply wonderful. He's a good kid himself. Got a dandy curve. That seems to run in the family. Why, Mr. Graham, it's time we went back inside with the ladies' axillary. Come here. As a schoolteacher, you're supposed to set an example. I just did. For both of us, Kathy, maybe I'd like to run in the family, too.  We've gotten ourselves into the state league and maybe they're too tough. Still, there's no use playing a team. You know you can beat. They look pretty rugged now. Oh, maybe they do. But remember, they use the same ball and the same size plate that you do. Okay, let's go. All right, let's go. 35, 45, 50, one dollar. There you go, Mr. Jackson.  Isn't it, though? Al Graham has done wonders with those boys. He certainly turned up a pitcher and that Chris Saunders not taking any credit away from Al, but you might say that I really discovered young Saunders. Is that so? Then you must come out tomorrow and watch him pitch. It's the league championship game, you know. Well, I'd sure like to, but business comes first. There won't be any business tomorrow afternoon, Mr. Norton. You might as well close up shop and see a ball game. Just a second, fellas. Before you got on the field, I'd like to say a couple of things. One is that you've done a beautiful job this year and I'm proud of all of you. The other is that it really doesn't matter whether you win or not this afternoon. What do you mean it doesn't matter? Oh, yeah, that's great. And I'll be in there praying for you every minute. But remember this, play hard, win if you're good enough, sure, but win the right way, fair and square. Because if you can't win that way, you don't win at all. No matter what that scoreboard says, you all know that underneath, but believe me, sometimes it's easy to forget. Okay, let's get out there and win that championship by 25 runs. And congratulations again. Appendant the first year. I'll bet most new coaches don't do that well. I had a hot team and first-class inspiration. Well, the light's still on in the house. Chris must be staying up late.  The season's over now. But not for working girls. Good night, darling. Night, honey.  Yeah, I wasn't sleepy. What have you got there? Oh, why, those are dad's old baseball scrapbooks. Yeah, I kind of thought it'd be nice to look at them tonight. He'd have been very proud of you today. Let me see those scrapbooks, Chris. Oh, here's the one I was telling Al about the other day. A picture with a first name that's just the same, Al. You know, Al Martin. Yeah, hey, I remember dad telling about him, that big game in St. Louis. I was looking for that clipping. I'm sure it's here someplace. I'd like to... Kathy, what's wrong with you anyhow? Oh, Chris. Chris, look at this picture of Al Martin. Okay, what's her... Hey, that's why that's... Yes, that's our Al. Al Graham. Why, Chris, you're the same man. For the love of Mike, Al Martin. He's our coach, Kathy. Why, he's been coaching me. I wonder why he never said anything. Why he... Al Martin. Hey, where you going? To see if there's still anybody down at the sentinel. For the first time in history of Bloomfield, there is going to be an extra. Al, I've been trying to reach you all morning. Oh, you've seen the extra. Yeah, I had a good luck. What's a big idea? What do you mean? Smaring me all over the front page like that.  But you are Al Martin, aren't you? Martin Graham, it doesn't matter, does it? I don't understand, darling. We're proud of you. All of us are. You... You mean you don't know? Don't know what? Hey, Al! Oh, there he is, Mr. Jackson. Hiya, Chris, Mr. Jackson. You're a difficult man to find on Saturday morning, Al. I can guess why you're looking. I'll wager you can't. We're having a banquet tonight. Bloomfield is down at the Civic Hall.  Tell him about the collection we took up, Mr. Jackson. Later, Chris, later. Well, Al... But this is quite a surprise. I didn't expect Bloomfield to act like this. That is, I forgot such a small town might not... I feel very honored, Mr. Jackson. Suppose you come along with me and we'll make a few arrangements. Glad to. See you later, Kathy. You too, Chris. Sorry I flew off the handle about that story. Oh, you sure a great guy, huh? Hey, why the funny look, Kathy? Chris, if I wanted to find out something about baseball, something special, whom should I ask? And don't say Al. Oh, I guess you could write to the sport news or commission a channel. Hmm, that's a good idea. But I don't think all right. See you later, Chris. Think I'll walk over to the telegraph office. Oh, Kathy, what are you doing out here on the parking lot? Waiting for you. Why aren't you in at the banquet with the others? I wanted to talk to you alone, Al, before the banquet. Okay, shoot, honey. I wondered why you were so upset this morning about the story. A few things you said. Well, blame my report is curiosity, but I wired Commissioner Chandler's office about you. Do you want to see the answer I got? Don't bother. I know what it says. Al Martin was kicked out of baseball two years ago. Oh, Al, darling, why didn't you tell me yourself? So you could spread that all over town, too? I didn't ask to be Al Martin here. All I wanted to be was Al Graham of Bloomfield. You wouldn't leave me alone, so all right, I'll be Al Martin. But you're not. Well, surely you'll tell them. Why should I tell them?  That's okay by me. Oh, but someone will know someone's bound to remember. Not for a long time, maybe. Not unless you tell them, Kathy, and I don't think you will. I don't think you've got the nerve. The nerve? To break those kids' hearts. To break Chris's heart when he finds out what a great guy I am. No, you won't say a thing, Kathy. And you can bet I won't say anything. Al Martin is going to stay on this gravy train until it cracks up. But Al, I thought we agreed on honesty. Well, that telegram ought to tell you different. Come on, I don't want to miss any speeches about me. Bloomfield Union High School Baseball Team, and with the blessing of the faculty and parents, I take pleasure in presenting this gold watch to a great athlete, a great leader, and a great inspiration. Al Martin. Go on, Al. Open it up. There's writing inside. Well, here, give me. Let me read it to everyone. Hey, folks. To Al Martin, winner lose from the gang. It's for you, Al. Thank you, fellas, everybody. You can't know how much this watch and everything means to me. Kathy, aren't you happy?  Yeah, it happens I need a watch, but I can't take this. This morning, you read in the sundown about Al Martin. It was nice stuff, very flattering. Made me sound like quite the guy. Well, that's only half the story. The rest of the story goes like this. I was kicked out of Baseball, barred for life. Hey, Al. Yeah, kicked out for not reporting a bribe from a gambling syndicate. Not exactly a happy ending, is it? That's why I changed my name to Graham, why I eventually came to Bloomfield. I thought I could hide here. Then, when you found out who I was, I thought I could keep going and hope you'd never find out the rest. But facing you folks now and especially these kids, I see that would be more cheating. You know, on Baseball, they give you three strikes and you're out. I had my three and for a while there, I thought I could get me a fourth, but I guess not. So that's why I better not take this gold watch. Thanks, Justice Singh. Where are you going? Away, I don't know. It's a big world I hear. But what about me and Chris and the Baseball team? Did you come here to rub it in? You think I want to go? Then why are you going? Why? Because a town won't want a guy like me in it. That's why. You think Bloomfield wants... Will you do me a favor, Al? Would you open this window over here? Some fresh air might not hurt. Window? Sure. Sure, but what's a big idea? For a wise guy, you're pretty dumb sometimes, darling. Life is not always just like Baseball, you know? That's a big idea. Bloomfield wants to give you that fourth strike if you just stay in the game. How about it, picture? Right, Coach. This is John Lund again. You know, here's something I think will interest you. During the past two years, more than 225 stars of stage, screen, and radio have appeared on family theater. That's quite a record, and one we're proud of. And why do we enjoy being on family theater? Because its ideals and objectives seem important to us, and necessary. It brings us a weekly reminder that individuals and families can obtain God's help by praying for it, humbly and sincerely. Many of family theater's regular listeners have written to tell us that they feel as we do, that they enjoy the program and approve of its ideals. We're always glad to hear from you and learn what you think of family theater, for it'll be a better world for all of us when more and more people realize that the family that prays together stays together. Thank you for being with us, and God bless you. Our grateful thanks to Guy Madison, John Lund, and Colleen Townsend for their appearances, and to Wade Miller for writing our plays. Original music was scored and conducted by Max Tehr. This production of family theater incorporated was directed by David Young. Brief portions were transcribed. The supporting cast included Eddie Firestone, Fred Howard, Norman Field, Sam Edwards, Dick Ryan, and Peter Rankin. Next week our family theater stars will be James Gleason and Nancy Olsen in Night Elevator. Your hostess will be Claudette Colbert. This series of the family theater broadcasts is made possible by the thousands of you who felt the need for this kind of program, and by the mutual broadcasting system which has responded to this need. Be with us next week at the same time when Claudette Colbert, James Gleason, and Nancy Olsen will star on Family Theater. Your announcer, Merle Ross. This is the Mutual Broadcasting System.


Hello everyone, my name is Laurent Bernay and I'm here with Eric and I mean, we're very happy to be here today. It's our first CubeCode in person since 2019 in North America. So that's that's great to to be here with you So today we're going to to discuss our migration of the way we build container images to Kubernetes and as you will see it was a bit eventful and it got pretty interesting So before we start we both work for Datadog So we have a few numbers on about Datadog on the left hand side of the slide What what matters is with we are an observability company and and we do a lot of things in the observability space But today we're not going to talk about that of the product We're going to talk about data dogs infrastructure because we both work on the data dog infrastructure teams so we Essentially work on the Kubernetes environment, which is why we're going to talk about building containers in Kubernetes and As you can see with the numbers, I mean we have tens of thousands of nodes dozens of clusters and and with this come a lot of challenges around Many things but in particular building container images So before we dive into an interesting issue, let's do a quick overview of how we build things at Datadog so Quite a while back, I think like up to four or five years ago We were using this very simple setup to build our applications So we had GitLab runners pulling jobs from GitLab and using Docker machine to provision AWS instances and running job on them So pretty standard pretty simple When we started our migration to Kubernetes, we needed to in addition to building applications We also needed to build container images and if you're familiar with the way we build Docker images It usually means having access to the Docker daemon, which basically means being roots on the instance where you run So that's why we couldn't reuse the runners We were using for applications because these runners will be used and there was no way we could run a workload Which will end up being root on the machine because it was running Docker commands So we ended up having other runners which were just standalone machines running Docker Doing one job and when the job was done. They were just Being killed and replaced by new ones that was working fine until Sometime when the company had grown a bit and we're having many more engineers many more builds every day and Docker machine was starting to hit limits We're hitting API rate limits with the AWS API and it was starting to be tricky to To scale with the scale of the company and so we migrated our workers to Kubernetes, right? This was easy because at that time we had enough of knowledge of Kubernetes at Datadoc to be able to do that And it was pretty successful But as you can see to build Docker images, we were still using the dedicated runners The next step in this journey is some customer were starting to ask us for arm binaries Right because we provide the datadoc agent for instance that people use for monitoring and people were starting to use arm CPUs so we need to provide them with arm binaries And so what we did is because we're running on Kubernetes We set up Kubernetes nodes with arm CPUs and we're running builds on this on this node to get native builds Well as you can imagine the next step Well, of course to be able to do that we need to have Docker images that supported arm architecture And so we wanted to have multi-arch images to do this We transition our runners from running simple Docker builds to running Docker build X Which allows you to do multi-arch build by relying on emulation with QMU So it was to be honest. This was magic. It just worked, right? We were able to provide to get images which were both working on x86 and arm by using emulation However, as you can imagine for some builds We move from 10 minutes on native x86 to more than an hour on arm emulated, right? So it wasn't ideal And and today the talk is going to focus on this part like how we build images and how we transition to to Kubernetes So as I was hinting I mean this system was starting to show its limits the way we're building Docker images It's what the last workload running outside of Kubernetes Which was starting to be a pain for the team managing it because Everything else was managed with a single team by a single team doing everything on communities And by this team had to manage dedicated instances outside of Kubernetes to for this workloads It it also requires Investing in this legacy platform, right because we wanted to do multi-arch builds and native arm builds We need new runners, right? Which was not something wanted to invest in and and finally as I said before this was building images raise a lot of concern in terms of security because your builds are basically roots on the on the machine they run on This gets us to the main topic of the presentation today, which is well What if we could actually build images inside Kubernetes? It is very attractive, right? So if you look at how people do this because many people have tried and tried to do it you have multiple ways The first one is to use Docker in Docker So what you do is similar to what I was describing before which is you create a community spot in which you mount the Docker socket and from there you do Docker builds You can also use standard on builders so builders dedicated to build images inside communities and I gave I gave a few example and Finally, you can use a dedicated build demon and this is what build kit is about So can we do can we use this option as you can as I'm sure you've guessed the first one was A no-go from the start. There was no way we would give build pods access to being roots on the on the Kubernetes nodes The second one standard old builders actually worked pretty well in something we we try them and they work But they were a bit more complex to use because you had if you want to do multi-arch images It means you have to distribute jobs You need to run one job on x86 nodes one job on arm nodes and one job to assemble the multi-arch image and push It to the registry so it works, but it's more complex Build kit D although the build demon was very attractive to us because well the UX is great, right? You you can use the same build X command to build locally on your laptop to build on a dedicated instance builder and to build with remote builders allowing native builds So this is what we're going to focus on today So what would this look like like let's get back to our build infra running in in Kubernetes So well what we wanted to achieve is simply is this right when when a job is to build a new image What what we want to do is one of the worker is going to run the build It's going to do a docker build X build command and contact a build kitty demon and do the build What is nice is because it's remote built and build kit support build X supports it We can actually have demons running on x86 and demons running on arm nodes and build X is able to build To to use the remote builders to build the native image on an x86 node and a native image on an arm node assemble them and create a multi-art image of course, we wanted to make this safe and as you know the main challenge with with building images is Most of the time it requires being very privileged right because you have to install packages for instance, which requires being roots But what we want you to achieve is rootless beyond right because we don't want containers to run as roots because when you do this If there's a container escape you hand up with workloads that can actually do things on the host So what do we mean by rootless beyond well? It's beyond where the main demon inside Kubernetes is running as Non-root right as a normal user However, as I was saying before we also need to be root to run some commands And this is where user namespace come come into play right a user namespace is a way to simulate a root user But you're not roots in on the whole so you just root in this limited namespace with limited capability and User namespace are very powerful and they interact with many low-level canal implementations like capability mounts or security modules If you're curious about how this works There's this great presentation from a hero at cubecon in 2019 where he explained exactly how this works So to give you a very quick overview. This is how it looks like You have a built-in worker the Pid one is running as you ID one thousand So a non-privileged user so that's completely fine exactly what we want and Then everything else is running on a user namespace and the UID is zero But I put a star because it's not really zero. It's zero inside this namespace, but it's not food on the host a Very quick example It's very easy to to simulate if you want to try it on a Linux machine this command will create User namespace and in there you can see we're roots in the namespace But we can't touch a file that requires being a root on the host So that's why the touch at C slash X fails because we're just root in this namespace We're not root on the host so we can't do something that requires root permission on the host And if we touch a file we can't we can modify you We say that it belongs to root in the namespace But if we exit the namespace the file actually belongs to you ID one thousand right? So this is how the magic works So this is the the the main intro and I were going to dappy to interesting and fun things To be to be fair when we started working with buildkits Everything mostly works right more than 80% of the builds just work out of the box But today we're going to focus on the 20% that were interesting Because otherwise was the fun right so we're going to present to you like three different issues And we're going to go into increasing complexity. So the first one is Well, actually simple enough. It took us only a few a few hours to understand what was happening So this issue started with this extremely complex docify, right? It's we're just getting an image from a public registry and running a co-test What could go wrong? I mean this felt like something that should work, right? Well, when we run this inside our Rootless buildkit environment here is what we got so that was of course very surprising because this is like the most basic docify you can use and And something that's interesting is we've got an operation not permitted message But it says something about mounts. So something might be happening with the file system So that's pretty surprising. Let's look into it So what we wanted to understand is what was happening. So what we did is we a stressed the buildkits Did demon and we started the built and we looked for permission errors, right? And this one is actually Pretty interesting, right? we have this command that is trying to set an extended attribute and it's actually a sea linux attribute and it's failing and Well, I made the joke about it's always DNS because I tend to talk about networking issues and to be fair It's often DNS, but it's also quite often a sea linux and the only thing I need I know how to do with a sea linux Is how to disable it? I'm sorry So, I mean we had a very good inside of a very good idea what might be happening and And so what we did is we downloaded the layers of the image extracted the table and look at the content of the table And as we suspected there's actually a sea linux label labels on the files, right? And it turns out if you're root on a host Use a namespace you can modify the security context of a file. This is what we do at the top of the slide However, if you're in a username space, which is the second part of the slide and we're entering the username space used by bilkitD We can't modify the sea linux attribute of a file, right? And the kernel disallow it you can't do it if you're in the username space, which makes sense So this is the issue that we open an issue upstream to be fair. There's no magic. Nothing. We can really do But it's pretty easy to mitigate right either you remove the sea linux attributes by pulling and pushing the file or You use an image without any sea linux label Which to be fair is something I would recommend you do and we were looking at because one was an upstream image and The one just the release just after when we were using I couldn't have the labels anymore Let's move to issue number two So this one as you're gonna see was slightly more complex and took us a few days to understand So the docker file is a bit more complex, but once again, no rocket science, right? We're just downloading a depth file and we're trying to install it and Of course as before this works perfectly fine if you do docker build it works perfectly fine if you use buildkits in root mode However, when you use buildkits in rootless mode and you do exactly this for this specific app the build times out so because We are very scientific when we retry it, right? And this time it failed again, but it failed in a very different way It felt with an error saying well Address already news. That's very weird. So at that point. Well, we're like, let's try and understand what's happened What's happening? So we tried again This time it was consistent at least we're seeing the same error address already news So it was very confusing to us. So what we did is so well, let's try from scratch. Let's delete the buildkits depot and try again Well, this time the build times out. Okay, we're back to what we had at the beginning. Well, let's try again This time it fails with the same thing So at least it's consistent with fire the way of repulsing the failures in a way that is reproducible, but doesn't really make sense yet So let's debug So we use this with this way to debug right which is pretty simple We just added netstat at the beginning and at the end of the command to see what was happening And when we start with a new buildkit depot The first netstat show no port bound which is very expected, but the second one shows that the port is bound by the app demon and the build hangs Okay, that's what so maybe package installation is starting a demon right happen sometimes Let's do the second build when we do the second build this time netstat is showing that the port is bound and package installation fails with address already in use So we're getting somewhere right it looks like package installation is starting a demon and It looks like the demon is still running when we do the second build which makes little sense because it's a completely separate build So can we reproduce this by using a different method? So we use this very simple reproducer So you have the docker file on the top left of the slide and the script on the top right So it's pretty simple. We just start from moving to we had a script We run the script and we go the script is done and As you can see here everything works fine except we never get to the last line of the docker file Because with the build hangs so exactly what we were saying before so it's a good thing. We've reproduced What seems to happen is well If we looked into the container We actually see sleep still running which is what was suspected before I should remember some app was still running And that's why the port was bound. So it seemed that the process is leaked or and never stopped So let's let's look exactly at what's happening under the hood. So this is the anatomy of the bilkit worker So we have the bilkit demon When we start the build build decks is going to do an exact in the pod it's going to start the build steps so here it's going to run bash bash is going to run sleep in the background and What's important here and we're going to come back to this later is there's no process sandbox We can see all the processes in the container So if you exactly to this pod and run ps we see our build step but also we see bilkiddy with this kid we see we see everything and When when bash exists so bash is not there anymore the build hangs and the process is still there and never cleaned up So at that point we're like well we got curious right what has what happens if we actually kill sleep inside this hanging build build pod well We were able to kill it But it was never garbage collected because it's and it became a zombie which was also kind of interesting to us so At this moment we took a step back How does it work usually so build step use usually run in a process sandbox and when the step finishes All the process inside the sandbox are killed However in rootless mode we run build kit with this flag that is very clear like this flag says well You don't get a process sandbox and because we don't have a sandbox We can't keep track of all the processes Starting during the build and we can't clean them and you can really clean them up So this got us to but why do we need this flag and? The reason we need this flag is because of the way prokfs work in containers So when you create a container you have your own prokfs, but for security reasons every runtime is Every single runtime is not going to give you a full prokfs It's going to give you a limited prokfs where some of the directory in slack procs will be either masked Which means it's going to be an empty directory or made read only so you You can't modify things system-wide right and And that makes and that's for security reason We don't want to expose too much things to containers and we don't want containers to modify things on the host However when you do this, so you're in a container you have a prok which is partially masked for security reasons If you want to create a new prokfs, which is what we would need to do to create a sandbox for our build step We actually can't because there's a kernel check. That's called mount to revealing Which is verifying if you the prokfs you have access to is partially masked or not And if it is partially masked it won't let you create a new prokfs And that's why we actually need our worker to not try and and create a sandbox for the processes And that's why all the processes are seen inside the build kit process namespace In conclusion, I mean there's no Resolution to date we we chatted about it with maintainers in this issue extensively and there's no real way to do it There are potentially multiple mitigations The one we use we're using for now is we're making sure that our dogfiles are not starting demons in the background Or if they do we explicitly stop them. So that's easy enough Something that could be interesting in the future is Kubernetes expose a notion that is called prokman type where you can tell that a specific container Will have a prokfs that is not masked so fully accessible Which means if you remember that mount to revealing will be okay And we will be able to create a new prokfs for our build step inside the sandbox inside the container Sorry, so that's extremely promising But it's the feature has been in alpha in Kubernetes since 1.12. So we're not exactly sure if it's going to To be G at some point. We could also use jobs for builds where we use Where the build key demon would be used for a single built which would be which would avoid this issue of course, I mean, there are some security limitations because What I didn't see what I didn't show you before is if build kitty can be used to run multiple build step at the same time So inside a build kitty demon you can actually have processes for different build step at the same time in the same process In the same set of processes and of course it could lead to processes a build thing process from all the builds so not too big of an issue but still not ideal and This gets us to the last and third issue, which is the most complex one Okay. Yeah, so our last talk was at cube corn was ghosts in the runtime. We like ghosts. So this time ghosts in the file system So we're just going to build a go program. So here. It's the local volume provisioner fairly straightforward We clone the repository Check out a specific tag and then run go build What could possibly go wrong? Well, we get a compilation error consistent read redeclared It's clearly declared in two files here read dot go and consistent read dot go But the strange thing is When you build this yourself on your laptop or in a docker build not in rootless, it works perfectly fine So what's happening? Let's have a look at the directory contents. Well indeed in this vendor directory. We have both files consistent read and read so The compilation area is normal at this point, but how did we get in this state? Because in the master branch, you just have read dot go and at the tag we have just consistent read dot go So we clearly shouldn't have these two files at the same time So we check at each layer The get clone shows that we have read dot go. That's expected the check out at the tag shows that we have only Consistent read dot go so that's fine too, but we've already seen that in the end at the next step We have both files for some reason. So something's wrong with the file system here. What's what's going on? So we use overlay FS as our snap shutter for for builds. This is a fairly common file system to be using in this situation and the way overlay for FS works is that it's what's known as a union file system So the idea is you have a set of directories that are the base and you want to expose a mount point Where changes can be made but without affecting the the original files And so what overlay FS does is it has an intermediate or what we call the upper layer which? records the changes that have occurred on the file system that's exposed to to processes through the mount point and So for instance a file that's deleted will be marked in the change layer the upper layer as a tombstone file So that's a special character device file major zero minus zero and that will serve to mask the file that's in the lower directories and And so that it doesn't appear in the mount point any longer if the files being removed You also have for instance directories in the upper and lower layers combined so that The the mount point exposes the the combination of the changes and the original files depending on files that are masked changed whatever So that's all very good Let's try and reproduce the steps of the of build kit here So first we unshare the username space. That's What build kit is doing that's because we're running in rootless We create some directories To to provision our overlay file system So we have layer one which is the lower layer where we're going to do the get clone then we have a The layer two which will be our mount points that we will expose and we have the layer to diff Which is going to be the changes that occur Through the overlay file system and so in particular when we actually do the get check out after mounting the file system The L2 diff directory is going to record the changes that are made So that's all very good We look at what we end up with in each of the directories and we see so layer one the check out We have read.go fine layer two difference. We have consistent read.go. That's expected That's the new file and in the mount point that's exposed We see only consistent read.go. That's consistent with what we've seen before but it's We haven't reproduced the problem at this point So now let's pile on the step three of our build Where we list the directory contents because it's the equivalent of when we build and we get compiler error So we unpound the previous overlay file system. We provision some new directories for the The difference layer for for our step three the layer three directory Which will be the mount point that we expose and we mount this so one thing to note here is that in the lower directories We actually have two directories because we have first the changes That were made by the get check out and then we have the base which is the get clone And we list the the files and we've reproduced the problem we We see both files so clearly there's something wrong here So if you step take a step back and look at what we've done So we have a layer one where we do the get clone We then have an over layer fs where we do the get check out and so we see that In the difference layer we have consistent read.go and it's exposed through the mount point We've then piled on step three So we have an extra difference layer, which is not too important here And we have the mount point which is exposed which shows both files, which is our problem So at this point we begin to suspect that the problem is somewhere in the layer two difference Directory something's going on here and indeed one thing to note. I mentioned earlier tombstone files We've removed the read.go file and yet we have no tombstone file in the listings So where is it well, maybe we've missed something and yes, we have so in overlay fs There's actually an optimization For directories, which are where all this contents have been deleted and But the directory still exists. You have an opaque flag basically it avoids over layer fs from having to recurse in sub direct in under Layer directories if all the contents have been suppressed It's an optimization How does it work? so basically in our case, there's The opacity that the opaque flag has been set on the layer to diff directory and that is why we're not seeing Read.go any longer in the layer to Mount point But how does it actually work? So what's done is that overlay fs sets an extended attribute called trusted overlay opaque on the directory that it wants to mask Or it wants to mask the lower layers So can we see this fire this extended attribute? So we do the operation in the username space get file attributes and we've got nothing so that's interesting because How come in step two then? Read.go is ending up being masked If we can't see the extended attribute it shouldn't work Now if we rerun the same command but in the host username space the initial username space We do see the extended attribute so it is being set but that also is a bit strange because Trusted the trusted namespace of extended attributes is actually subject to Permissions you can only set a trusted extended attributes if you are Capsis admin you have the the system administrator capability in the host User namespace, which is not the case in our overlay fs sequence here in rootless So we shouldn't be able to set that extended attribute and yet we have So we have a bunch of mysteries how come trusted overlay opaque is being set because we shouldn't have the permissions to do so and When the directory that is flagged as opaque is Mounted as an upper directory The problem doesn't reproduce, but when it's a lower directory it does reproduce so at this point we Resort to kernel function tracing we rerun the git checkout step with kernel function tracing And we look for the operation where the extended attribute is set and there we realize that a Function that's being called is vfs set extended attributes no perm and this makes us suspect that well The credential checks are actually being bypassed in this scenario for some reason which we don't quite understand So looking at the source code and the the the git commits We realize that on the kernel that we're using and I'm going to kernel there was some work done to make Overlay file systems work in username spaces and the change that was made was that indeed the Credentials check on setting extended attributes for trusted opaque overlay and For removing them were bypassed. So you don't have a credential check But it's also interesting to note that there was no change made for the get so we still Don't quite understand certain things we do understand that We're having the attribute set But we can't read it So that's why in step three We see both files because we can't read the extended attribute. So the opacity is not being on it fine, but in step two Why don't we have the same problem because we can't read that extended attribute either and It turns out that well if we think about it fast systems do a lot of caching So maybe if we drop the caches, we'll see something different So again, we reproduce our case. We reach the point. This is step two We reach the point where we list the files and we see consistent read.go only so Opacity somehow is being respect is being honored even though we don't expect it to be We now drop the caches and learn behold the two files are there So in fact, what's happening is that in addition to the extended attributes there's an opacity flag that's being set on the directory entries in the file cache and that so long as that The de-entry is in memory and the file doesn't the directory doesn't need to be re-read from disk We're able to honor the opacity so in short the kernel we're using added a patch to make user overlay FS work inside username spaces But it was a partial change. It only changed the set and the removal of the extended attribute and Thanks to caching. Well, sometimes the opacity is honored and sometimes it isn't which leads to some rather interesting behaviors Now the nice thing is that actually in kernel 511 New option was introduced to overlay FS mount options called user xh and what this does is it changes the namespace That's used for the opacity and other overlay FS Extended attributes and it makes it possible for any user to set The extended attributes simply because they're not any longer in the trusted namespace The other nice thing is the build kit actually the overlay implementation adds user xh support when it's available and so all we had to do was wait for kernel 511 to be available for our distribution for Ubuntu and Then simply roll it out to our nodes renew our nodes and from that point it just worked so at this point we've solved I Think pretty much all our problems with building images and so we have some pretty good results As long as said we you know right from the the get-go we had something like 80% of the images were actually building fine with build kit in rootless mode and Really it was just a matter of getting past the last few hurdles I'm starting with a mono repo for building images dedicated to building images was clearly very helpful here because it really you know Sifted through all the problems very fast So it allowed us to decommission our dedicated Docker runners Giving us you know easier node lifecycle management the the developer experience team no longer needed to manage a dedicated set of Docker runners and things like that and It helped us get native multi-art builds Because well, yeah emulation is way too slow. So currently we have several hundred distinct images that are being built on Cube Or I mean all our images are now built on Cube anyway, and so this system is now handling More than a thousand builds a day perfectly reliably and we're really happy with it So our messages are that build kit, you know, it's really good. We we've had a very good experience with it It gives us remote builds. It gives us multi-art images really easily Rootless is or was a little bit bleeding edge But you know with the changes in kernel 511 over a fs in user name spaces has become really Very usable. Well, it works fine I think really the only Problem we still have slightly that you know that could affect us is the process Sandboxing and so I think at some point we'll have to check out the prog mount unmasked option that Laurent mentioned And that's it. Thank you


President Mohamed Abu-Hari on Monday nights received the national leader of the All Progressives Congress, A.P.C. Ashwajubullah Tinubu, and pioneer interim chairman of the party, Chief B.C. Akhonde, at the presidential v-line, Abuja. The visit held at the president's official residence. They both met the president to pay condolences to him over last Friday's air crash. The crash claimed the lives of Chief of Army Staff, Lieutenant General Ibrahim Atahiru, and 10 other officers and men in Kaduna State.