Hi there, it's Sandy Alnok and today I'm going to talk about picking a lighting direction, choosing where the light's coming from. And what I'm going to do in the cards I'm going to show you, I'm going to paint one and then I'll just show you two others that are finished. I have chosen the quote unquote wrong direction for the light to come from. Because on a figure like this, any kind of figure that's a character or something whether it's an animal or a person, you generally would want the lighting to hit the face. So you want the lighting in this particular case, you would want it to be on the right or above or something so the rabbit's face would be in the light. But there are times when you find yourself having made an accident by eliminating a decision like you just didn't think about it. If you were deciding you wanted to create this whole background and the lighting was from one direction and then you put a figure in it later on in the drawing, your figure requires somehow different lighting. You need to have them be the same, they need to match. In this particular case, since I knew that I wanted to consciously make the choice to use lighting coming from the left so only the back of the rabbit is going to be lit up, then I could make some choices about what the scene would be. So I drew a straight line so I have a house in the back, some bushes behind a sky and then some snow that's going to have some light cast from the house out onto the snow that the rabbit is walking through. That tells a story so there's a reason why when I end up having to paint the face really dark, there's a reason for that as opposed to what we sometimes do is just get to that part of the drawing and we're like, I don't know what to do here and you just throw something in there. So the things that I'm trying here are to create enough darkness in the scene outside so that it's definitely night, that he's coming out from an area that's lit, but everything else has to be good and dark. So I'm trying to create a transition a little bit in the bushes. I don't really have a transition going elsewhere, but if I can just achieve that in the bushes where the right hand side is darker and that left hand side stays a little bit on the paler side, then when I try to create that whole look that there's light coming from the house that the rabbit is exiting, then it starts to feel correct. Now, one of the reasons why this whole thought even came to me is that right now I am working on putting together another class and it's going to launch on Black Friday. I'm very excited about it and in that class we're going to talk about shading. And it's going to be taught in pencil, but you can think about it in terms of any medium because I'm just talking about the lighting. We're taking out the factor of color. We're not going to use paints. We're not going to use markers or color pencils. Just going to use a graphite pencil so we can start to see lights and darks because that's one of the areas that I find most people really struggle with and it's an area where I feel like I've got a lot to say. So I'm creating a 10 lesson class that's going to be level two. It may look difficult, but it's not as difficult as you think. So in a week that will be coming up. So on Black Friday you'll see a video about that and a preview of it, etc. So get yourself excited about that. And I wanted to put this video, by the way, in one of the lessons in which I'm talking about what to do when you end up with inconsistent lighting. So in this particular case, just changing the storyline is one of the things that can help you to figure out how to change the lighting. So if I were to just have this little guy walking through the forest and needed to have some backlighting somehow, it would have to be the moon. But in this particular case, since I created a structure just by putting a line there, then I was able to create a reason. There's a reason. There's an actual light source coming out. And there's also, in addition to the rabbit being dark, the whole face of the rabbit and the back of the ears being in the light, the back of the feet being light, there's a streak going across underneath of the feet of the rabbit. So there's one line that is the shadow of the rabbit itself. And it's just a thin line. I used Duochrome Cabo Blue with some Pains Blue Gray for all of that snow. And I made it a little darker in that line coming out from the toes of the rabbit so that that's the shadow. And then there's a white streak on either side of it. Because when you're doing lighting like this, it's really important to really emphasize that light when you're creating something that's all about the light. And this particular painting is all about the light. Everything in it depends on knowing that that is a house on the left that is shining a light that the rabbit is walking out of. Now, I am not going to say anything about why this rabbit is leaving the house with a full sack of presents. Maybe he's coming out of his own house? I don't really know. I would think Santa and Rabbit would have to have left all the gifts under the tree. But, you know, that's the part of the story I haven't worked out yet. So you can tell yourself whatever you would like to explain that. So here I've put some shadows a little bit onto some of the fur and on the bag of packages. And I'll add the first layer into the jacket and just make that an anthraconoid scarlet type of red. And then I'm going to keep adding to it. I'm going to keep darkening colors because I need to emphasize that white. I need the white edges to be nice and crisp and white. And I need the other parts to start to really fade out. It's scary to get to that. I understand that for a lot of people. Like trying to go from a very light to a very dark is something lots of folks struggle with. But that's what's going to give you realism. It's why I did that class because a lot of people just look at my work and they're like, how do you do that? How do you do the lighting? And I'm hoping that these lessons that I'm in the middle of right now, I'm hoping that all of this is going to get the point across and that it's going to help people in more than one sector too. I was debating whether to do one class for Copic and one class for watercolor and one class for pencil and everything.  And I figured I'm just going to take out color and we'll focus on pencil. But can you see how the lighting is already showing up? It's highlighting the back of the rabbit and that area around the building, which is just a line. I didn't even bother drawing in a whole lot of a building. Just give it a structure. So there's something that is casting that light that the rabbit is walking out of and adding darker and darker color. Now, watercolor always lightens about 30% or so. So give it more, but you can also add more layers. And I'm just drawing it in between all these layers so I don't get bleeding across all of them and adding more darks because those darks are going to be what makes the lights in the highlight areas look like highlights. You may have also noticed that I painted some yellow ochre over top of that package bag of packages. Why can't I say that today? The bag because I decided I didn't want it to just be a gray bag. And which would make it a white bag with gray shadows. I wanted it to look more like a cloth, kind of a brown bag. So I just went over top of it. Easy enough to do. So the last thing to do then is to add snow because all my cards in the winter time seem to have snow. Everything for Christmas is ready to go. And I kept it really light so you could really see the image. I mean light in terms of lots of empty space in between. If you're trying to cover something up, if your painting didn't come out great, if your coloring didn't come out quite the way you want, then add more snow. And then nobody will notice any of those parts. But this is going to give the whole thing a nice airy kind of a feel. And that's what I was hoping to go for. And then I used a different one of the new stamp sets. A little mini set that has a very tiny mouse hanging up a stocking and a bird with a stocking. And used that on the inside of the card. Now I told you there were other cards. This one I painted Cabo Blue, that duo Cabo Blue in the background. And then I mixed kind of a grayish brown color to do the negative painting in those trees. And I left all that white along the highlights on the rabbit and the tree. On this one I did Moon Glow and Pains Blue Gray to do the same kind of a thing, but I had to keep darkening the trees. It wasn't working well. And then I used just some neutral tint for the rabbit and everything else. Leaving those whites white.  One last thing before I go, I wanted to say thank you to all of you who left me feedback on my video the other day when I was talking about why I blog. And I also have some more feedback I would like from you. There's a survey link in the doobly-doo. It only takes a few minutes to fill out, but I'd like to gather some information about the community. And I'm planning on making some changes, doing some exciting things in 2022. So I would love your input. So check out the link down there and I will see you again soon. Bye.


Now I'm going to deliver blood to my upper limb. I'm going to deliver it using this heart on this side. And then I'm going to bring it back using this heart on this side. Do you have two hearts for those two jobs? No, but it's going to look like a spaghetti mess anyway. So we might as well separate out our arteries, which are taking blood, feeding the limb and our veins, which are draining the limb. So let's start with our arteries and let's just clarify that we're doing them on each of my different hearts. Keep in mind also that right side and left side are different in the arteries, so it's not symmetrical. So I'm going to highlight the pieces that are unique. First of all, who's the vessel? Blood is going to come out of the left ventricle no matter what and it's going to travel up the ascending aorta. It's going to start into the aortic arch. And we have two branches that we can go to if we want to feed the right arm or we want to feed the left arm. That's an L. If we want to feed the left arm, we're going to take the last branch and that is the left subclavian artery. And that's all I'm going to tell you about the left side. Everything else I'm going to do is going to happen on the right side. So as soon as you see me draw the right subclavian artery, that's the point at which left side and right side are now identical. But my first branch off of the aortic arch is the right subclavian artery. And I'm going to highlight, this is going to be kind of stretched out because we don't actually have a body there, but this is the right, excuse me, this is my brachiosophallic artery. And my brachiosophallic artery stays brachiosophallic until we have a branch.   Where's it going, doggies? It's going to the head. So I'm going to draw it in green so that you know that this isn't a vessel feeding the arm, but it marks an anatomical location where our vessels are going to change names, which is why I'm including it. It feeds the head, and this is the right common carotid artery. So the right common carotid artery branches off. At the point where the right common carotid branches off, the vessel now has a new name. Now it is called the subclavian. Ah, the right subclavian artery. Check it out, here's my left subclavian artery, here's my right subclavian artery. Now we're symmetrical on the right and the left sides. The right subclavian artery is so named that you can tell you reach the lateral border of rib number one. I'm going to indicate such, it is a junction, but it's an anatomical distinguishing mark. It's not distinguished by a branching vessel. So it's the lateral border of rib one. Oh, lo and behold, fascinating amazingness. We get a new name. Now this vessel is called the axillary artery. And the axillary artery travels until it reaches another structural location, and this one is kind of odd. So it's not odd, it's actually kind of exciting because it marks review of a muscle that we all know and love. The axillary artery changes when it passes the inferior border of Thierry's major. So if you're in Caverland and you identify Thierry's major, that's the place, that's the place where the axillary artery is no longer considered the axillary artery and instead is the brachial artery. So let's make sure that I've got my right colors going on. Now we're the brachial artery. No need to write artery out, we can write A. All right, the brachial artery has clinical significance because that's where you listen when you take someone's blood pressure. So the brachial artery exists all the way down the arm until the cubital region, and the cubital region is basically your elbow pit. It's the anterior portion, here that's my, see my elbow, hi, it's the anterior portion. It's where you give blood in the cubital area. So let's just make a note of that. This is the cubital region or area of your arm, and at the cubital region, the brachial artery branches into two other arteries. So we'll go like that and then we'll branch out because it's always good to branch out. So is it clear that I'm saying that the cubital region is where the brachial artery branches into two other arteries? And if you had to guess, you are now in the forearm. So go ahead and guess, like what would be the easiest name for the lateral branch of the brachial artery? How about if we just call this thing the radial artery? Oh, but of course, and if the lateral branch is radial, what's the medial branch? The ulnar artery, okay, super cool. Clinical significance, radial artery is where you most commonly take a wrist pulse. You can feel the pulse, feel the pulse in the ulnar artery as well. You have to be calmer and more sensitive. It's always good to be more sensitive. I think that's awesome. You now have blood to your hands. There are other arteries in this mix that are feeding specific places.  And many of them have names and feed specific structures. So please understand that I'm leaving out a whole bunch of details, but I'm giving you this big picture. You can imagine holy capillary bed has taken place. We've exchanged nutrients, we've taken out the oxygen, we've dumped in the carbon dioxide, and now we need to come back. If we're going to come back, let's come back through the veins. The interesting thing with veins is that they are highly variable, which means, ugh, it means that the anatomy of veins differs, especially superficial veins. It differs significantly from person to person, which makes it, of course, a little more challenging to pin down details, like there's more figuring out depending on the person that you're dealing with. So keep that in mind. The other interesting thing about veins, and this is true for arteries as well, but veins, we have super superficial veins. Like the veins that you actually can look on your arm or your arm is a great place to look for superficial veins, they're close to the surface of the skin. And if you think about that, I mean this is a good conversation to have. Why are your arteries, you have some arteries that are deeper than other arteries, and you definitely have arteries that pass through a relatively superficial area, but in general we try to avoid letting arteries come to the surface of the body. Why? Well, your artery is the closest to the heart as far as the pressure of the heartbeat fueling the blood flow through the artery. So if you whack an artery, you're going to have the heart actively like pushing blood out of that artery. It's that spurting, like if you see somebody who whacked an artery, the blood doesn't just flow out, it spurts out, and it's spurting out because your heart is pumping it.    What was my point? Superficial veins versus deep veins are important, and my little patterns, this pattern is pretty consistent.  So I want to talk about superficial veins first, and I'm going to make my superficial, that says superficial. I'm going to make my superficial veins lighter, and I'm going to make my deep veins darker, and that's just going to be my little color coding so that you know who I'm talking about. We got to start at the heart with the arteries. We're going to start in the limb with the veins, and I'm doing it that way, even though it's hard for me to visualize this, it's hard for me to start drawing it because I want to start hearing go backwards, but blood is flowing from your hand back up. So there are two superficial veins that I want you to be aware of. There are two deep veins that mirror these guys. We do have a radial vein and an ulnar vein, but they're deep. So these guys are superficial, so don't be distracted by that. I'm going to keep my cubital region about the same level so that you can have a perspective of where these things are happening. And the lateral vessel, superficial vein, is called the cephalic vein, and it's a superficial guy. And it travels all the way up, and I'm going to finish my adventures with the cephalic vein because it's actually going to, where it dumps in, I want to make sure that I get, I can't decide where I want to put that until I get the rest of this filled in. The medial superficial vein is the bacillic vein. And remember that we also have deep veins. We have the deep radial vein and the deep ulnar vein. The bacillic vein travels medially, and there's another superficial vein that's super important and significant, and that is the median cubital. The median cubital is a branch that connects cephalic and bacillic, and there's another superficial vein. The reason why this one's important, and these three are important, is because the median cubital, cubital vein, is a site of blood draws, a common vessel to try and poke to get blood out of you. When you give blood, most often they're poking the median cubital vein. You can see it. It's also in your elbow pit, the cubital region. That's a nicer way of saying elbow pit. All right, so all superficial. Now, I'm going to draw a deep artery in here. Excuse me, a deep vein, because there is a brachial vein. I want to make sure that I get this. We'll do it like this. I'm drawing like dark blue, and the brachial vein splits, and it actually becomes two brachial veins, and I think that we do need to know the brachial vein, and it's darker because it's deep. The reason why we need to know it is because the place at which the bacillic vein joins the brachial vein is the mark of a new vessel, and the new vessel is what would you guess? Brachial vein joins with bacillic vein to form, who are you going to guess? The axillary vein. Now, the axillary vein, we know that after brachial vein combines with bacillic, that's when it's named axillary vein. The axillary vein, the cephalic vein, flows into it, and I'm not going to put, we don't change its name. This whole thing is the axillary vein. So cephalic flows in at the point, exact same point of this whole thing at the lateral border of rib number one. It flows into the right subclavian vein. Now, I'm completely out of room. In fact, I might just get rid of these guys. Goodbye. And you already have these notes written down, right? So I'm just going to get rid of all that. So I have a little more room because we now have the right subclavian vein, at the point of, okay, look, this is why this is important, and I'm completely out of room. So here is a branch of the jugular vein. Ay, ay, ay. Ay, ay, ay. Oh, I see what I did. The jugular vein flows in and joins the subclavian vein. And when that happens, it both become the right brachiosephalic vein. And the right brachiosephalic vein, hold on a second, I got to write all that down. The jugular vein from the right side of the body flows in and joins the subclavian vein and becomes the right brachiosephalic. The right brachiosephalic vein, it does. The right brachiosephalic vein and the left brachiosephalic vein, that means brachiosephalic vein from the left side, which has the same pattern. They join together to form whom, whom are we now? Now your inferior v... superior vena cava. You need to draw this yourself again. I wish I would have spread this area out.  Blood is flowing back to the heart in our veins. All right, I think you guys are fantastic. Now let's go talk about the brain.


The hundreds and hundreds and probably thousands of man-hours that went to put this air show together. Let alone have 160,000 people show up at your doorstep, you know, now it becomes game time. It's a special one because it's a centennial of marine aviation.  And they're part of a team that's second to none. So I encourage the folks who talk to the Marines here that they see it at the air show. They are the pride and joy of the Second Marine Aircraft. We're calling it Celebrate the Heritage, which is celebrating 100 years of aviation in the Marine Corps, which is huge. It's very hard to describe what it feels like. It's the awe in America. I mean, it shows the innovativeness of Americans and it shows where we've gone and where we're going to. The way the Marine Corps maintains it operates and why we operate.  The only reason the Marine Corps has an airplane like that is to support that guy in the crowd. We've done it. Libya, we did it. Afghanistan, we're doing it.  Stand by, ready to go take the fight to the enemy wherever they are, if need be.  It's eye-opening, you know. I said the interaction in the Marines and everybody get together and have a good time. We've got a lot of respect for everybody out there. To integrate that with the local community is great because the local community has been a part of our traditions for 70 years here at Cherry Point. And really, the first thing I do is I turn around and look at the crowd and look at their reaction, and you know it worked. The feeling of energy and patriotism and freedom is just huge, and you just got to be here to feel it.


One of the biggest problems we have in the tech community is tribalism these people that identify with a certain team That's a certain operating system or a certain Linux distribution.  You know, you need to quit identifying with the software you run It doesn't make any sense, right?  I'm saying the software is bad and you don't need to defend the software It's an inanimate object. I promise you I didn't hurt the software's feelings


From the annals of the throbbing history of the West, the daring courage, conquests and achievement of frontier fighters. Hand in hand with the brilliant record of covered wagon and stagecoach and spanning the continent goes the glorious epic of the Wells Fargo Express. The Indian by 1854 was no longer the greatest menace to the stagecoach driver. Instead, hundreds of miles of desert and mountain were infested by highwaymen. 25 miles from San Francisco, an old concord coach with eight passengers, the driver, guard and a treasure chest, was thundering down a narrow mountain road. Driver and guard were complementing themselves on two things.  Sure spinning along this trip ain't we Sam? Yeah, Bill. Looks like I ain't gonna have no use for these shotguns of been holding the groceries ever since we left the mine. Yeah, ain't that force less at its best for you. Just look at the six of them go spanking along there. Hey Bill, look, coming right at us. Hey! Hold up! Hold up for a fire! Hold up! We got you covered! They'll help me! Put down your guns, Sam. Move over it! Go on to treasure chest!  Are we going to be held up?   Yeah, here's your treasure chest. A good hold this time. There's $25,000 in it. Three out of your search for passengers. You two tear up the cushions. Make sure no dust or nuggets are hidden.  If your face matches that voice of yours, I'd say you will...  All right, man, give me the valuables. I'll get back on the coach. All that you remember, we got a bead on you. It might be wise for you there, Mr. Driver, not to find a name to match my voice. All right, let's go. Oh, I have 2,000, my folks. It's getting so that as fast as the miners get the gold dust into the treasure chests, the highwaymen are just around the corner to take it away. Well, when we pull into San Francisco, there's going to be the very net to pay this time. Express companies are going to have to pay up or get out of business. A wall? Mr. Fargoon, his office now? Well, yes, Mr. Royals, he an office. Everybody an office. Everybody walls? Everybody from mines and business houses. All I say, everybody's every one. Well, I said they're talking big business. Oh, not only big business, but big talk. Good day, Mr. Fargoon. I'll talk this over with my partner gentlemen. Good day and thank you.  Henry, a most astounding offer was just made us in your absence. It's a good chance for us to get at least 60% of all the express business in the west. 90% these men say. Well, let's discuss this in my office. I'd like to have wild talk floating around in the general offices. Henry, one of our competitors' stages was held up yesterday morning. And from the treasure chest alone, the highwaymen got $25,000. Yeah, it's a nice little hall. Something's got to be done about both the highwaymen and the responsibility of the express company. What does this group expect? Wells Fargoon Company to make good the laws of every treasure chest? If we'll make good this powerful group of mine owners and businessmen will give us enough business to cover the entire west with Wells Fargo offices. Well, I appreciate the public's confidence in Wells Fargo. After all, we're a little more than four years in the west. It's the opportunity of a lifetime for any company. If we can see our way clear to do it. If we guarantee to make good lost by theft, that means that from the time our agent gives the consigner a receipt, that receipt is as good as our word. And our word will have to be our bond. Exactly. Well, the whole scheme is breathtaking and it's possibilities, but dangerous possibilities. It could be done if we had three things, improved and faster coaches, a treasure chest that only gunpowder could open, and guards who believed with us that the highwaymen must go. Now, men, Mr. Fargo and myself have agreed that you drivers and guards of Wells Fargo passenger and express coaches represent a new type of American manhurt. In your hands will be half of the treasure of the nation, as well as the lives of men and women passengers. Mr. Fargo will give you your instruction. Each coach will have two guards. Each guard will have one shotgun across his knees, another that can be snapped to his shoulder on a split second's notice. He will also have a brace of pistols. The treasure chest will be a one-half inch cast iron, heavily padlocked and placed in a larger wood, brass-bound, heavily padlocked outer box. Your orders will be, first, think of the safety and well-being of your passengers. That means throwing out your treasure chest on demand if it looks like a shooting affair. If it doesn't, it's your lives against the highwaymen. The future of Wells Fargo, and the growth of the West is in your hands.  I'm glad we're just 14 miles from Placerville. Sure will be a comfort in sight to see the lights of that town twinkle enough to us friendly like. I should make it there by 10 o'clock. I got a funny feeling middle of my stomach. Ain't gettin' squeamy speed, are ya? Not after four years of holding a shotgun in your hands? Well, it ain't squeamish, because I ain't up here. I hear something called by a high-flute word for reminiscence or some such thing. If we ain't been held up 50 miles out of Placerville, we ain't gonna be, then. Triggers in this part of the country ain't heard yet about our cast iron treasure chest. Don't think it's easy pickin' us. Well, I'm ready for them if they come. Lem here tells me we'll be comin' up to the top of a grave. Once down, we'll give the horses their head and be into Placerville before you know it. Is that? Sure. Make anything out, Pete? Can't see a thing, but I'm here in a tube. Well, one more mile and we'll be at the top of the grave. And then, if it's banded, we can give them a run for it into Placerville. Come on, Bet. Come on, girl. Get on. Get on. Make anything out yet? Lem here, it's a party of men. Pack news with them. They're old cloaks. Hi, Wayman.  And not before I try and get to the top of this grave. And they're the king and on it. I'd just like to let them have what both fells. We got eight ladies and four men in that coat, huh? Don't think I can make the top of the grave in time. Maybe I better slow down before the open fire. And we gotta think of our passengers. Ah! Ah! Well, what's wanted? Throw out your treasure chest! You there in the coat ain't got nothing to worry about as long as we get the dust in that box. And you don't pull a gun on the eye. You can keep your personal valuables, too. All right. Here's the chest. It's supposed to be $50,000 in that.  I met up with you once before, Mr.  Oh!   Eh, if you've got enough padlocks on this. Eh, eh, she comes now. Open now. That's a good moon light enough to see from it. Well, I'd be... There's an iron chest inside this one. Creepers never get this open without blasting it. Whales Fargo Express, huh? Well, I have to remember to carry gunpowder with us next time. All right, Tully, you get that chest on my horse. Remember, anybody fires one shot and we'll come back and wipe you out. All Ian, let's go! Now we know who did the holding up. And as soon as we get into Placerville, we'll get a Whales Fargo posse out after that gang. Yeah, I mean, better all get back into the coach. We're gonna string it into Placerville. Come on.  And I'm quickin' my job as driver for Whales Fargo and joinin' with the sheriff and the posse.  I figured, Sheriff, that it would take rattlesnake and his gang some time to get gunpowder. Blast the chest open. They'd be leavin' some sort of a trail. Thank you, right. And iron chest is the thing that's gonna hang more highwaymen than anything else. And most of the time, we're gonna recover the treasure, too. This is where we was held up here. Gettin' lighter all the time. We reckon we can pick up some sort of a trail. We reckon we will. Here's the marks of the coach and your horses. Here's their horses. Whoa! Some of rattlesnake's men went this way and some the other way. It's funny he should be dividing his party. Sheriff, that was just to throw us off the track. They went off into the brush but came back into the road again down here. They all went off each. Must've been to their hideouts. All right, Dan, you take the lead. Everybody pull it in. Rain up, men. I hear voices. Sounds like a whole parcel of men. It's rattlesnake. They've blown up the iron treasure chest.  Dismount, men. All on hands and knees through this seed brush. When you're close enough, shoot, shoot, a kill. This sure enough is a hole! I'll call out to rattlesnake and then you men start shootin' because they'll all do the same. Now, rattlesnake dick! Surrender in the name of the law! Surrender in the name of the law! No! Well, Ben's done that, critter.  And Wells Fargo got their treasure back within 24 hours. Well, I reckon there'd be more respect now for Wells Fargo's stagecoachers and for the law. The West, I hope, will be a might safer for everybody that wants to come out to the last open frontier. And so was established the reputation of Wells Fargo, a reputation that was to go down in history as having helped tremendously in making the West safe for the pioneers for civilization. Each stage driver, every agent who guarded passengers and treasure chests, were sworn servants of law and order. Each one a true frontier fighter of the Old West.


but it's really really successful in that case where you might be in close quarters for a long period of time and being able to talk honestly to people that are willing to hear it and not pass judgment right away is huge for resolving those conflicts quickly especially in a group that might be like working together on a really important project in a short period of time it lets you get through that very quickly we've used that on we do weekend retreats basically that are kind of about emotional intelligence at base I can talk more about it maybe out off the mic and things like that but that is a technique that we use there to deal with inter-conflict while we have a very intense emotional like 48 hours together do you find having elastic being remote if you're a partner that sometimes having two people in the same room is more intense? yes it does make it easier harder it depends on the problems it is nice in the way that I can step back and sort of process something and recognize if I respond right now it's not going to be good I need to go take 10 minutes and come back to it without a weirdness in the conversation you don't have to explicitly state that you can just walk away from the keyboard but it does make it harder to for instance say you know you're three hours behind me I really need to talk about this but you're not available so we can't talk until Monday and now I've got to deal with the burden of carrying this conflict until Monday right when we can actually sit down and have the conversation so some of one some of the other yeah so at elastic you guys at least have told everybody else and I believe to be true that you are like a truly blameless or like the company believes in this or a company that might believe less than that how might they transition and either from like the top down or from like more grassroots type thing yeah the blameless and just culture which are totally worth looking into the resource here I think is about patient safety like there was a talk last year at Dev Ops Days Boston that talked about the just culture at Brigham and Women's so that's a really good place to understand this so applying it is tough because it's cultural change like anything else so you run into a lot of the same problems when you say how do we do a Dev Ops initiative at our group that hasn't never done that before because there's a lot of changing hearts and minds that comes into place I would recommend starting at something practical so thinking of blameless postmortems for instance and you can apply that specifically to that condition or whatever you want to apply it to but being able to start at that practical non-personal level might help move into the more personal side of things and say hey we do this for our incidents can we do this for this argument we're having right now and try and retrospect what's going on I hope that answered and maybe we can yeah right yeah all the companies that may not have the same blameless culture you have we do blameless performance but in those blameless performance or retrospectives remember what I call them we're often looking for root cause in a complex system that is like what would you say to somebody that says hey but I think that there is a root cause to your complex system so what would I say to someone who says there is a root cause in a complex system first study a lot of John Allspaugh who is one of the biggest and most prominent proponents of there is no root cause and furthermore is it human error is a myth is the other side of it too as far as how we study reactions of systems so there's again a lot more where I would probably start having that conversation I think is to address what you're actually after when you look for the root cause I think like I said earlier when you're looking for root cause you're more often looking for closure on the incident rather than how can I fix the problem you just want something to say oh this is what caused it now we no longer have to explore or learn from it so if you can get to the point where that statement is explicit in the group and you sort of understand that collectively I think gives you the best way to move past it from there I think that's it great thank you I just wanted to announce that after three so we have one more talk today and then at three we all we're all going to go to the keynote room Metcalf Large and we're going to do like a raffle so three o'clock raffle in keynote room Metcalf Large hey hi Paul hi Mary do you give this talk often? so I've given this talk in do you know what a night format for it's like a five minute talk and it's one of those four minutes I've done that way and jumped into open discussion afterwards this is the first time I've given it so I have here nice to meet you this thing is exactly the way it was oh that's great alright good afternoon we're going to begin our next talk which is you can see this title is singing a missing role in agile team and of course our presenter is Deepak Kohan who is an Associate Manager in Quality Engineering as is stated on this slide so we'll go further ado alright am I audible at the back okay so my name is Deepak I'm an QE and I started in 2007 this was the year when the global economic recession also started incidentally and this Selenium and the WebDriver projects also merged which would then go on to become the de facto web testing standard so I did five years in PTC which is parametric technology and before coming to Boston I was not aware that PTCs had quartered 19 miles from here and after that I did six years in Red Hat and I've been doing the same thing for last 11 years like accepting user stories, writing automated test cases and finding bugs so you'll say it is a monotonous monotonous job right but if you have an eye for it it throws a bit of nuances and challenges every day alright anyone in this room whose responsibility is the software quality alright so there are couple of people who fell in this trap so this was already discussed in one of the earlier sessions that quality is everyone's responsibility and the agile recipes my talk is an agile recipe so it is only fair that I give you the disclaimer that the most important ingredient of any agile recipe is the people which practice that success recipe so you cannot copy a successful agile recipe and paste it on a different team and expect similar results so it can either do really well better than the other team or fail really bad alright so let's start with the heartbreaking story so has anyone seen this progress bar on a web page GitLab has it so it was back 4 years ago I was working on a user story and it was a ticket management tool which had let's say hundreds of fields and out of those hundreds of fields two were the fields which used to be pulled regularly because we had to show users the correct data a very updated data so what we used to what we used to do till that time was we used to discreetly update those two fields by pulling making two network calls in the back end of the user so in that sprint there came an enhancement request that you need to give users a visual clue that you are fetching the fresh data for those two particular fields so my developer friend used this JavaScript library I really liked it because it was on GitLab, it was on my favorite e-commerce site so I thought it was cool so as a QE I tested on different browsers because JavaScript and then I did regression test as well and one Friday morning Eastern time and unfortunately the Friday evening India time we released it and I had this habit like teenagers when they put their cute selfies on Facebook or Instagram they refresh the feed to see how many likes so after every release I had this habit of monitoring emails IRC as well as the to see whether these are not the actual flippings I just made them so there was a huge outrage so everyone said that this is a usability disaster this change this particular loader was reacting us from working and if we don't work on cases, Red Hat loses business so we quickly reverted the change and to bring back the sadness in the system and then we had a retro like every team we had a retro and out of the retro everyone said that this happens we could not we could not do user behavior we thought it was a good way of implementing this change but we could not so we actually brushed it under the carpet but then I did a retro with myself so I'll explain in another slide that the retro which you do with your inner self is the best retro ever so this was the struggle so this is the user support engineers in our case and this is the QE and this is the developer if you look at these three categories of people which two are more likely to go for a drink together or maybe share a laugh or go on a team outing like QA and developer because they have that team bond proximity and because of that proximity which is a psychological thing you cannot change it because of that proximity what QE does normally in agile teams is they test the change they don't test why did the why did the developer implement that code change so very existential question but QEs mostly fail to do that and more so in today's time when we don't have traditional QEs every QE is also a developer of some different kind so they are writing code day in and day out so once you see a piece of code on your developer friend's machine your mind becomes biased now you have three ways of breaking that code which you have already seen but it's not a black box to you now so there were hundred other ways of breaking that code which you will miss now then we thought about why don't we do one thing why don't we bring fresher perspective into the QE thing so same work stream we bought another QE but without making any capacity changes so there was user story one there was user story two and there was first QE and primary QE and secondary QE so primary QE will work on user story one for 50% of his time and the same thing secondary QE will do so because no two people think alike I haven't made it up it is neuroscience our mind is connected like our fingerprints it's different so different people perceive things differently so if you look at crowd sourcing testing platforms which have you know boomed up recently they exploit this thing so they take a website and throw it to 500 testers all over the globe so we thought that this will at least make some improvement in our process it did but there was still a gap because to work in this system both the QEs needed to attend all scrum ceremonies like daily stand up planning retro everything and the proximity challenge which the primary QE earlier had the same thing happened with the secondary QE as well right then we came up with a new thing called cynic the cynic is someone who is not part of this work stream a completely different person of maybe a different project or someone else on the bay right so what does cynic do so how do you the first thing is how do you find a cynic so these are the qualities of a cynic cynic should not belong to your work stream or project which means that he is shielded from the project related discussions right the cynic should be naturally a vocal and SRT person so we don't need to make any any effort on finding these people in red hat right and the third and most important thing he should be willing to help again in red hat you will find because people contribute to different projects so you will find cynics and then if you don't find then you have to incentivize right now what does a cynic do cynic is a very fresh and mostly end user perspective to your work because he has not seen the code he has not been part of planning or grooming or scrum so he is looking at your changes as a user and cynic is your accountability and if we go back to this line what we have to do is we have to nudge this person from here to slightly here and cynic does that he nudges the primary secondary QE towards the building the right product side now the take away is from this talk so if you look at what happened with after that incident that usability incident first we tried to introduce a secondary test then we identified the gaps then we tried to implement a cynic so we were constantly improving finding gaps and improving the system which is the essence of an agile team and then again the thing which I said earlier best retrospectives are the ones which you do with yourself right because the actual retrospectives are very formal the third thing which you can do is you can sign up a cynic you can find a person in your bay who is not part of your project and ask him if he can give 30 minutes per week to you and see if it works for you as I said this is a recipe it can work for you can work better for you than it worked for us because in our team we have this we have a UI work stream a mid tier work stream services work stream so the QE working in UI work stream automatically become natural cynics of because they are the consumers of the service layer right they become natural cynics of the service layer work stream so what you can do is you can find a cynic a very assertive and vocal person and try to sign sign him up for your project and some physical takeaways so if you look at this Tiffin carrying jute bag I bought this three of them from the mountains of Kashmir in the north of India so I actually want the feedback for this so for that I am incentivizing that's a nice bag so sorry maybe but right now I just want feedback for this talk so whoever uses this hashtag you can take the photo as well so whoever uses this hashtag in next 30 minutes and provides a tweet as a feedback and because I have bought them with my hard-earned money so I will choose the best three tweets and I will give away these bags any questions about the cynic as part of the team and that aspect to it but I noticed that there was a bit of a similarity between the secondary QA and the cynic because you are already pulling in an engineer from some other there is no similarity between cynic and secondary QA secondary QA is part of your work stream so he has to attend your scrum ceremonies but doesn't the cynic also have to so for cynic you have to the only interface between your project and cynic is the primary QA so just like code review you get you write code you get your code reviewed from your body developer right so let's say all the stuff which was developed came on the QA environment now I have to test it what if before testing I would do a cynic to get a fresher perspective 30 minutes for per split it's not much to ask for so my secondary question so if QA chooses a single cynic and like frequently demos with them after some point they are going to become buddies and so they are going to have to find a new cynic and so maybe Redhead won't have this problem but at smaller companies they might have trouble finding enough cynics to see if primary QA and cynic become buddies that's okay that won't make any difference because it does not put any bias in cynic's mind towards the work which developers are doing if they are buddies I think there is going to be a little bit of a but he is a QA right the job of QA was initially to start with to break the code right but then we went into simply rubber stamping the code we need if you go to Jim right the most common example of accountability partner is the weight loss thing right you start with weight loss you make a new resolution you start with weight loss and over a period of time you stop going to Jim right but if there are two people who are each other's accountability partners then there are chances that this weight loss resolution will go till March or maybe mid of April thank you any other questions feedback me thank you everyone thank you for your time I don't want to take them back because I have done shopping from Boston


Thank you, everyone, for coming. We'll get started, I guess. So we're here to speak about something big that's just landed a few weeks ago, is we are finally getting right packages. So we'll go to the usual, why do we do want to do that, and what the challenge is, and what we currently have, what's upcoming. We'll tell you about some of the issues that we are foreseeing. You will tell us about the issues that we have not foreseen, and all of us together will find a solution for making this work somehow. So who we are. I'm Pingu. You may have known me before. You may have seen me at the previous vlog, not the one last year. I work in the Fedora infrastructure team, and I'm leading the right-getting initiative. And with this, I'm working closely with Dominic, who's going to introduce himself. Oh, yeah. I'm the Fedora CI Objective Lead, and I'm also, I work for Red Hat. I'm a manager in the Fedora CI space, and also in the rail space for those who work for Red Hat. You may have already seen me. So this is all nice and easy. You know who we are. We know what we talk. But why are we doing this? Well, the answer is easy. We want to say ride to be more stable. We want to ride to be something that people can actually use on a day-to-day basis, and stop with the idea that ride, eat your babies, and you shouldn't touch it. And it's OK to break roll-hide. It's OK for ride to be broken for weeks in a row, because it actually is not. Just think of it as roll-hide is our master branch. There is a reason why roll-hide is our master branch. There is actually no real reason for that.  You have CI for that. You fix things. You want your master branch to be working all the time. So there is no reason for Fedora not to have this data as well. So we want to have a more stable master branch. We want to have compose working in roll-hide. There is frequent time where composites in ride are broken for days, if not weeks in a row. And when composites don't happen in ride, when the composites fail in ride, that means that no updates are getting pushed, which means that all the build that you have nicely been working on, all the updates you have nicely been working on, are not getting pushed to any of the people who actually run roll-hide. And we do have some of these errors, and there is even one thing in the first row here. Thank you, Kevin, for running roll-hide. So every time we break the compose, Kevin cannot get his latest package. So he's very sad, and that makes him know. And when Kevin is sad, it makes us sad as well. There is a side thing on this, is that if we make a roll-hide more stable, we'll actually get some benefits at Fedora branching points. When you ride the stable and composites, when we branch from roll-hide to do our beta release, that branch is also going to be more stable. We will stop having to run and make heroic efforts to actually fix our branching model and get beta out there just to compose. And faster update release, that's basically what I was saying. In roll-hide, if you don't compose, you don't get updates. So we want to avoid that. There is also a side effect of this, is that when currently you dump something in roll-hide and that breaks the wall, you can go on your weekends and enjoy enough and or you can go on PTO and let people with a broken roll-hide for weeks. So we want to get a little bit in the mindset of, well, if you break it, you fix it. But if you were at Neil's talk yesterday, you may remember that he was pointing out that not everybody can fix things and that's where Alexandra is definitely right, where we want to increase the teamwork in Fedora with if you are working on something which is of significance importance and you don't have all the access that you need to, well, then you need to actually reach out to the people who are maintaining the part that you can't touch and work with them to make sure that the entire of your change with the dependencies is still working and passing. But if you do have the power, if you have a proven package, well, you have no excuse to actually fix what you break.  Most of our contributors, some of them working at Red Hat as well are volunteers. They are contributing to Fedora on their spare time. Weekends, evenings, PTOs. We cannot dump something in raw hide and let something else take their time off to actually fix what you break. There is no reason for me to push something in raw hide that breaks the world and let someone else contribute, who is a contributor who is giving the project free time to go after my mess and clean it up. So we have to be careful. When people say on the developer list, and if you follow it, you've seen that before, it's okay for raw hide to be broken. No, it's not. It's not. If you break it, you fix it. If it's okay for a part of raw hide to be broken, but that part of raw hide should not affect the entire project. It's not okay for me to break your work when you're contributing freely on your own time. So what are the challenges to make this happen? Well, the first thing is, as I've started, it has been a long process. We have been trying to make raw hide more stable for years now. We have been speaking about getting raw hide packages for a long time. So the first challenge that we were facing was just let's make it happen. Let's actually get raw hide packages. But we have a few conditions for that. We have a few requirements. We want to actually fit into the existing tooling. We don't want to reinvent the wheel. We want to create a new build system. We don't want to iterate a new body part. So we want to fit into our existing infrastructure. We also want to interact as little as possible the packages workflow. And that's something which is detourist because, as I said, contributors contribute under free time and the more change we impose on them, the more chance we have to drive them away or it's going to take them more time and it's going to be less interesting for them. So we need to be careful about the experience there. So we need a smooth user experience. That doesn't mean there is going to be zero changes. I mean, we are changing the way our packaging workflow works in federal. So there will be changes to the package or workflow. But we want to try to have them as minimal as possible. And then there is a big challenge as well, which is going to be also on a lot on Alexander and Dominic is the false positive side. How do we make sure that the tests are sending proper results, that we don't have package that are blocked by because the test is faulty rather than because the test is valid and the package should be blocked? That is a metaphysical question that happens with every NECI.  So how do we plan to do that? Well, we plan to do, to roll out in phases. We start with a smaller change and then try to go for the bigger one. We want your feedback as early as possible. Even when we did the change proposal to FaceCo, we did a call for feedback on the dev list and we actually got very good input there. Some of the inputs we already had in our mind, but they were not properly worded in the change proposal. Some of the inputs we had missed, so we added it. And we actually definitely want a polished user experience. So where we are, we are in the, we are currently getting single built updates in Rohite. So how does it work? So that's basically the entire workflow of it works. It's unreadable. I know it's meant to be unreadable here. Just the idea, you have, the user is on the right. You have 10 systems behind it. Hopefully most of them, as you can see, the user does not interact with them. This is the simplified version of it. So it's the one I'm going to run you through. When the Packager does a build, it lands into a certain tag in Koji. The tool called Robocignatory takes the build, signs it and moves it to a secondary tag. There Koji's body sees it, creates an update from it, waits for the CI system to report that the build update is tested and the test passed or the update is not tested and therefore it should not be gated. And if everything is good, it moves into the right build and you get your usual right experience. I'm going to be digressing a little bit from this workflow here just to go through the decision making. So how does the... Just to reiterate on that, like the first step, just to emphasize this part, the first step is one where you actually make the decision and the rest is automatic. So it might be complicated, but by default, these things will just go through. So no watching and chapering things through the process. Dominic, this is definitely right. It's all automatic. The other thing that to consider is what changes from the current workflow before getting is basically the second box here. We were using two tags. We are using three. We did not use body. We are now using body in the right. That's basically the amount of change we've done. So I wanted to go a little bit on how the CI system makes a decision, what can go in and what cannot. So we have the CI system that is running the test. The test lands into an application that is called ResultsDB. So before I go to this, I'm going to the three applications which are key and the three names to keep in mind are ResultsDB, GreenWave and RiverDB. These are the three items which are used to make a decision. So the CI system sends the test results to ResultsDB.  So every entry in the database is about an item that was tested, CodgyBuild in our case, and the test that was tested and the outcome of the test. As well as the URL which is in there when you actually want to send the user, well, it failed and this is the log where you can find why it failed. Then we have GreenWave which is, it's a decision-making, it's a policy engine. It goes through a set of rules and based on the results it finds in ResultsDB, it makes a decision. For this it has two source for making a decision. One is its own configuration file which is managed in the Fedora infrastructure which contains policies, rules that would be applicable to every packages in Fedora. So if you're at David and Tim Flink's talk yesterday about RPM Inspect, if one day RPM Inspect becomes stable enough and reliable and useful enough for the community to say, well, we want RPM Inspect to pass on every single package in Fedora, we will put that in GreenWave's own configuration file. Currently there is nothing in GreenWave's configuration file. Everything is in this gate, in the package specific. So you can have per package, you can make a decision per package, what are the tests that I want my package to be gated on? So which means that if you don't want your package to be gated on, you just don't tell GreenWave to get these, that's it. So it's entirely up to you at the moment. So you put in your this gate repo a get in.yml file, there is documentation online on how to do that. That's what I was missing. I wanted to add a link to that documentation in the slides. So I'll have to do that. So you put a get in.yml in your this gate repo that tells GreenWave, I want to test, I want to gate on this and these results. When GreenWave receives a new result about your package from ResultsDB, it checks that file and it says, well, are all the rules satisfied? Yes, no, I'm missing some. And it will announce that. But it listens to GreenWave messages and every time they GreenWave announce, well, this bill can go through. It checks, is there an update corresponding to this bill? Yes, the update goes to stable. If the update is not, if the bill did not pass CI, then GreenWave's body say, well, it blocks and the user can go and they can wave the test results. So waving is basically saying, well, I have checked the test results and I know they fail, but they should not fail. This is a false negative. It shouldn't, they should be passing. It was something in the CI system itself. There was a network issue. There was a drive issue. There was, could not talk something. So we try to minimize that except that, you know, Schlinger and Scott and Murphy's low being what they are. It happens. One of the way we want to in the future avoid waving test is being able to retrigger a test. So if you have a network issue, you should be able, well, looking at the logs, this doesn't look like an error in the test. So let's rerun them and see if that passes that time. But sometime you end up in a situation where you actually want to wave the results and that's why WaveDB does. You as a user, you say to WaveDB, that test result here can be ignored. Then GreenWave will receive the notification from WaveDB. It will again pull the data about your build from ResultsDB and say, well, if I ignore that test, everything else is clear. So that update can go through, which when body picks up the message and say, yeah, they can go to stable. So I just included some links here. You probably don't want to click on them because there are APIs, they are meant to be consumed by computers and not really much by human. ResultsDB has some sort of UI on top of it which makes the JSON spread here than the GreenWave and WaveDB which are really meant to be just used by computers. But if you're ever curious of where they are, where they live, how they look like, you can feel free to have a look at them. Yeah, maybe from my HandWavey perspective, just some of the background for this is this is actually a pretty nice differentiation between the tools that implement this and the policy. So even I who don't know what these tools specifically, I can go in and understand what the rules are. If I'm a packager, I can put a rule in my package saying what I want to be gated on. I have control over that. As a looking at Fedora as a whole for a Fesco or the community, we can set policy, central policy that applies to everything without changing these tools again. And as a packager and or a proven packager or whatever, I can use a waving mechanism to say whatever these tools are telling, I know better, right? Because we should trust our judgment and our experience here over these tools. So you can, that's what the waving mechanism is for. So you can set the policy for your packages, just sure wide, or you also have the control to wave and say, this is what the results should be and not what the tools are saying just to kind of summarize these levels. So I just wanted to come back to the waving part a little bit. So we still want this to be the last result but we understand that sometimes it's not. And to make the user experience, waving is actually pretty easy. You have the body CLI and you can just say, well, body updates wave and then you put the identifier of the body update and a comment on why you were being this. My favorite test when I have a test that goes through the entire pipeline and make sure that everything works and the comment is, this is fine. You know, you probably want to have something a little bit more descriptive because you actually want to, we want to be able to go back to these waivers and say, well, why was it waived? It was waived because of that tool did not behave as expected because there was a networking issue because there was an atrive issue because I know better than the tool. Me, as an example, we use this internally as well. And at least in the beginning the most common used comment for the waiver was this is an example waiver. That's, you can use that but looking back, trying to understand why exactly you did waive that test result a month ago that's going to take some debugging. So it's like kind of like to-do, add comment or to-do, add description. You just don't do that, please. And the one thing I wanted to point out is that when you use body updates wave you can waive fail tests but you can also waive missing tests. That's especially important at this time where things are still in progress. We can't re-trigger the tests yet. So if for some reasons the CI pipeline or something else broke and no results are showing up after a certain time you are able to say, well, you know what? Just let that update go through and I know that something is wrong here. It has been reported. Let's just go through so that I can keep on with my work. So we got some early feedbacks. The first one is that it works. So yay! Because the second one was that if you do a single update in Rohai it currently get between five to seven emails. And for people who do a number of significant work in Rohai that is a little bit too much information to buy emails to receive. There is something else that we found out. That's actually not a feedback but something we found out is that if you use build route override you end up, I need to go back here, you end up in a different tag than any of these ones. Which means basically nothing, so you will end up in a different tag here which means nothing is moving your build into the next stage automatically which ends up making basically the update being, the build being stuck. So currently this is work in progress. Currently do not use build route override. Don't. You shouldn't need them because everything goes through and you don't need them. If you need them that basically means you probably are looking for a multi build update situation in which case you actually don't want to opt in into getting yet because this is not currently released. So we don't support that. So we only support single build. So if you're looking at a build route override there is probably something wrong here. Something we clearly saw was that we released single package getting on Wednesday and it was right in the middle of the mass release for F31. The master builds. And when the builds landed into the signing tag, well, you know, when you have 20,000 builds the robust signatory takes them one by one and where you sign? Yes, no. Where you sign? Yes, no. Well, it takes a little bit of time. And when the queue is filled with 20,000 builds it means that your right builds that are coming next to it they are just one in the queue. And it can take a little bit of time until your one in the queue comes up to be the next one in the queue or the one that is currently being processed. So we will need to make some adjustments about signing master builds. How it's going to look like. We don't know yet. Can we have dedicated signing? We don't know. Can we have a worker based model for signing? It's up in here. That's something we have realized we've, you know, pulling this out. I'm happy that we're finding it out now rather than later on because that will also impact the multi-build getting. But that is something we need to fix. On the roadmap, we are still working on, you know polishing the single build experience as well as working on the multi-build updates. So I'm happy to tell you that currently in body, in git we have fixed it so that for a height update you will only get between two and three emails depending on if you test pass or not instead of five to seven. So less emails for you. We may still go to reduce that. We have kept some email notifications because we've seen they are interesting. If, you know, the community thinks that they really are not interesting, then we can always look at reducing that number again. We have also fixed the build with a right issue. So if you end up using them, you should not need them again.  You have one comment also on the emails. What we're looking at is also to see if we have a CI dashboard, you know, where is the trade-off to see what is useful to people, whether you want to opt into emails or whether you want to rather check on a dashboard. I think that's the balance we'll have to find and kind of iterate on. So when you see things happening there, watch the lists and give your feedback or what's useful to you. One other thing that we know we are currently missing is when you create an update for a certain build, you have no way of monitoring what's going on. But the CI pipeline actually announced, hey, I'm currently starting to work on this build. So we actually want to be able to comment on the body of that saying, well, hey, if you're interesting, this is where your tests are being running. You can monitor them over there. So that's something which we have in the pipeline is, I haven't put it in the roadmap here, but it is something which we have in our head and that we want to get sooner rather than later. Okay, and the next big thing that we are working is the multi-build update. So this is, again, how it looks, and again, it's unreadable, and again, it's involved in systems, and again, I have made a simplified version of it. So the user has to do a little bit more work here. The first step that you do is you create a site tag. So a site tag is going to be the place where you do your work in an isolated environment from everything else in Rohide. So that's, and it will contain everything you want to have tested. So you create your site tag. It's going to be a, FedPackage just landed the work on this Lubomir, which I'm not seeing in the audience as put it off. So you will be able to do a FedPackage site tag request or create, I forgot exactly what was the syntax, that will automatically create a site tag for you in Koji. Then you will do your usual git commit, git commit, git push, FedPackage build, but there you need to use the dash dash target and use the site tag that you've created as a target for your builds. Once you're all done, so you have 10, five, 200 packages that you have rebuilt, you will go to Koji, sorry, to body, and you will say, please create an update from this site tag, and you will pass in the name of your site tag. Body will interact, will interrogate Koji and say, well, what are the builds? There you see. What are the builds that are present in this site tag, and it will come up with a list of the builds. It will move them to a dedicated signing pending tag where robust signature will be picked them up, sign them, move them to a testing tag, at which point body will pick up the list of builds again and will send a notification to the CI pipeline. This site tag with these builds are ready to be tested. The CI pipeline will do its magic and will report to results.tb, GreenWave, we come back to the usual suspects, and GreenWave will tell you, you know these list of builds that you tell me about, this site tag, they are all good to go. I have a question on the back. So the question is, I'm going to repeat the question while Alexandra picks up the microphone to answer it. The question is, will it be possible to actually define tests for a set of packages rather than individual packages? And now we want to do that. And I'm going to say two things just before giving you the microphone. First, you're putting my next slide. That was the very last question here. And second is giving the mic to Alexandra to answer it. So for now, the first iteration would be just that site tag will pass if every of the packages in the site tag passes its individual gating policy. So that's going to be the rule because we configure policy per component. But obviously it's not where we should stop and there are talks about how we implement the more generic gating policy, but we need to really work on the place for it and the way how to define it. So basically it's not technically too complicated to define additional rules, but we really need to figure out the user experience around it, where to put it, how to update it, how people are going to manage that. So if you're interested in this topic, follow Fedora CI, Fedora develop conversations and provide feedback and participate. I think one aspect to also think about there is why would you want specific tests for a set of updates? Like why should they not be part of the individual packages as well? Yeah, so the answer from Steven was that there are ecosystems out there that have a whole set of tests. And under that we had the talk yesterday about the Fedora server saying that came up as well. So I think that the trade off there to think about is, is it another test I want to add to specific packages or sets of packages? Or is it something I want to monitor specifically as a separate test, for example, say I have RPM inspect run on the entire distro, I have CI pipeline running for all tests that have packages that have tests in this kit, but I might also have a separate test that's called Fedora server that provides feedback if any number of packages are, or a specific set of packages are involved. So I think there are multiple models that we can think about and for now I would say what you can do is definitely add tests to the packages involved and start from there. So there has been sort of a reflection about getting composes and I actually believe, and you can correct me if I'm wrong, that Green Wave already tests and publishes decisions on composes if they can because through or not. So more on confirms that actually our composes are already getting with the same tooling. Yeah, so getting composes definitely a good augmentation. I think what it shouldn't replace is testing before something actually lands because this is the point where we can affect a change before it breaks someone else and I think that's a place we all want to be where we're not under time pressure. There was another question here. What is the expectation? So the question is what is the expectation for how much time this test runs? So I have a package that is, so the Federal Gazer easy fix if you want to look it up, it has a very simple test.yaml that all of these stores is calling the fail module from Ansible. So it just directly start and crash so that I can test the actual getting mechanism. And that overhead is about eight minutes. Look at that. I know. So. The. The. The. Six hours. Partly paralyze the course, maybe 14 hours. So. 12 hours. But then they get real assurance that this stuff works because it runs thousand something real deployment type cases. Where is the balance? So the question was about how long should actually tests run? What's acceptable? I think that varies per package because essentially what you're doing as a maintainers, you're saying I'm willing to wait this long for my change to go in like how sure do you want to be? I think the answer is probably going to be different whether you tests bash or whether you test GCC or something that's more complex that involves a lot of other components. So I think. That's why I'm asking for what is acceptable. We need to have a policy as a project. I think we're the policy. So the question was about the policy where this comes in is in and I'll touch on this in the foreduce the objective talk later when we talk about reverse dependency testing and when you actually start testing or running tests from other packages. The short answer is as long as it needs, it'll take as long as it needs to take a good guideline this is essentially if we look at this as part of the developer workflow if it takes more than an hour or two that starts to feel really long. More than a day that's really questionable how useful that feedback is but that shouldn't be like for example if the infrastructure goes down you shouldn't say oh this takes too long let's just wait everything. Like that's also the flip side. So that's why the guideline is still as long as it needs to take. If you think a test is essential then that's how long it takes. If you say it's not essential then it doesn't need to be included. So we have updates and reverse dependencies there's a list of packages and craters like Sound Blend, NSS and whatnot there. It's about one hour for everything that goes in the upgrade of the operating system or client-level server and rebooting machines in the process. If we are adding the same to raw product that's probably okay. But if we are adding or wanting to add something more that hopefully should be post composed in that sort of reports back that more rare use cases fail. Well differentiation between rare and not so rare is a valid one I think. But the question is like you shouldn't ask like the question you have to ask yourself here is not how long should I have to wait? But what happens if I don't catch something and everyone else can't use raw hide because of what I did? I think that's the honest, that's the hidden cost. Blocking everyone else and breaking everyone else's workflow. I would question the claim that there should be a unified policy on the time of testing actually. Because the good thing about skating is that while you haven't landed in a compose yet this is your part of the process and you don't block other people from doing their stuff on the compose. So it actually gives you a freedom to decide for yourself and for your level of involvement how much you're okay with waiting and how beneficial these tests are for you. So I would say our current framework allows us a full flexibility. We can use various CI systems in various cases. You can configure them per component, per group of component. So there is a freedom on that. We need to have certain guidelines on generic tests but for component-specific tests we actually can allow very different policies for people. I would say this is since the... So when it comes to package time, the time it takes to run the tests, I would say since the tests are in your hands it depends on how long you're willing to wait to wait for it. If you start to affect other people because for example you make it so that your test runs on your dependencies without consulting with the people that are maintaining these dependencies then it's probably a problem but that's something that you need to communicate with these people. And you say well free IPA tests run for six hours and that's only impact and all the people that are working on free IPA are okay with that then I think it's fine. If the open SSL for Excel well I don't want to wait for six hours and I would rather that we do the one hour side than maybe you can do the one hour side for the open SSL in the six hour free IPA. I think there it's a matter of communication. You need to have different places where you do it. So I think two things here. One is what will probably help is especially if you have do more testing and become more involved. Let's look at the Fedora CI SIG that kind of have that exchange with others who do this to kind of see what works and what doesn't work. And also based on my experience at least the lowest amount of time you should wait is slightly more than the developers are comfortable with. If you ask someone how long are you willing to wait for test results? You should definitely wait longer. That's kind of the lower bar. So what happens in the case when we've been talking about what happens when free IPA has a test and breaks raw hide and then... So you're thinking that something that... So the question is what happens if free IPA lands in raw hide? Something would long that the break raw hide. What happens if something with a long test suite lands in the right and breaks raw hide in the way that was not cut by the long test suite so you have to wait for X amount of time before a new build arise and lands. And I would point out that... Yep, there it is. You can arise wave missing results. For example, if something... If you need to go something quickly through... You can wave results. You can wave missing results. Of course, with the caveat that saying let me just add this quick fix that only fix one problem, definitely not introduce more. I would recommend that you run at least a basic set of tests and use judgment before waving. It's a valid case of waving. I'm just saying you should... Especially when you use reasons as let me just do this really quick or there's a deadline or let me just add this one fix. Those are classic cases of... Oops. The complementary version of this is if right is broken for six hours, it's still better than the state it is today where it can end up being broken for weeks. One small question on the whole process. So the question is, is it possible to cancel a build process when it's being... So it's not built. It's already finished, so you cannot cancel it. So it has already been built. I would say if you bump the release and just resubmit it, the old one will be... The old one will still... The test rate will still continue, but the new one will take precedence over the old one. Right, buddy. That's an optimization for the CI pipeline which we may want to look at at one point. Although... Yeah, that's something we can discuss. Question. I never brought a gating test. So I created gating YAML, test YAML. How can I trigger it without actually building the build in body? The same way like when I'm writing spec, I will locally build in MOOC. Can I somehow trigger the test so I know that I wrote it correctly? So the question is... Yeah, sorry, you had the microphone, so I probably don't need to repeat the question. I'll start. So there are several options. We still are lagging behind with documenting them properly, but we will work with that. So you can run... You can create pull request in Pagore which will be used by CI systems and they will provide feedback on the pull request. So you will get your test results as a flag on the pull request. You can also run scratch builds and I think some of the CI systems are triggered on scratch builds, but probably not all of them right now. And you can run the test locally, but you need to follow a doc for that which we're going to publish. Basically, that's an answer. There is some documentation on how to run the test locally. I think there is probably improvement to do to the documentation and I think we should also look at providing some sort of wrapper to help on doing this. But what we don't have right now as far as I know is proper testing and pre-validation of the actual definition, I think, that's what you're asking. Yeah, there is no a lint for the YAML files. So... So your followup already given us quite a bit of feedback here. There are some things that we are aware of, especially with regards to the multi-build. There is a question of site tag proliferation. How is Koji going to handle this? If a lot of people are using site tags in RAI to do their development work there, that's something which we will need to test and see how that behaves. I mean, if everybody uses site tag to do their work, I think this is a good problem to have, but that would rather that we anticipate it a little bit before then we actually hit the wall because Koji is not able to manage that. There is a big question about on merging the site tags. How do we result conflicting builds? How do we provide feedback to the user with instruction on how to fix that? There is a question about race condition between Koji and body, because body will store the list of builds that are present in the site tag and that's what it will send to the CI system and that's what it will receive information about. And what happens if just before making the decision, the user realized that, hey, something is broken, I can or I've missed that one package and I can push it as well. Then suddenly you have one more build in your site tag that body was not aware of. So we need to be careful, very careful here, but being sure that what body pushes to stable is exactly what is present in the site tag and there is no, we try to minimize the possibility of race condition within the two here. Then there is the entire question that Stefan has already raised was, how do we test site tags? Do we consider them the sum of all individual package tests present in the site tag? Can we create site tag specific test? How do we handle that? So I'm not going to go back to it because Alexandra already covered it. We also have some point of attention for testing itself and I'm going to let Dominic on that one. Yeah, maybe just to add onto the previous slide. So the goal here is not to have something that's perfect. We want to have something that improves stability and makes things, makes our world a better place and there'll be a lot of points where we can improve this and I think if we iterate together, it'll be better but the goal is not to wait until everything is finished then have the system that solves all the problems that would probably take a long time to wait for. You mean wait until everything is finished and then deliver the system that solves all of the problems? Yeah, exactly. So yeah, we're testing, there are a lot of things here that are a lot of aspects that you have to think about. Like when we do have tests, we have to think about stability, right? It's like how stable are our tests? How stable is the testing infrastructure? How reliable are things? All of these play a role. Some of these we can influence. Some of these you can influence. Some of these are easier to influence than others. So that's just, we like feedback's good on these things but it's, we have to tackle specific problems to solve them, I think. The impact of these tests, like something to keep in mind. It's like when you do this, of course you're slowing down individual development, package development. The benefit is not breaking others and giving yourself the time to fix things, address issues. But also as we ramp up and as you contribute to tests, think about what does this do for others? If this is run as a reverse dependency test on other packages, can they, can people understand the results? Can they, can they, like, do they know what this does? Can they interpret the results? How can they, how can others contribute if they see an easy fix? We are aware that we have more than one architecture that Fedora runs on. So some tests do. Do our Fedora tests run on multi-arch right now? I don't think any do right now. That is definitely something we want to do. It is a question of prioritization. So if you have a case, well that is really important. He can reach out to us and the answer will most likely contain somewhere the word contribute and please. So that is, we are aware that it's a thing, but coverage and getting things in at least for one architecture is a priority we made. So if you have feedback on that decision, please reach out to us. Maintainability. It is with tests, I'll probably get to that a bit more in the CI objective talk at 330, but think about how maintainable tests are. Like if you add something, especially if you add to contribute a test to another package or take something to the upstream, think about what would it take to maintain that? Does it make sense? Can I augment something that exists? Can, or even can I take my tests upstream? Even further upstream than Fedora. The further upstream, the better usually. And otherwise the same principles as with other code apply, like how to write maintainable code. I'm already come to the connectivity upstream and of course we are aware that we don't just have packages but also modules and containers and these are also tested but to different extents because the infrastructure that is tied to those is just different in some cases. So this is mostly a question of priorities. Not that we think one is more important than the other. It's just that we have more RPMs right now and they're also used for modules and containers. So it seems a logical choice to start with the packages. And with this, well, you've already been very helpful but we would like to invite you to help us more. We need more, more testers, give us more, take a look at the current workflow, tell us what's right about it, tell us what's wrong about it. Don't forget the first part, please. If you want to test the multi-builds, reach out to us. We will be looking at deploying this in staging in the coming months. So if you're interested to do this, if you have a set of tests or packages, you want to be looking at testing. Yeah, again, reach out to us. We will work with you. We will see how to get this working in staging with you. This is going to be a great friend ride. So don't hesitate. Give us your feedback, that's the main thing. Try it, give us feedback, tell us what works, what doesn't and tell us how we can improve it. And also your workflows, right? Think about what Alexandra said, that the freedom you have to choose what tests you run, how much to run, where to run them, use that to decide what would fit your workflow and then if you're unsure how that fits with what we've implemented, talk to us and we'll try to help you or we will help you figure out whether that fits into what we've built. If it doesn't fit, do we need to change what we've built? Maybe, or maybe we have suggestions on what you could do differently and we can talk about how we can make it work. And with this, we would like to thank you and take any questions left. Mohan. So, two questions. First one. So, what is the process for when the CI just fail? Like when I, if I say body update, blah, blah, blah, does it automatically do, up to the stable or do I need to do something else? So, the question is what happens when the tests fail? If I do a body update wave, what happens? So, when you do body update wave, what you're saying is you're actually sending a request to WebRDB via body to say, well, wave the test, the failed tests. So, you can, when you do body update wave, you can specify a single test, you can specify all the tests, which is the default, whether they are missing or just failed. So, you're sending something to WebRDB. WebRDB will send a message that there was a new wave added when wave will pick this up and say, well, this WebR concerns that test result that concerns this package, what were the rules for that package, what were the test results for that package? If I ignore that test result that I was told, that I was just told to ignore, what's the outcome of it? If it's still false, then it's, nothing's gonna happen. If it becomes true, the package can be, can go through. It will announce it, but it will pick up that and say, well, this can go through, move the update to stable, push the build to the right. And second question, a kind of related to the like that testing, is there any plan to add some distro level testing, especially when like, a text bump takes, fails the composer, that's the most common thing? So, the question is, the question is how to, is there any plans to do distro level testing, especially when a bump spake failed? So, then bumps, broken dependencies and these kind of things. Yes, more in the talk on the Fridorsi Objective. We also have distro wide tests being developed right now, like the RPM inspect with David and Tim, that they presented and now present some more options and if you have more ideas, then that's a perfect place to contribute, because this is like, the system allows you to add other systems that provide feedback and test results. So, there's no single place you have to put them, the messaging system remains the same. And the intent of this is also that we can start deploying the systems, seeing the results and then deciding as a community, do we want to gate on those? I think that's the key difference. There was another question. No, not anymore. We are right on time, so if there are more questions, if there is one last question, then we are happy to take it, as always we are happy to let you go to the next. Okay, so if I have a lint, there's something that shouldn't be tested, but shouldn't block the package going through, is there a good way to see the results for the whole distro about these lint-style checks? So, the question is how to get distro wide testing without getting, is that correct? You can add test results to any build. You can basically run tests on anything and report the tests and integrate this into results DB. They will show up in the body tests results page, which is basically how Tasker Tran and OpenQ have been working so far. And as long as you don't document that in green waste policy, whether that policy is in distro level one or package level one, it won't be used to make the decision about the getting process. And individual packages could opt into gating on that by choice, for example? Is there a good way to see the results distro wide? Distry to see the results that would, basically, yes, by results DB also all the information, so that would mean basically we need to create results DB for all the results about a certain test during a certain period of time, probably, and see how the trends goes. So there would be ways to monitor that. So the question from ExxonR is, in the body update page, so we saw the test results that are being considered for getting all of them, we are showing all of them. Which, okay, Mohan got the last one. But we should stop after that. I'll say I cannot view these results, or these tests. Is there any plans of policy about not being able to waive certain tests? Dominic seems to be very good to answer that one. Yeah, so that is not implemented right now. I would say in the future, it would make very much sense to restrict certain waivers. But that would be probably a decision for Fesco and the community to make. So I think technically, that is possible. It is definitely not planned in the near future and would not happen without community involvement. Like this would need wide consensus, it's my opinion. And I think with this, we're gonna close because it's about time. Thank you everyone. Thank you very much.


In 2019, Saint Lucia was the first OECS country to launch the Global Hearts Initiative in collaboration with the Pan-American Health Organization PAHO. The goal of the Global Hearts Initiative is to reduce premature deaths from cardiovascular diseases in target countries by supporting governments around the world to scale up efforts on the prevention and control of non-communicable diseases such as strokes and heart attacks. Saint Lucia's training of health workers under the Global Hearts Initiative was impacted by the COVID-19 pandemic leading to a suspension of training. However, senior medical officer for non-communicable diseases, Dr. Shana Seraphilbert, is happy to report a scale-up of training for staff of wellness centers on the island as of January this year. Every letter in hearts speaks to a particular module which is aimed at improving high blood pressure, control, blood sugar control, and an overall just cardiovascular risk. And today we're here because we started as a country, Saint Lucia signed on to the Hearts Initiative in 2019, in October 2019, and we launched Hearts because we thought it very valuable for our country. Hearts, as the name would suggest, is a program aimed at improving our heart health. If we improve our heart health, guess what? The whole body is healthy because our heart is the major pump and controls lots of things in our body. And the slogan for our hearts in Saint Lucia is happy, happy hearts, healthy nation.  All of this comes into the acronym, HEARTS. According to Dr. Seraphilbert, over the past 10 years, cardiovascular diseases such as strokes and heart attacks have been the main killers accounting for about 30% of deaths in Saint Lucia. Recently, representatives of 12 wellness centers underwent training in capacity building, and according to Dr. Seraphilbert, the ministry intends to train more staff as part of efforts to improve the quality of health, thus a reduction in cardiovascular events and hospitalization. One of the things we're also targeting is that patients should not just everybody come at once, but come to the health center at a time when you get your slot of time with the physician and the nutritionist and so on. So this is one of the things we want to see, especially in this year, we want a multidisciplinary team and we want patients not expecting that when they come to the health center, they just come and grab a prescription, because that is not what we want to improve that. We want more than just a prescription, we want counseling, we want patients to actually have a holistic experience. Of the 32 wellness centers on the island, 20 have so far benefited from the Global Hearts Initiative training. From the Communications Unit of the Ministry of Health, Wellness and Elderly Affairs, Julia Zepita reporting.


The Commission of Judicial Selection announced today it is recommending three candidates to Governor Mark Dayton for consideration to fill the current vacancy in Minnesota's 9th Judicial District. This vacancy will occur upon the retirement of the Honorable John R. Solene and will be chambered at Aiken in Aiken County. The three candidates are David Hermerding who is the Chief Deputy Crowing County Attorney, Brooke Malick who is a partner at Gamelo, Kuali, Pearson, Malick, PLLC where she primarily handles cases in litigation, civil and criminal law, and Sarah Wingey who is an assistant Aiken County Attorney. Now Minnesota's 9th Judicial District is a big one. It consists of Aiken, the counties of Aiken, Beltrami, Cass, Clearwater, Crow Wing, Hobart, Itasca, Kitson, Kuchaching, Lake of the Woods, Benolman, Marshall, Norman, Pennington, Polk, Red Lake, and Rosa, all those counties. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.