In the front line of the fight against tuberculosis is the general practitioner. He sees the patient first, and upon his skill and judgment, depends the patient's future. There are certain diagnostic procedures every general practitioner is capable of carrying out. One of these is the examination of the sputum. To refresh our memories, Dr. Ralph S. Muckenfuss, Director of the Bureau of Laboratories of the New York City Department of Health, will demonstrate a simple technique of sputum examination. Dr. Muckenfuss. The reagents needed to demonstrate tubercle bacilli and sputum are carbofugusin, acid alcohol, and alkaline methylene blue. We need also an alcohol lamp, a platinum loop, and a glass slide. Flame the loop until it is red. The selection of proper particles of sputum is important. Small, purulent, orcazius particles should be fished out and smeared thinly on the slide. After the smear has dried, it is fixed in the flame. The smear must not be burned. The slide is flooded with carbofugusin and heated to steaming, but not to boiling. This heating must be continued for three or four minutes. The stain is poured off and the slide rinsed with water. Stained smear is then decolorized with acid alcohol until no more color appears in the washings. Finally, the specimen is counter-stained with methylene blue. A drop of oil. The short red rods are tubercle bacilli. Finding tubercle bacilli in sputum is direct evidence of pulmonary tuberculosis. Failure, however, does not mean absence of disease. If repeated efforts fail, the sputum should be submitted to a laboratory where special methods may be employed.  A positive reaction means infection by tubercle bacilli. I have the pleasure of presenting Dr. Esmond R. Long, Director of the Henry Fipps Institute of Philadelphia, who will demonstrate the test for us. We use purified protein derivative instead of old tuberculin because it's more uniform and is standard. It's put up in two strengths. Accompanied by vials of sterile buffered saline solution. When a tablet of the first strength is dissolved in one cubic centimeter of the saline solution, one tenth of a cubic centimeter contains the recommended dose. If no reaction follows the use of the first strength, the test is repeated with the second strength. Use a one cubic centimeter tuberculin syringe that has been sterilized. Inspect the needle for sharpness. Flame it to make sure that it is sterile. Draw the sterile buffered saline solution into the syringe and transfer it with aseptic precautions to the bottle containing the tablet to dissolve the tablet. Withdraw one tenth of a cubic centimeter into the syringe. Cleanse the forearm with 95 percent alcohol. Insert the needle interdermally. This is important for if the solution is injected under the skin, no skin reaction may result. Here's a diagram to make this point clear. This is the wrong way. If done like this, the right way, a small white wheel will rise over the needle point. Hold the needle almost parallel to the skin as it is inserted. Inject one tenth of a cubic centimeter of the tuberculin solution. Did it hurt? No doctor, no more than a mosquito bite. The test is read in 48 hours. Hold the arm in a good light and flex the little at the elbow. Taught muscles may obliterate the reaction. Look across the arm rather than down on it. A positive reaction is one that shows edema and redness of varying degree. If there's no edema, the reaction should be considered negative. When in doubt, pass the finger over the tested area. Sometimes the induration can be felt when it cannot be seen. No examination of a case in which tuberculosis is even remotely suspected is complete without a good chest x-ray. Only a specialist should be trusted to interpret the x-ray film. But it is exceedingly valuable to do this in consultation with the general practitioner. Dr. Edgar Mayer, assistant professor of medicine at Cornell University Medical School, will explain to us certain fundamental facts in the x-ray diagnosis. In reading the x-ray film, we must know its appearance and health and the wide variations that can come within normal limits in a properly exposed film. The appearance of the healthy chest varies within wide limits. The hyalum is an irregular conglomerate shadow lying at the root of the lung. It is made up chiefly of shadows of blood vessels, lymphatics, bronchi, and tracheal bronchial lymph nodes. In the adult, the calcification of these lymph nodes at the root of the lung are consistent with health. In diagnosis, the con end of shadow means less than the location. Lesion situated at one apex only are in about 90% of cases tuberculous and almost invariably so when they involve both apices. On the other hand, a lesion at one base only is rarely tuberculous and at both bases almost never tuberculous. The exact interpretation of the kind of shadow, namely the mnemonic lesion as against the proliferative or the fibrotic, is possible on the x-ray film only in the most typical shadows. This a mnemonic exurative form is a confluent shadow. This a proliferative form is discreet and this a fibrous form is rather linear. Such terms as active or quiescent should not be applied to lesions on the single film until corroborated by history, clinical and laboratory findings. Thus, this patient with a fibrotic scar had a recent hemorrhage and this one with a confluent shadow had no symptoms of active disease for five years. To illustrate first infection forms of tuberculosis, we have here a primary bronchonemonia in a child of six months that was nursed by a maid who flatly denied any subjective symptoms but who showed extensive pulmonary disease by the x-ray. This film of another primary complex shows the pulmonary component, the gome focus, and the lymph glandular hyalurshadow in a child of three years. The tuberculin test taken in a routine survey was found positive and checked up by the family physician revealed as a household contact a grandparent with this evidence of advanced tuberculosis. He had been treated for so-called asthma and chronic bronchitis for many years and subsequently sputum examination in this contact showed tubercle bacilli. In reinfection tuberculosis, this latent apical shadow gave no clinical symptoms nor any definite physical signs. This early infraclavicular infiltrate occurred in a medical student with only slight gripple symptoms and gave no obvious abnormal physical findings. This patient markedly emaciated had a profuse hemorrhage from the lungs who expected to find great lung damage but the x-ray revealed this minimal lesion in the lung. This athlete boasted of his great physical strength and belittled his cough and sputum ascribing it to cigarettes. The x-ray revealed this multiple cavity formation. Without a good x-ray picture, no examination for tuberculosis is complete and suspected tuberculosis cannot be detected nor ruled out without it. These three procedures are important aids in making the diagnosis of tuberculosis. Yet it is not enough merely to establish the diagnosis. It must be done promptly, for good prognosis is largely dependent upon early diagnosis. Let me illustrate with the results of a study of a large group of patients known to have had tuberculosis. Here are 10 persons in whom a diagnosis of tuberculosis was made. An average of six months elapsed from the time of the appearance of the first symptom to the date of diagnosis. When classified according to the stage of the disease they fell into three groups. 40% were in the minimal stage, 50% in the moderately advanced and 10% in the far advanced stage. Here's another group of 10. There was a delay of six to 12 months in diagnosis. This group was classified so. A third group suffered a delay of from one to four years from the time of the appearance of the first symptom to the time of diagnosis. And only 15% were in the minimal stage. The group in whom a definite diagnosis was made four or more years after the appearance of the first symptom fared badly. 45% were in the far advanced stage.


Hey everybody, it's Brian and wow it has been September since I made a video is actually longer than that since I've actually made a tutorial Life kind of got flipped upside down for me. I know the last video is kind of a what's Brian been up to and I've gotten a Flood I mean a metric ton of email going how come you haven't made a tutorial and did you stop programming? No, I didn't Little bit of what's going on in Brian's life if you don't care. Just go ahead and you know skip this video Really been focusing on the void realms Facebook group I want to make sure that you know It's kind of like a safe place for people to come and talk about their programming feelings without everybody looking at them Like they're crazy. So we got Where's the counter thingy? 966 wow we got 966 members in there It just keeps mushroom clouding beyond belief and actually I've gone in there and asked questions And it's kind of amazing because it's much faster than Google Also battlefield one video game. I've been waiting for this to come out for forever and ever and ever Not a bad game not as good as I hope but it still eats up some time in one of my all-time favorite games Elder Scrolls Skyrim the digitally remastered version Yes, it's the same old game But you know it's better because it looks better and it's actually really fun still I forget where it was but Some Chinese gaming review site It's like they never give out five stars and the only one they've ever given out five stars to was Skyrim Amazing game if you ever get a chance to play it Also, I got these flipped around here. Let's just do that. I went through a bit of a relationship breakup I'm not going to Unlike the rest of the internet. I'm not going to use YouTube as like my own therapy session But just know that it Ended it was a very long-term relationship and that's always rough to end with and it was Back when I made that last video. We were going through some problems Wherever that last video went here Um, so that's kind of why I've just been focused on you know family and life and all that but My sense of humor has not died.   Sorry. That's just that's bad even for my channel All right, maybe one more Hey girl May integrate on your smooth man. Okay. I'm not even going to read that. It's just wrong Um, but I've also been focusing on mobile programming something that's very new for me So I want to start making some Mobile application tutorials here very soon I'm still on the fence if I want to do it in android or samarine. Um, if you're looking for really solid android tutorials I know that buggy Buggy has some really good ones and then this programming knowledge This guy's tutorials are amazing. He has a bit of an accent But his tutorials are just amazing. I've been learning it and I think I've learned Where am I? I think I'm like 30 something I just got done with the database ones and I've learned more in his tutorials than I've learned through four android books So definitely, you know kudos to him and definitely check his videos out. They're definitely worth it Um, but like I said, I'm torn between android or samarine Um, like my last video said samarine is kind of like a cross-platform way of doing it You can write once and deploy anywhere kind of similar to cute Um, I haven't become disenchanted with qml. I'm a little What's the term I'm looking for Not thrilled with the way cute handles things with qml on mobile devices It just seems like a real pain to get things to work where where the android studio or even samarine it seems a lot easier So I think that in some ways cute has some room to grow But I was actually working on a tutorial. I need to pick it back up I was doing collision detection and cute and you can see nothing's moving but these are supposed to move and You see this timer ticking away Basically they're going to move bump into each other and then Tell where they collide because they really isn't built-in collision detection in qml, which kind of bothers me Um, but silly me. I don't know why there would be built-in collision detection and of course my uh, my baby my Magnus opus, if you will um a high performance Http tcp server, uh, but slowly building this backup. I've got it so that it'll detect ip addresses So, you know because some servers have multiple ip addresses things like that So you can do any any or this specific ip ports number of max connections and then whether it's single threaded or pooled Found a lot of bugs in my original build. Um, let me see if I can bring up siege here There's still a couple little bugs that I need to work out But like um, we'll just say you can handle all max thousand connections. So we'll do Thousand this is probably gonna way go over my My limit on this box because I haven't really played around with the file descriptors yet or not descriptors the file hard limits But I mean as you can see it's just cranking away. Um, just a mock http server um I don't know. It's just something that's always fascinated me as tcp programming So I might do another full-blown tutorial set on tcp servers even though I feel like I've really Kind of beat that to death, but uh You know similar to my last video. It's the you know single Uh, where each connections run on one thread or pooled where they get a pool of threads or threaded wherever connection has its own thread Uh, but I did find some bugs And there's still a little bit a couple more bugs in here like if I get a Rejected connection, which it doesn't look like it's going to do That it'll actually keep accumulating those connections instead of releasing them. So Let me stop that, but uh, that's really what I've been working on here Uh, I do plan on getting back into making tutorials. I really want to get back into cute because it's kind of my love But I want to get into mobile programming I've heard a lot about go lang You know, there's a language called go from google. I really want to start looking at that I've heard some interesting things on that as well But I'm interested in you know, kind of what you guys are are working on and looking at so um, I may actually start a thread Out in the voidrums facebook group. So if you haven't joined it already, please do so. I may say hey You know, what are you guys most interested in? Um For some reason I keep wanting to go back to java and c-sharp. I never really did a full Full tutorial set on that. So I may go back to my my roots, which was c-sharp and do like a full end-to-end tutorial set, which means unfortunately I would have to be in windows Uh, because visual studio runs in windows, which I heard they're pushing it onto the mac But anyways, um, it's not that I don't like windows It's just I'm much more comfortable in linux, which is really weird because I grew up as a windows guy I didn't touch linux until about someone to say 10 years ago Um, it just I don't know. It's weird But anyways, um, that's enough of my ranting but that should answer your emails. Why have I not been making tutorials? It's because I've been busy moving and buying furniture and you know meeting people making new friendships Busting out the nerd pickup lines, which just crack me up. I actually want to use a couple of these Yes, you must be the speed of light because time stops when I look at you. That's I'm using that. That's my now Okay, I own that forever and ever Anyways, that's all for this one. Um, hope to see you guys on the void realms Facebook group. No, it does not cost money. I don't want any of my stuff to cost money at all I learned programming free of charge. Thanks to the internet. So I want to make sure that Everything I do is free of charge. So that's it Maybe if I figure how to turn this thing off


It's the most vague statement ever DBR in It's the most vague statement And I didn't know what I did you know what I did DBR in I just went and did it. I just went and did it. I didn't care Need bones around here. I Need some quality bones hurry up There's no bones There's one there, but that's too far away Run you have 20 seconds to get way over there. You're not gonna make it cuz the fish is blocking you Sad faces were had that day twist that television Well, it's the zoom in skill You have to yes a mod you first have to get the mod I have two mods I doubled up yesterday.  That's a press perfect Perfect way to say it. All right, we could jump back on the optionals now. I Think I'm going back to camp aren't they?  I don't have any poison ammo.  All right It messes with a lot of melee weapons. I Don't want to know Well, no, if you if you go into the options, it should be a ranged only thing Right. I thought it's a you could do a range only mappings poison Y'all of this I made a lot of money from that. All right, so now let's go back into over here and fill it up in my Yeah, if you go into options, this is how and There's the controls They have separate settings for melee and range so it shouldn't affect it like when I go to melee is like all default stuff here It shouldn't affect it But there could be stuff that I don't know because I don't play melee yet That happens Register loadout. Oh, no, I haven't done it yet Poison now I can register this. All right now the poison is registered.  Let me look at that and it was just there chat poison is Is it the poison mushroom? I hope it's poison to where's poison one? Why aren't these a daughter?  Ah, there it is. That's coding. I Have them craftable. I don't want to coating Toast to Toast to it's a mushroom Paralysis is parish rooms put that all the craft Sleep is sleep herbs. I'll put that on all the craft Cool. All right Guys if you need the session ID is in chat. It's also in the title Done with while leveling now I can play the game That's so funny Done with while leveling now I can play the game. There's a bug flying around my leftover food That's kind of nasty you cover this up freaking summer man Bugs just crawl out everywhere So now we do some optionals Yeah, I should have some new optionals We got engine F we got Toby little of them. We got pookie-pookie. Let's do engine out again Since I got his number chat. I got his number. Oh, so you can craft and by ammo. Yeah, yeah You can craft you can actually craft during combat too Which is actually Kind of cool Yeah, that was some good eats That was a really really good eats You know Take you take you take you and take you usually start Okay, so I got on my shortcut bar That's the start with sticky. I'm getting better at that shortcut bar Because they have a PC version of the shortcut bar The function keys You can't remap it.  I have some shortcuts here for crafting So I only have sticky ammo slicing ammo and cluster bomb I can go in here and add some more but I don't want to I'll do it when I'm in town That way I can register my load out.  I need bones. I Just pass by some ball. All right. I'm gonna get some bones man.  Give me. I saw it. It was mine um, I need some some some what are they called Quality bones for my last two pieces of my bone alpha set. There's one so I need I need like I Think six of them Damn, that's him Alright, let's go kill him There's actually some bones there. Let me go to those bones first Then we'll go to him. Let me get these bones first and then I'll head over you guys Wait, where are they?  It's so dark. I kind of see Leave me alone. All right, fine. I'll go to the other guy. I can't even see the bones. There's something there They are they making me do a chat. They made me break out the shoddy chat Now where the freaking bones go? I need this bone good So I can go help them fight the dude Okay Now we can go Hit that guy. I only got one quality bone though. So that makes me a sad panda. I'm here guys I'm coming with a cluster. All right. You're ready There we go Where's where's the indicator? There it is. I can't even see it anymore. There this I hold that thought he ran away chat That was good shot Three those ought to be enough. He's drooling. Oh, he's drooling. I tried to okay. They got the job done Eight years man, geez you've been following me like since forever, man Since forever Bones wait a minute Block that didn't go as planned He hit me all the heels came through chat. Hurry up. Pick up. Ah Crap. I was looking at mama. I know he was on me. I'm gonna use bones. I don't know about y'all Be with you says YouTube's is white night chronicles.   There's my slice block Yeah, yeah, we did What why are they have the only like every option is this band? What if I don't want to return to camp? Whoa? Apollo hit the button Everybody's showing some love in the chat. Thank you so much eight months, dude Bunny crazy That one of guys nuts Almost folks. He's crazy. Oh a unique mushroom colony. Give me. Give me hurry up. Oh I need these four things and points and possibly Missions Okay, we're good Thank you so much. Mr. Apollo man.  Oh That was a good chunk a couple more of those chat and I'll be done with first footprint We'll be moving on to unchartered waters waters waters Again, if you want to join this is the session ID for PC. It is also in the title TIA group stage all his TIA is TI season. Oh man TI season is here already Feels like we just This feels like it's been like it. This has it Feels like it's if you have how you how you doing that? Feels like it's been a year Less than a year. Oh, it's in Canada, Canada Canada It's in Canada. Yeah, I got a dress Actually, I don't have a dress. Oh, I could do this one too neat full set height Yeah We got our alpha set we got our alpha set bone alpha in the house So I need I need quality bones. I don't know what the third bone is though For that bone up there in More optional chat more optional Yes, okay, I do believe that's by there. Let's go do it. Let's see We haven't done this one which is a Tobi Kaliachi Maru a Tobi Kobarachi Kamikachi Yay, Jubilis Whoa, that was awkward. Why are you dropping me off halfway, man? All right, let's get some bones while we wait. You're first Because we need them quality bones. Oh They had our renovations for it. Okay neat neat By the way, whoever is testing the chat you're an all-star hero You've been helping me like every night and it's been very very much appreciated. I don't know who test it is But thank you test Bones now leave me alone and the mules. You don't see me As you minus. Okay. Thanks, man. That is That is some that is some monstrous help pun intended Pun absolutely intended next bones will be down here, right? Let's go over there. Keep on moving Oh, no I'm not gonna be able to get this aren't that well. I guess I got a fight. Oh, they left never mind Um, what is this? This is mining I need quality bones in the mystery bone. Whatever that may be for my next upgrade to my My bow gun I don't think this one will have aggro. It will not that last one had two qualities which is I think the most I've seen. Oh nice falling boulders Job uh Yeah, I'm going up there now also the uh The waypoint thing on the map is helping me so much I'm getting lost a lot less Because to get to certain points. I just I just pin it and let the flies let take me there I have to thank Whereas on ps4. I was trying to navigate the map myself and I was failing horribly I was like, well, how do I get over there, man? All right, here we go Let's start with uh some sticky You jerk It hit the freaking rocks chat Whang I am officially upset at the universe He who controls the salt controls the universe bones Quality there we go Can't believe it hit the freaking thing with the thing and the stuff We will kill tobayashi miru chat Wait 25 million is what the ti is at now I thought Dodo was a dead game Misspelling of game. All right three poisons. Is he he's drooling? He's drooling chat.  He should be drooling again. He is I'm gonna give him a special treat. This is a special thrill sausage prescribed by buana eat up We did it All the work chat all the work It's a flower bed. That's some points I bought this and I blame you for it. Sorry I'm sorry Get it before time runs out. This is this is good. Oh, that's rathalos. That's not That's right. That's not the other dude That's the skill. I like that I'm sorry certain valin. This is a good game man. It's good. It's the um The multiplayer, I hope they can fix it though, but that's like the only thing against the game right now Oh, and the the cutscene nonsense, but I have developed a method I have the technology and the skill To get around that and I think this method what I learned from the ps4 version helped me this time Because I'm having a much better time because I know what to not do So Like don't expect to go in co-op and do all the story with other people because they ain't gonna work Plus it helps that I'm a I'm enjoying this bow gun Uh, oh, I gotta Get my harvest You do have a garden in this game too All the harvest Oh, that's new There's something to explain what these things are Duration x3 I guess that speeds it up by that much I'll go to The large gun I remember the exact name is fun too. Yeah, that's what I have the heavy bow gun


Well, actually we have a segment on the show Today is oh goodness it is March 9th, so it is my two-month anniversary here two-month anniversary March 9 Thursday vlog update Yeah, things are happening stuff's happening and this our special guest is brought to you by the wind Because the wind it was crazy here in the Detroit area actually some big areas of Michigan, North Ohio and everyone doesn't have power So we're actually we're in this weird little pocket of electricity, which is my office and we are kind of shocked by this and Yeah, it flickered a few times yesterday, but it stuck it out. It's not a juice So that's why I was at work early today because it was either that or said at home in the dark with no internet Oh, no, and I passed three big B coffees on my way here that are dark So I did get to me Hortons. I'd like to me Hortons to me. Oh, it's good. It's my every day Yeah, he's his every day, but they unfortunately they they're all close because like everywhere doesn't have power Yeah, also by weird coincidence my house has power. Yeah, the people across the street from me have power Yeah, his school lucky you know power So now that we've covered the more the mundane part of the news there are yeah, he's part of the mundane news now And so he gave me being the proper father. I have a bag of donuts and for him Actually, he tries to eat an entire bag of donuts at Tim Hortons, but we didn't let him Briss is like what you like and he's like an entire box of Tim bits. Let me pick them all out She's close to guys the box. I'm like no no no. No, he's and he's like what?  Yeah, he was just gonna get a whole box of donuts He told her good job. Don't let me hang it dude. Yeah So he's gonna hang out and we're gonna find things for him to do like I pick up messes and clean things. Yeah, yeah anyways interesting things that have occurred I Wanted once again over thirty two hundred maybe thirty three hundred subscribers by this morning. We're working on more content I was my wife's birthday. So She forbid me to come Saturday and Sunday because we went out for a while and come make content Which is what I usually do is my quiet times man I know wives good news is this weekend more content on the way I got stuff that I'm reviewing all right You don't you know what that is do you know you're gonna learn Got some more unified stuff. So we're gonna build a unit wireless charger But it's wireless and it's a wireless antenna right for some testing and depending on how you use it You could be charged That is So wonderful play on words there So we we talked about a little bit, you know, I do I have another podcast I do called smr.us for sunny morning Linux review with a couple co-hosts And so we cover a lot of security topics Which is why I'm always so deeply interested in security and the happenings of things I was part of what I report on and We were I was also Monday in the IT in the deep eye casters link to it on our blog large system slash blog look down here Where we they were great podcast talk about technical stuff and meet with other tech companies I was one of the guests on there So I love talking about security research and we had some incidents and we were kind of reviewing some of this And so far this year. We've had some major human errors For like the Amazon S3 turns out their their debrief on it was typo I meant to take down Two servers and took down 20. I don't know the exact as they didn't realize the exact I don't command he typed But basically the idea was supposed to take down a small number of servers Then they took down a very large number of servers and that's what I always mess up some mundane detail So it's interesting to look at these large companies like Amazon are literally taken down by a couple clicks You know one wrong click I should say my user We also seen this happen with I think was git lab a typo of a one instead of a two during a back-up restore It caused them to lose 12 hours of customer data in the cloud and Yeah So yet another typo, so it's really it's just fascinating to me that all of these things that occur now attention to detail I Donuts for you. Yeah, these donuts are clearly So now we have a vault seven to talk about goodness. Yes The CIA the largest dump right? Those are very subjective terms sure pretty big either way either way. It's very big There's a lot going on there So I've been I tweeted a couple things. I some of it's my amusement Was like, oh look, they don't care about Windows phone at the CA because most of the hacks pointed at things in the Android and iOS markets what really is scary all 16 of you on the Windows phone can breathe a sigh of relief I believe the CA's clearly doesn't have a threat So with all that it's it's amazing and it's actually a massive Security problem because under normal disclosure when a security researcher finds a hole They have a process by which they will notify provided This is not something being exploited in the wild. They'll tell Microsoft. They'll tell Google whoever has a security problem They'll let them know What it is and there's a disclosure period by which they have time to fix it before the bug is that disclosed And it's often to be patched turns out the CA was learning and buying from the whoever was selling it to them Some of these bugs which of course makes us all less secure because someone found them they're using them exploiting them and now that the Database dump has occurred so many of those bugs are now completely out in the wild for everybody to use Not just the CIA or any department that had a copy of it. So it is a massive Breach of security. They we found out well wonderful things like the we're using our TV to spy on us Yeah, that that now technically there it's not happening in our country because that would be illegal But the capability is there. Yes vulnerability exists. So the Samsung TVs had a false off basically You think the TV is off. It's not really it's recording What you're saying if they have a camera on them, I think don't they some have a camera? The important part was the microphone. Yeah, you don't want to use all the TVs on because right that they would just hear We're watching but maybe that's interesting. We know it was to Vizio Yeah, yeah, yeah, so using that very much like 1984 Yeah, very much like 1984 TV's are spying on us. So they were using that and I think the other thing too was the CIA They they violated I Get well, I guess really it was an agreement it wasn't really an act or anything but that under under the Obama administration they said any time we've any time the Administration finds a hole or any of these vulnerabilities. They will share it out. Yeah, and and I explainedly was not doing that No, this is a great Insecurity and a problem that they were collecting and hoarding them in one place because well this could happen They could end up on WikiLeaks and it did but and it did happen This is also why government encryption backdoors are such a horrible idea because government's ability to keep a secret has not been good Sorry conspiracy theorists who have these crazy conspiracy theories. Gerardia's no one's that could have keeping a secret. We went to the moon Yeah, so but what we're doing business wise So many things as always Yeah, installing lots of new cameras and schools and putting in More and more redundancies So we're seeing more and more clients and of course now power is gonna make them concerned again Because they can't operate without electricity, but redundant internet connections. We have even a smaller Fabrication company not that big but the time is worth it I mean, it's only actually a hundred dollars a month to put a redundant circuit in because they need internet or they lose visibility of their Ability to dispatch so now we're seeing it I did a review on this for pfSense which are still a favorite tool to use do these are installing boxes that allow for Redundant internet connections. This has been a really big request for a lot of our clients We're also building a lot of websites for clients right now revising some that haven't been updated since like 2012 and Yeah, it's it's time for facelift I mean if your websites even a couple years old if that's part of you Excuse me if that's part of your marketing strategy, you really should keep it up to date make sure it's responsive Look looks of websites change just as technology allows and just as people's Preferences change and so a website built two or three years ago is not going to look Like the typical website that you see today. It's not going to be appealing to people You also want to make sure that it is responsive. You want to when you look at it on a mobile device It's it needs to shrink down and scroll differently than it does when you're looking at a desktop So you want to make sure you have all these things taken into account when you're building your website? Because so many more people are looking at them on mobile devices than on desktops anyway Yeah, I mean you see everyone winding around bumping into things staring at their phone They should hopefully be staring at your website. Hmm. See or your app or anything. So exactly Other things that I have going on is has also made a little bit hard for me to produce some of the content is we've been onboarding new clients I have another proposal to do today Ineptitude scales really really well There's a wonderful way to put that yeah, so think about that for me think about what he just said But it's a wonderful way to put it in aptitude scalable And I'm not one to I know a lot of really good IT companies out there and I rarely ever We don't we we may compete for clients But they rarely leave those other IT companies for us because they're competent and good then we have the other guys and Yeah, then there are others then there are others I mean it's a client of ours that they've used a combination of us to do part of their deal But we do not all of their IT and that's because This guy really hates me and he sends me really he says some wonderful emails But he's clearly inept and they got hacked again, and they've finally had enough of him And the reason I kept him for so long was because he is somehow connected to the original owner of the company Who is now retired and the people in charge now are like we've had enough time to cut the cord Yeah, they they experienced 30 hours of downtime At their company for him trying to get it and part of it was he oops the restore and had to start a restore Over so his predicted 12-hour restore turned in a 24-hour restore, and then it didn't work right still so it was bugs to fix and The guy just becomes unresponsive didn't call back very often and on Surrendering passwords. He's being very Obsinent and just put the same password for everything first. I said you have the same password for everything They said we don't think so They were right because the passwords didn't work. So it's been a back-and-forth now the good news is As I kind of suspected he saved all his passwords in browsers on different logins that he had I was able to extract all the passwords from the browsers now I have all the passwords to the things all around the office by doing a dump of all the passwords from the browser Which I completely preach against he's never mentioned to them that it's a bad idea He told them yeah go ahead and save them all and so they're learning that they should be using a Password manager such as last pass I sent them a link to the video and they're like wow How come he never tells but this I said well good news is he didn't know about it So I was able to extract all the passwords He didn't surrender to you to all the different devices and equipment they have as they have a lot of things that have web logins So I just dumped all those passwords out of there now while it worked in this instance This is absolutely not anything we recommend doing yes Use a service like last year's like this but because of his general follies of being an IT guy It's not a very good one Yeah, he also didn't practice in-state security himself Which is kind of this time worked to my advantage because now we don't have that they were debating what they have to get an attorney involved Just to take over IT because the guy just was obstinate. He also has zero documentation. So we map their network I've shown how I've done this before we use in Tool called zen map. I have a tutorial on it We create an entire maps of their network by leaving a computer plugged in and just scanning the entire thing finding all the Crazy devices that are on our network They found things they didn't know we also were looking for Does everything match what they think they have? Because sometimes they go what's this device and what's that device because I that's kind of the questions we start asking I want to make sure he hasn't left different things because the size of the company you can easily hide some things This is a larger company that we're taking over for so they tolerated this for quite a while But yeah, so we got that I'm gonna do the we're still using the new max backup I've got to do a tutorial on it. That's my only real complaint about it was it took me longer to understand the product because their documentation is a little Lackluster so we're gonna do actually a good documentation for it I've thought about reaching out to some of these companies because the folks over at screen connect They you know, they really liked my review of screen connect They're like wow you have a great video on it. I said, well, thank you and you know, yeah So I thought about as a service because I enjoy reviewing their products to say hey, you know Hook me up with a dealer or something I read your product But I also want to make sure it's a clear line if I'm reviewing your product and we're using it's because I like it Not because I'm being paid to so I got to figure out if there's some way it is not to cross that line Or maybe they can just throw money at me. I don't know. I don't know how that would work like He's like to throw money. Yeah They have to pay a video I would fully disclose in the description and in the first part of the video going I was paid to create this sure to be an honest review though Yeah, because well, then it becomes less of a review and more of a tutorial But either way with the max backup It's a tutorial because it will serve a couple purposes one of them is going to be a Tutorial for my other employees who work here So to make sure they all have a clear understanding of how all the features work I was set it up and how to do restore and then you know break out each of those step signer It's actually not that hard to use But there was just some clarifications that I you know, I think could have been done with a video really easy So and it will be done by me The video and that's just the product we're using right now and it's nice to have a video on it Absolutely. Yeah We're gonna be also something that I suppose I'll be working on more is our social trek blog Postings talking more about how you can use social in your business and some strategy ideas and things like that So be on the lookout for that stuff If you're in the area, you'll probably be seeing my Smiling face a little more going around visiting some folks and like we said earlier if you want to if you have a Business and you want to be on our Thursday vlog. We will come to you and do a remote. Yeah, all this stuff's portable Yeah, you can you can put our muscle behind, you know your business. Yeah, so yeah, we want to go sit with some of our clients I you know, it's funny I'd like to see if our manufacturing clients would be interested in showing someone what they do because it's interesting to me going to These place one of our larger clients is kind of like it's like a giant how it's made It's a three-story manufacturing building that they was they manufacture packaging and it's really kind of cool I mean they make the entire cut package the styrofoam so everything for Companies and you know one of our clients is actually Ford Motor Company So they make some of the things that go inside the The packaging that comes with special edition cars and things like that's really cool stuff They do I you can say I'm a nerd for this or whatever. I love packaging Like to see the way that they come up with you know the ways they come up to package things. Yeah, it's fascinating to me Yeah, we we actually didn't want to throw a box away the other day for some network equipment And I'm like the box is so well made. Yeah, and I gotta give it to Apple I'm not a I'm not a big fan of a lot of the Apple products, but they do beautiful packaging like Apple stuff's fun to open That's actually the only reason I buy Apple is oh, I wish we and the Lenovo does it too So the lowest got this box in the laptop before you open it's one of the yoga ones Sets down recessed in there like you can't grab the edges So you flip this lever and when you flip the box is over it's got a little magnet hold the box again I mean flip it over the laptop rises to flush so you can just pick it up, and I'm like oh That's just that's awesome. Yeah, so this just all those things is like that little bit of packaging is like Oh, that's and we were just playing with the box Like this is so cleverly engineered like I want to keep the box Yeah, for sure so it's about Yeah, I try to think of it maybe we want to do some more job projects with you To gotta get out there gotta get to it. It's always a struggle between Dragon is camera covered around and documenting things that may be in violation Of course, we can't ever take this any medical clients or anything like that There's just too much risk involved, but we have a lot of them We have to follow the HIPAA rules and all that stuff So it's like er can't really share any of that But I'm working on more documentation in ways we can generalize it And I'm gonna do a video soon on how we do documentation my plan for this weekend provided It doesn't become amazing sunny and 70 where I want to go outside is to just hammer videos out And create content what I do a lot of times is for my creation I come up with all the ideas that I'm gonna cover and then I just do them all in one day That way I'm like just set up for it. It's quiet. It's usually after hours on a Sunday And I just spend Sunday doing it. So Yeah, so it vanished. It's a way to do it Yeah, you don't have to just because you're you want to produce content and put it out weekly doesn't mean you have to produce it Yeah, one week. Yeah, but they want to release it. It's how we release it We'll cover that in strategies. Yes, and I wanted a firewall video that I talked about last week that you know How firewalls protection how the threats manifest themselves inside of small businesses? There's a good article, and I think I can find it again on Inc.com and it was about how a small pie company can teach small businesses about security And it's because they're small businesses that've been hacked and they have a lot of statistics for the small business been hacked because they just They're such an assumption that only the big companies our targets in reality is the small businesses are just low-hanging fruit that you never hear about We hear about it, but then we can never report the actual company names or the numbers So once again, they somewhat go unnoticed unless you actually watch our videos here Hopefully some big news site picks it up But this is a constant threat of small businesses because they often don't have adequate security or have a clear understanding And they just think I'm not a target.  There you go. So enjoy that Enjoy that. See you guys next Thursday and hopefully I will knock out a few videos in between As always like and subscribe and all those fun things. Yeah. Hey, thanks for watching. All right. We'll see you next week


Why is this game such a big deal? It's probably the gigantic story or the 14 playable characters, which is to this day the most in the series, or maybe the main villain, Kefka, or maybe the battle system, or maybe the incredible soundtrack. It could be any number of things, but be forewarned, I'm not going to slobber all over the game, but I will start with the obvious highlights. For a brief overview of the story, there's an evil empire and there's a rebellion, referred to as the Returners. Some of the evil empire's slaves named Terra escapes their control after discovering an Esper, or a magical being, but Terra has no memory of anything about her past. Some dude named Locke stumbles upon her and agrees to bring her back to the returners until she can regain her memory.  What I really like about the story is how high the stakes are raised. I don't want to spoil anything, but I will say that suicide is involved. Offing yourself in a Super Nintendo game? Woof, that's heavy. And the way it's written and the way it comes about is completely organic. You really do feel the weight of the successes and failures that have been throughout. But yeah, the story takes no shortcut.  Another major positive is the battle system, which is very detailed, featuring all sorts of cool stuff like customizable magic, formations, relics, tons of items, tons of equipment, and special abilities. Sabin, for instance, has you input a sequence of Street Fighter style commands to execute attacks. His brother Edgar has a weird set of tools to choose from. Locked steel stings from enemies. Realm, uh, draws people? Okay. Setzer has a, uh, uh, slot machine? Okay, whatever works. The point is, they do a great job making each character truly unique. It's very well done, and it holds up really well over time. It's by far the most enjoyable part of the game. My only criticism of the battle system is that it could be a little more balanced. The game really encourages you to employ all sorts of tactical maneuvering with formations and equipment or whatever.  Regardless of what else you do, I know RPGs are pretty easy by nature because of the ability to grind, but you pretty much have to disqualify yourself from using certain attacks if you want to make the battles even a little interesting. Still, I don't think that's a big deal. It's not a deal breaker or anything like that. Final Fantasy VI also shows the right way to do quick time events, particularly during the opera, for example. You fight your way to the villain up in the rafters while the show continues below. You have to make sure you select the right lines in the opera or you'll get booted out. It's an interesting idea, and there's all sorts of little instances like that that add quite a bit of variety. So I completely understand how someone could become obsessed with this game because there's an overwhelming amount of special abilities you can obtain for certain characters. There's collecting all the espers, there's having each character learn every spell from each esper, plus there's stuff like the Colosseum, which is a rather ingenious idea where you can bet on items and battles to win better items that are otherwise unobtainable. The OCD collector gamer type would be in Hog Heaven here. Now, here's the part where I have to be careful because I'm going to criticize Final Fantasy VI, but let me break it down this way. If you look at the opera scene on YouTube, you'll find comments like, one of the most touching moments in video game history and it's such an iconic scene and I had tears in my eyes and... It's like, okay, why is that? What exactly is it that makes this scene so emotional? Because here's exactly how the scene comes about. Our heroes need to find a way to get into the evil empire, so they decide to seek out this rich dude, who's a setser, who owns the only airship in the world. They find him at this opera house trying to get out some broad named Maria, who's playing the lead in that night's show, and he plans to kidnap her so they can elope or something.  And so Celis agrees to perform the opera accordingly and does so. And that's it. It's strange because the opera seems like it should be this big sweeping emotional moment and yeah, the music is undeniably fantastic, but it's not this big moment at all. The words behind what she's singing don't really have any meaning because she's merely filling a role. She's not singing about anyone, but the character in the opera. There's not even a read between the lines meeting anywhere. In fact, the entire scene isn't significant in the slightest to the main storyline. So then what's the point? I think the developers at the time wanted to make the biggest game possible for the sake of demonstrating what was possible in a video game.  The development team wanted to prove something and why not? They should absolutely be commended for their efforts for making an incredible game. But it needs to be pointed out that the story is just huge for the sake of being huge. Why? To serve as a high profile example of how storytelling can be done in a video game. And as a result, there's many, many playable characters for no other reason than to have many, many playable characters. There's an opera scene just for the sake of having an opera scene. And yeah, at the time it's very impressive, but now it rings a little hollow and a little self-indulgent and it makes the game more of an exhibition and a little more of a show-off instead of being immersive and engaging. Oh yeah, the depth and the narrative makes Chrono Trigger look like an episode of DuckTales by comparison. But by the time you get to Cyan's backstory, which is really over the top, or when you're introduced to Strego and Realm and all the stuff with that general Leo guy, it's like, enough already, I just don't care about these people. Let's get on with it, please. Another issue with having such a huge story is that it leaves too much out of the player's control. There are times when you can't help but feel like a gopher running your party from one long story sequence to the next without much to do in between. I know this is kind of an old, tired talking point, especially with modern games, but if I wanted to read a book, I'd go read a book. I'd like my game to actually be interactive more often than not. Don't get me wrong, a huge part of what makes Final Fantasy VI unique is the narrative, and there aren't many other games on the Super Nintendo that had me playing for hours just to see what would happen next. For instance, it truly has one of the all-time great video game villains in Kefka, who is one screwed-up character. He's like the video game version of Randy Macho Man Savage. He's always entertaining, but he's always got this unstable edge that can freak you out. There are a lot of laugh-out-loud moments, like when you meet Gaw and random Kefka stuff. And there are a lot of sappy moments, too. Like, geez, could they lay it on any thicker with Cyan's backstory? And some of it works, and some of it doesn't, but it does have a very satisfying ending.  I don't mean for the criticism to come across that harshly, but it does go with the territory of being such a high-profile game that's been raved about for 20 years. I do want to get across the fact that I do enjoy the hell out of Final Fantasy VI, and there shouldn't be any doubt on any aspect that it's easily in the top 10 Super Nintendo games ever made. And yeah, ultimately Final Fantasy VI is great because of its wealth of creativity and battles, because of the incredible soundtrack, and because of its undeniably memorable story. But as time goes on, it rings just a little bit hollow for me because it comes across as an overwrought exhibition of game design, albeit a very impressive one.


I imagine most of you are probably on Twitter. So feel free to take photos and post it on Twitter. Use hashtag FOSSTEM, F-O-S-D-E-M. Let's get trended on Twitter. Because there's no actual official registration, FOSSTEM doesn't really have a good explosion like in land yard and websites like that. We should make it like a trend. Okay, so this talk is about continuous integration at a distribution level. And here we are with Martin Pidge. Martin Pidge works with Ubuntu, I guess. He works on Linux, plumbing upstream, and in Debian Ubuntu, SystemD and stuff like that. And have created and maintained a distro-level CI in Ubuntu. Please welcome Martin for the talk. So welcome, anyone. And yeah, it's a pleasure to speak in front of so many people. I'm overwhelmed. And of course, lots of thanks to the FOSSTEM organizers and the volunteers. It's a great conference, so thanks a lot. So yeah, I've been in the Ubuntu developer since pretty much its inception in 2004, and a dev developer even longer, long time. And in Ubuntu, we have practiced continuous integration at the distribution level. And it is, we do test-based gating for like the 30,000 packages that we ship. And as far as I know, we are still today the only major distro which does it with that rigor. Like test-based gating of the development of the distribution. And so I want to share some thoughts and experiences and maybe convince you that it's a good idea. So what I'm gonna do. So I first, why did we do this? And then how do our tests look like these days? How do we use them for gating? And then I want to lose a couple of words about the infrastructure where we run all this because at this scale, you imagine it's not accepted easy. And then how did it change our life doing all this? And then we should have about 10 minutes of Q and A. So where did we come from? In the first few years of Ubuntu, we had this six month development cycle. And in the first four months, this was basically anything goes, toss everything over the fence, feature development. And then we hit feature freeze. And in the remaining one and a half, two months, we tried to fix about half of the regressions that we introduced in the first four. And during that time, as a developer running the development release, it became kind of a day-to-morning exercise to fix your broken boot or fix your broken x-word or try to restore half of the packages that you upgrade, removed underneath you because you're not paying attention. So while you learned a lot about how all the system worked, it was certainly not a very enjoyable experience for non-developers. And as a result, we did not have a lot of non-developers using the development release. And as a result of that, we were losing a lot of potential feedback for how the development series worked in real life scenarios. We didn't get much critical back reports. And as a result, a lot of regressions trapped into the stable release. And another problem was that in the distribution, you're facing a lot of archive-wide changes. Things like library transitions or attribution in your Qt or major Python version. So back then, these things were often not finished. People basically tossed them into the fence, half done, saying, oh my God, I have this deadline to make or there's feature freeze next week. So it's dumped it all in and sorted out later. And in the event, it became someone else's phone. And usually in the end, the release team had to clean up all these bits and kick out packages and desperately rebuilding packages against the API. And it was just a horrible mess. So once Ubuntu became popular enough and was being used in mission critical deployment and commercial products, this was simply not good enough anymore. So we set down a straight goal. The development series must be stable and usable and safe to be used at all times. Nobody's able to knowfully break it anymore. And so by using this kind of continuous testing and continuous integration, we basically want to ratchet towards perfection. We never ever wanted to have a bug which we can automatically detect and then in the development release. And yet another problem was that many upstreams already have test streets. It's just that like every project's done differently. So there is no uniform and thus automatic way to actually run them in a distribution at the time when you need it. So that was the situation. So initially like everyone else, we started with some rather naive approaches. So we set up a QA team and that created a similar duration of standalone test streets. Like Ubuntu desktop tests, Ubuntu server tests and whatnot tossed them into Jenkins and run it on the day and then basically pestering people about you broke the tests and please fix it. Well, of course, none of that really worked because it's too late. I mean, at that point when they detected it, the damage was already done. Also it's never an inflamed game. Developers point you to QA, but it's your tests that are broken and QA point you to developers. No, it's a software that broke. Also none of that was really reproducible and easy to replicate for a developer. So the conclusion from that was the only people who are able to be responsible for the test in a meaningful way are the developers of the very software. So developers are responsible for testing, they're responsible for gaining and the QA team basically only does the infrastructure and making sure that the test results actually arrive and provide some kind of result to people. Like if someone has a question, how do I simulate, I don't know, a Wi-Fi card or X or whatnot and the QA team could help you with that. But it must be possible for every developer to replicate this test easily without reading lots of manual pages. So the idea was instead of having the centralized test, we would put the actual test for a source package in the source package itself, where developers can develop them by themselves and they don't need to go to any QA process. And these tests would then be triggered when either the software itself or any of its dependencies would get up. And we, of course, would use these tests for actually gating. So once these tests break, the package doesn't land, period. And this was named auto package test. This is both the name of this whole kind of test and it's also the name of the tool that you're actually running to execute the test. And back then it was submitted as a Debian enhancement proposal number eight, which is why you can still hear a lot of people calling them Debian test, so don't be confused. How do these look like? So this is one of the simplest and oldest tests that we still have in the archive. So it's the kind of toy example where you begin. So every source package which the test needs to have the Debian test control, which gives the test metadata. Basically it enumerates all the tests that we have and it says, like in order to run the simple gzip test, we need to have these and these packages installed. And you can also give other properties, but I'm going to this later. This one doesn't need anything. And so for every test that you enumerate there, simple gzip, you'll need to have an accompanying Debian test slash test name. And if you see here, this can be very simple. It's basically a slope test that gzip and gansip does what we expect it to do. And so the contract is if the test that you run exits to the zero, then it's passed and otherwise it's failed. So you can write the test in any language. You can compile it if you wish. So that keeps you pretty flexible as a developer. So once you have the test, how do you run them? So this is the auto package test program. This is the thing responsible for creating a temporary test that. Copy the tests inside that, run it, make sure that all the logging is right. You get the results back and any kind of artifacts back. And you can influence the final behavior of the test, but details aren't that important. And it supports various packets there, which you can run the test here. So one of the oldest ones that we have is like a churrude, which you see like in the second row here. Like churrude is basically Debian's tool to manage, build and manage a couple of churrudes. So very simple. But of course this kind of limits the things you can do. So you can't start services, you can't interact with a kernel, because that doesn't work in churrude. So after we created all the package test, these new fancy technologies came along, QEMU, containers, whatnot. So these days, we have backends for LXE, LXE, QEMU, arbitrary SSH posts, or cloud instances, ADB if you want to run something on Android. And these days in production, we're actually using QEMU by way of using OpenStack instances. So you have a full virtual machine to mess around with. And we use that for X86 and the PowerAid, and for the other two architectures, ARM and S390, which we support, we run them in LXE because we don't have OpenStack integration yet. So a slightly more useful example is the schema that we use in a lot of libraries. So we basically do a simple compile link run cycle to make sure that the library that we ship or update still works. So this is the simplest test data data. It should be straightforward, I hope. And this is what the test does. Basically, it creates a simple C file, which, like in this case, it interfaces with SystemD's logindy. So we instantiate the logindy monitor, call it and basically make sure that the smoke test is right. Then we call GCC on it and run the program and if it succeeds, the test is good. So at first sight, this looks very simple. But on the second look, there's actually lots of things that can go wrong. So for example, we update the program to the upstream version and our development package is missing the specified new dependency. So it's trying to use a header file which is just not there. Or it actually forgets to install new header files. Or package config followed wrong and we did have new upstream versions which ship with a broken package config. Like all of that happens. So as I said, you'll see the same pattern and lots of libraries and everything that went to these days. So it is just... And a more intricate example is the, for example, the one from SystemD package where we run an upstream test. So upstream ships the test.net for the test.py, which while exercises network D. And as I said, normally the test is to live in that in such test. But with the test directory clause, we can say that the test is to be looked at in a different directory. So in this case, test slash network test, py. And this is a way how we can interface and run upstream tests by saying basically just giving the metadata for the test. And that's it. And you also see the new restrictions column here. So this is where you can say like, how well isolated does my test that needs to be? So in this case, we need to have root. And we also need to run it at least in a container because network D, you need to set up a couple of VEs and start services. And for example, this wouldn't be safe to run into root. And there was one like a stronger version of that would be isolation machine, which you, for example, using the network manager test. We had the logic kernel module, like Mac <IP_ADDRESS>, HWSU, which simulates Wi-Fi devices. And this is the kind of stuff that doesn't work in the container. Also our kernel tests, they do really nasty things to the test bed and beat them to a pulp. And so you really want to nicely contain this in the VM. It's just not safe in the container. But I don't think you want to go too much into the details of that. I just want to give a broad overview of what's possible there. For example, in system D, we have tests which simulates the suspend for seeing how login it works or closing and opening up the lid. Or we create this, we create a loop partition to make sure that system integrates this properly. Or we install a couple of, the same mission critical packages like D-Bus, network manager, or XOR and LightDM and make sure that, whatever that means, system D upgrade everything starts and there are no failed services. And that you can reboot the test 20 times without failures. So, that's it. When we started this in like 2012 or so, we still had a very few tests, as you see. And then we started to introduce this into Debian. And since 2014, I got happy that Debian also runs these tests except that it doesn't use them for gating it. And you see the adoption curve is really nice. These days we have more than 6,000 packages which have all the package tests and considering that you also cover their dependencies implicitly, that's quite a nice coverage already. I mean, it's still far away from the 30s house but it points to the right direction. By the way, the big leap that you see here was kind of sort of cheating. So we figured out a generic way to test Perl and Ruby and DKMF modules because they all look very similar. So we have a central way to run them. And so that explains a nice job. So, now that we have the tests, how do we use them for gating? Initially, there's a developer who prepares a new package update, say, GDK. And uploads it to the distribution. That happens with Deput. That's just the standard Debian tool to upload a package. But this does not land directly in the winter development series. But instead it goes into something which we call the proposed pocket. And this is a kind of an overlay on top of the development series where all the new stuff is being uploaded. And it acts as a staging area for basically CI. And this one has no human users because by definition, that's the bit that is broken. And then we have a thing called proposed migration set of scripts. If you're up on a Debian background, you might also know it as Brittany. And this then does all the necessary checks on it. So a test that the new GDK builds on all the architectures where we expect it to. That it is installable everywhere. That it doesn't break installability of other packages. So we do this for all the 30,000 packages and that ensures that library transitions can only be complete and you don't knowingly introduce uninstallability. And for the purpose of this talk, it also triggers tests, both of GDK itself and everything that uses GDK, like all the reverse dependencies. And only once all of that is good, proposed migration then lands either GDK itself or the group of packages that are kind of belong together into the development series. And with all of that, we ensure that packages develop. They never regress on architecture support. They're always installable and in theory at least the tests never regress. Well, that's the theory part, but for the most part of it. And so of course, in this case, if GDK break something, the developer then might need to do further uploads to adjust GDK users to like new idea or what not. So, and this is a page which we automatically generate. The developers can check the status of their packages. So instead in this example with GDK, we for example, see here that like it didn't build on powerful C64 yet. And on the IS-86 that's an uninstallable binary package. And that's like one reverse dependency unity just failed the test. So this is a very simplified output. So normally we test on five different architectures and not just one. And also, we wouldn't actually start the test before we know that the package builds an installable because otherwise it's pointless, but like it should illustrate the point. And this is actually still a very simple case. Consider updates of GDK, Python, Perl or Hapt and they literally trigger thousands of tests. Like every time we upload GDK, we trigger 5,000 tests times five architectures. So our machine retakes like about two days to grind to that. But the nice thing is after these two days, you can actually lend the stuff with confidence instead of just saying, yeah, I hope that it breaks nothing because it always does. And of course, the exact same thing applies to updates of stable releases as well. So it's the exact same thing. People upload to say Xenio proposed, Xenio is our current LTS. And then it goes to all this machinery and only this green, we can actually publish it. So that's the structure of the test. I want to explain a little bit the infrastructure on where we run this, both because I haven't really seen this kind of structure anywhere else and also to be completely honest, I'm a little bit proud of it. So like many other people, we of course start them with Jenkins. And that was okay when we had like 20 tests in the beginning, but already then it was quite brittle when every update of Jenkins then you read through these three screen folds of Java exceptions. And it's pretty hard to maintain and it's really not easy to replicate locally if you want to develop the infrastructure. And we got a lot of losing test requests and it's a single point of failure. And at a scale of 30,000 packages, the test doesn't work. So we needed something better. So I sat down the other day and designed something on the micro-service architecture, which should use standard child technology as far as possible. And I wanted to have small and loosely connected components. So where it starts is what we could call a policy entity. So this is the thing which wants to trigger tests and they're waiting for the result and then making decisions based on these test results. This can be proposed migration that I was just explaining for the distro, but it could also be GitHub, for example, where you want to use it for creating pull requests and there's a couple of other consumers for that. And the only thing that this policy entity does is it puts test requests into an NTP queue. So we use rabbit for that, which is basically standard implementation. For those who don't do NTP, like very simply, it is a job distribution system. So you have a couple of queues where you can put in requests and then consumers can take out the request and NTP ensures that all of this is very robust. It is atomic, you can arbitrarily parallelize it and it is a very simple API. So basically a consumer of that, if you want to retest request and do your thing and then acknowledge the result and it's basically five lines of Python or you can do it in a single line of shell. And like NTP system ensures atomicity or if anything breaks and the request doesn't get fully processed, it gets hunted back to the queue and the next consumer basically reattends it. So what it does in our case, we have a bunch of workers up there and they basically take requests from the queues that they can service. So we have workers for x86, for example, and then separate standard workers for power to see because these tests run on a different cloud. So you configure the workers a little differently depending on what kind of service they can support. So they take the request, so the auto package test command line, run the test in the cloud, copy back the results and store them in a permanent thing which is written in this case and then at the end acknowledge the request. So if anything goes wrong in the middle, it goes back to the queue and you never do the test request. And like at the moment, we have many dozen of these power workers so that it can scale with the size of the infrastructure and it's quite painless to set up. And for those who don't know Swift, this is basically open stacks, standard, like open stack level can give the simplification, but it's basically a distributed network file system. But as a simple REST API for querying and uploading and downloading is basically you get someone a URL that you can watch it in your web browser or download with a curve or URL or anything. So it's, and it's pretty simple to use because it's a standard component of the cloud. I as a consumer of the cloud don't need to worry about it and it's separate from all the instances you set up so it's basically the data is safe there. And finally, once the logs are all artifacted to our Swift, then the original investor can poll for the results and wait for the test results to arrive and then do its decision and the loop is closed. And finally, there's also a web UI, this results browser, which developers can use and it presents the test results and they have links to the artifacts of logs and whatnot and they have organized history. So it's mostly a developer tool, but it's completely independent of the workers and the policy entity. So it's replaceable and it's not critical. And the whole infrastructure has some due to charms so that's the open tools way of cloud deployment management. So it's very simple to deploy all of this into three local Lexi containers and with the exact same one command line, you can also deploy it into production on say open stack and they can redeploy the entire infrastructure within minutes without using any data. And yeah, the whole thing is maybe two or three hundred lines of code, so which is about the same size as a single XML job for Jenkins, just say. So now we have all this, how did it change our life? It provides an effective coward and sticks for developer. The coward is of course the developer, the better you make your tests, the harder it is for other people to break the software because of this reverse dependency triggering. For example, new kernels have a tendency to break Lexi or f-almore or new X libraries or servers have a tendency to break Qt and in turn your version of Qt tend to break KDB. And in everything in the window, we have tons of KDB tests so our Qt maintainer always has a hard time to land new Qt versions and I guess they are swearing on this all time but as I said once you actually get it to green then you can land it with confidence, that's nice. The other effect is that these cross package changes I mentioned, library transitions and so on, they either land completely or not at all, they will be forever stuck in the post if you don't finish it all the way through. And this both ensures that you always have good development series and it also makes the release seems like much more easy and of course as a developer you can wind all the things all the time you want against the machine but after the deadline and need to meet feature freeze and whatnot, the machine is very patient and it won't give you anything so you need to do your job properly. But of course like there is no free lunch it comes with a cost. As a developer if you have a large amount of tests then you need to keep the tests actually running and for the most part that's of course they can break with new upstream versions or changes to code but sometimes they also break for entirely unrelated reasons. Like sometimes the cloud configuration changes or an external website that you are talking to in your tests changes almost out in which case it's probably an actual regression in your software but still, I mean people are chasing a lot of test regressions which are not entirely obvious at first. And of course also having test infrastructure which is able to process tests at this scale is not entirely free. So we are basically building a reliable CI service on top of a naturally unreliable pool of hardware and clouds. So you need to invest a little bit on keeping it running. I mean this fine gentleman over there took over the maintenance of that and you can probably tell you a lot of the glory details of tracking down Chrome loops and whatnot. And another big problem that we face in the bunch of downstream is be important more of broken tests in Debian. As I said, we have so many tests now and most of them come straight from Debian but Debian doesn't gate test. So eventually the failures land on us. For example, every new Ruby version that we import tends to break two of our modules and in the buncher we just don't have the manpower or quite honestly the interest of tracking them down so we just tend to ignore those. But by last, after a few months of using this people can use it and nobody really discusses the if anymore. I mean people do see the value. So the thing that we do discuss about is tweaking the policy and making tweaks to the infrastructure or maybe discussing how to add in your architecture to it and so on. So in general working people feel a little more safe that way. So, and that works really great for software which is native to Ubuntu but we are the upstream. Say install or Jojo or Unity or Wacmer or Snappy. But as I said, it doesn't work so well when we are just the downstream. So of course these tests find bugs and keep them from landing in the development series all the time but we then need to deal with them and file bugs upstream and do our compatriots and so on. Are we doing this? But it points out that running them in Ubuntu only is too late for most of the bits because in Ubuntu most of the code in Ubuntu just basically happens to us, it gets important. So what we really need to do is to push all of this upstream. So tests are already running in Devian and we see here Kwin failed like half a year ago and nobody notices because nobody gets the algorithm done. But there's work underway to actually enable dating in Devian as well and yeah, then this won't be a lot of my stuff. But the real place where you want to hook this in is upstream. I'm heavily involved in the system D community as a developer both upstream and downstream and back then when we had to do an upstream release of system D it took me like a week or two to figure out all the test regressions that it caused once I packaged in the upstream release and it's a big pain point. So one of these days I tweaked our package to be able to build an unmodified upstream source directly from a pull request without any patches and then adjust the tests so that they would drop some Devian Ubuntu specific expectancies and so that we are basically able to run our downstream system D test straight on the upstream source and then integrate it all this with GitHub. I mean GitHub is a wonderful webhooks interface which makes it quite easy to interface with. So that nowadays every pull request needs to pass exactly these tests. So this is an example of what things go on and this is no developer's habit because when this happens the change is still fresh in the developer's mind it won't land until it's all fixed and as a result of that every commit and extension, every release of system D is buildable everywhere, it passes the tests and then we have good releases finally. And so that easily enables things like daily PPAs so basically have a script which takes coming master, buys the Devian patches, runs the tests, uploads it to PPA if everything is great and so it's fairly safe to use these daily builds because we know it's not gonna break the computer. And basically packaging a new upstream release now becomes an exercise of why do we do a change which is really the thing where you want to be at. And this is not really limited to system D so this is generic facility and we can interface with other projects in GitHub. Of course it's always a capacity issue and you need to negotiate the exchange of some credentials but by and large this is possible. Okay, well thanks for your attention so we still have a couple minutes left for questions and if we don't manage to do it here grab me in the hallway or write me an email or find me on RSC I'm happy to talk about this stuff, I love it. How do you handle if a package has to land together with another package? This means, for example, is a new cube version which has a new interface which process, there's also a new version which person's cube and they have to land together. Yeah, so the question was how do you handle groups of packages which need to land together? Yeah, this is done by the proposed migration bits so basically for example you upload a new GDK which breaks the old unity and you then upload a new unity which depends on the new GDK then this Britney script is able to figure out like GDK is uninstallable, unity is uninstallable but both together are uninstallable and both together if I run the test of the new unity against the test of the new GDK, then it's green and then it lands them both as a group. So basically if you do a library transition from Python 3.5 to 3.6 you will get like a thousand uploads in the station for posts and once you fix the last one then all lands is a giant chunk. And this is the real point to get these cross package updates done in atomic in a safe way. And this involves a lot of reasoning about dependency trees and it's all a bit of black magic. I didn't write this so this is the work of Debbie mostly for me and I'm very happy that we could just reuse this. So Debbie does use that part also. So stuff that migrates from unstable testing and also only lands with library transitions are complete and like packages build uninstallable but it doesn't do the test. So with the test, does it get run? Got it. Yeah. With the test, does it get run by the Adverb in test control file? How do you personally feel about the distinction between tests that are testing the upstream code versus tests that are testing the code as it's packaged to find on the system? So do you care whether the tests are running against the code as it's put into the package location or do you mind if it's only running against it? And I'm not sure whether I got it but so the only thing we, all the package tests is the actual binaries that are in the archive. So we upload something to develop post and that gets a couple of that and they get installed into an actual system and you only run those. Yeah. So you test the actual installation of the binaries. Yes. But when the test suite itself is run it's often testing a source tree of the upstream code not the actual installed version. They can be the same version but it's actually not the same file. Oh, you mean if you run like a main check kind of thing? Yeah, because you're actually in the developer environment at that point now they should be the same thing. You haven't actually got the assurance because a lot of the test policies that we need to use can't actually test in slash user, slash limb. They have to test in an unpublished version. Right. So the question is when you try to run upstream tests which are designed to run against the build tree and start up the system, how do you do this? And this is indeed like one of the challenges. So if you have your own thing right here this will test the system installed package. This is not against the source tree. But if you have packages upstream which are only one in my check kind of style you have to modify them to use the binaries installed in the system because otherwise testing the build tree is not testing enough. You might have a perfect development in build tree but then mess up the packaging. For example, you forget to actually install a file. So the tests need to run what's in the system. Depending on the test, is that simply not all this possible? Sure, I mean for some upstream tests it's easy to do like you introduce like a path arrival or maybe a dash install test or something. Even automake has a make install check interface but not a lot of upstream software uses that. So if it's easy to change the upstream test rate to run against the system then of course it's almost preferable to use that. Sometimes it's hard and I would say then just write a smoke test. So the point of that is not really to exercise all the little early details of your API. That's what unit tests are for and this is fine to run and make check. The point of that is to test integration and packaging so that you can write your own, let's say smoke test which only makes sure that the package by and large works. Let's say for Apache you install the package and you put like a bar www index.html and you do a wget and make sure that you get back the file you expect. And this is actually in this case, maybe easier approach. But I hope that since more and more people are doing this upstreams are more like willing to take patches which make your test suite run against the install system as well. For example, no patches for well with the install test and for example the glib package actually has a binary which has glib install test which you can just install and run because they were doing the same thing in OS 3 for example. So it becomes more popular. But it is like a problem. Yeah, and it's not a problem that you're currently trying to monitor or look at the actual test control script file and work out whether they're trying to do is left to the maintainer to do that. Yes, that's right. There is a potential problem there with the gating that you're using because sometimes the gating is actually operating against the not what you think it is. You're not asking it to account or not. And I must say, even if you actually just run the upstream test suite against your build tree, that's still better than doing nothing at all because it still gives you the thing that you, if your dependency changes and then your build doesn't work anymore because for example Q changes API, you have a K win and it doesn't build anymore. It still points that out and it holds back to your tune until you fix K win. So it's useful. It's just not, let's say using the full extent of what it's supposed to do, but these ripple tests are definitely useful by themselves. And in fact, that's what KDE actually do. So they only basically run a check. And yeah, that's okay. There's one question right here. Ah, okay. So lately more and more program are using Riltico and the test suite in some cases are going to generate some intermittent issue. So how do you manage those? And say multiple? Intermittent issues like in the test suite. So sometimes it's going to work, sometimes it won't. I know that it's ugly, but it's a reality for some software. So these are always our favorite of course. And this happens a lot. So I mean, like everyone else, if it fails like one out of 10 cases, quite frankly, we push the reach by button. And the, like in reality, the report that we have here, this actually has a little recycle icons behind regressions, so a developer can just reach by their own test. If it happens like one out of 10, it's okay-ish. If it happens more, there's someone to actually sit down and debug it, sorry. And as I said, I mean, the point of this is it needs to be really easy to reproduce it as locally. So basically with these overpackage tests, command lines that I did when I showed, please. And there is tools which build the M images for you. Stem of the image, M images are basically almost the same than our cloud images. Or we have tools to build these Stem of containers. And there's existing tools to build to root. And there's not really a lot of steps to walk from producing them yourself. And this has switches, for example, the stash of the shell when the test fails, then you get a, you can attach it to the test set and then debug your test and cycle faster. And yeah, this is basically the daily gory bit of what the developer needs to do to fix the test. Because broken tests are useless. Anyone else? This one over there? Oh yeah. Okay, I'll come back after this. Can you tell me about the underlying compute hardware to support all this? I'm sorry, I can't understand you. Can you tell a bit about the underlying compute farm to do all this? The compute farm? Oh, yeah. Yeah? So we run all of this on, so x86 and policy, we run on, oh okay, the question was kind of tell me a little bit about the build farm that you, so we have currently three open stack installations in Canonical, where basically whenever we have hardware we toss it in there. And so we currently have I think 70 or so power workers for x86, so they can serve I-86 and 8064. And about 28 power workers for policy. And as I said, farm and IBM Z-series, we can't run them in the end set, so we use LXC containers for those. And basically you need to scale them to the degree where the maximum possible cure line is still valid.   So this is the capacity issue. And as I said, we not just run distro tests, we also run tests for PPAs, and we also run test farm streams, and they all need to share the same thing. So this is indeed a bottleneck, for those you can basically never have enough. So as a developer you need to be a little patient. I mean if you have the least packages fine, it can go through in like an hour. But for GNC, now there is no fast updates of GNC. But I mean in return, like the more tests you trigger, the more important your package is, and the more the gain in confidence expect pickups. Because if you have the GNC, it breaks three packages which nobody looks at, you aren't gonna discover this until you release. And so it's worth doing that. I'm not sure whether this was your question or you have more specific ones, but yeah. So the point is you're running an open stack so that maintaining the infrastructure is someone else's problem. I don't want to maintain the data center, this is ISS drop, and they do this fantastically. And by and large as a consumer, I do Nova boot and Nova delete, and I'm happy to get by the end. And it just magically happens on the bank. Who had a question here? In your control file, you had isolate container or salation container. This one? Yes, exactly. So how is it like keyword or you have to find another control file to find the container? And no, these are fixed keywords. So these are defined capabilities in the other package test project. So all of these needs root isolation container and there's a couple of others they have fixed needs. I mean sometimes we need new ones, but yeah. So they basically say if you try to run this test in the charoot, they will just skip and say, sorry, your test doesn't have container isolation. And likewise, if you try to run the network manager test on Alexi, it will say sorry, it's not a full machine. So it gets skipped. So it's basically safe. And if you do try to run it on your own machine and it will do bad things. So yeah. So if you want a new one, then you need to talk to like the other package test guys like me and then we need to define a new one. And it does happen. Any other question here? This one has some infrastructure to handle complex applications and for example, this type of application. There's no, sorry? Do you have some infrastructure to handle tests of complex applications or this type of application? Which for example, interact with the user in some cases. For example, you can bomb GTK for one minor version and for example, in one minor version, they change the location of their cherished remodels. Don't change location, it means mine really easy. They are all part of the test path which they try to supply with the application. But your system application just don't handle encodings correctly. Do you have some test for this application? Yeah, so I didn't get the whole question but I think it was about how do we test test applications? Yeah, and we actually do. So originally you only have a cloud instance available so there's a lot of stuff available. In particular there's not XOR to install by default but the test can do whatever it wants to in the end. So we do have tests for example for the install or unity which basically install all the test profits and then configure the dummy XOR driver so that you can start ITM, you can start Unity and then maybe get the screen done and make sure that it works. And we do have tests for testing GTK or you can use Selenium for using Barser tests and whatnot so that actually works rather nicely and we do this. But of course it's limited to what you can do with like the dummy driver so you can't rely on an XOR graphics card there. So in order to do that, I mean we do have use cases where for example we'd like to run kernel tests or XOR tests or an XOR media machine, XOR API machine and we did talk about this and in principle we even have a plan how to implement this so that tests can be clear I need to run an actual hardware but we didn't implement this yet. So this would require setting up real IR machines like with Mars or in the record world satellite. So basically that you can automatically interact with real IR machines and have our tests there. We just don't currently have the capability but the structure would allow it. So some tests will require manual guidance by humans. Sorry? So some tests can be used in that set infrastructure but require some human interaction. Yeah, so this is like anything which uses human interaction like a test which is human interaction this is obviously not the target look here. This is not solving every kind of test case you might have but for the majority of kids you should actually try to run your test so that it runs on the medically so there is for example autopilot which can test desktop applications in a non-interactive way because humans don't scale. But I need them always to be these kinds and it's just not a scale plan, okay? Thank you, test. Okay, well then thank you I guess, yeah?