United States, and it is the Muriel Humphrey Brown and Mr. Hubert Humphrey III. Thank you very much. Ladies and gentlemen, honored guests, please be seated. And most especially, Muriel Humphrey Brown, we are here to honor one of American political history's great happy warriors. Hubert Horatio Humphrey was the mayor of a great Midwestern city. He was a United States senator for 23 years. He was majority whip for his party in the Senate. He was vice president of the United States. He ran for president three times and won the nomination of his party in 1968 when he came within a few hundred thousand votes of the presidency. Now these are the facts of his career, but somehow they don't quite capture him. To get a sureer sense of his real dimensions, one must speak of his nature, his character, his personality. There was in Hubert Humphrey a great joy of life and a truly buoyant civility. He was robust and energetic. He loved the battle. He was warm and affectionate. He was hearty and spirited, and he was nothing if not effusive. When he spoke, the words poured out of him. Some said he was deeply, endlessly articulate, and there were others that said he was downright garrelous. Either way, he'd laugh when someone said he'd never had an unuttered thought. He was a masterful politician. Issues were everything to him, mere strategy abhor. He was no bully. His art was persuasion. He asked that he be remembered as an effective man of government. And he was certainly that. He lived in clamorous times, but he was ever optimistic. He was involved in all the great struggles of his time, and he tried to affect them always for the better. In 1948, he touched the conscience of his party when he took the floor of the Democratic Convention to make a passionate appeal for civil rights. In the 60s, he was deeply involved in the struggle that followed U.S. commitment to Vietnam. He was a liberal who was an internationalist, a liberal who understood that America was great and has serious responsibilities in the world, a liberal who was strongly anti-communist. He loved justice. He believed our Constitution is a living document that is reborn every day. He was a passionate Democrat, small D and big D, who tried to make the world better according to his lights. And no one was better than he at infusing his followers with a fighting spirit. Hubert Humphrey was generous. After John Kennedy beat him in the Democratic primaries of 1960, he dropped out of the race and went to work to elect Kennedy President. It wasn't pro-form of the campaigning that he did for Kennedy. It was real. And when JFK won the White House, majority whip Humphrey mobilized his constituencies, mustered his majorities and helped JFK get the Congress to pass such legislation as the Peace Corps. He may have been generous to a fault. He was famously patient with human frailty and it was said that you could hurt him with impunity. Once an old friend betrayed him and had damaged Humphrey. Another friend, a fellow senator, took him aside and said, Hubert, I know you're not going to be rough with so and so after he did what he did to you, but couldn't you at least be mad at him for two weeks? Humphrey probably laughed and shook his head and refused. It isn't possible to exaggerate the number of people who considered him their friend. Thirty years after his great career began, thirty years after he was elected the youngest mayor ever of Minneapolis, Hubert Humphrey was told that he was dying. He fought his sickness with the same spirit with which he lived his life. A week after his last operation, Humphrey showed up at an AFL-CIO convention to give a long-promised speech. He was thin and won and his hair had gone white.  He spoke for almost an hour and he pounded the lectin so hard it jumped. In the last few weeks of his life as he lay dying an amazing healing process began. He had a watts line and he called his old friends and his old adversaries and one after one he told them, I wish you well. And the calls came in too from all across the country. Old opponents called in and young people just entering politics. Powerful political figures called and obscure farmers. It was as if all of them were trying to reconnect with a part of an unchanging political past, trying to touch for the last time a special spirit and special style that would go with Hubert Humphrey's passing. It's said that a lot of love passed along the lines those last few days. There was a lot of forgiving and a lot of encouraging and a lot of sharing of wisdom. His passing left Washington a lesser place. He left a big silence behind him. He was a fine man, a patriot and he understood that though good men and women can disagree on this issue or that, we must always stay bound by a common love of country. Hubert Humphrey was a robust and active player in the dramas of our time for more than 30 years. He was a vivid presence on the scene and looking back over his career, it's fair to say that his greatest contribution to his country was his life, a life that affirmed the vitality of democracy, affirmed the fact that the democratic process is alive and full of movement and action and great plans and decent dreams. I'm very proud today and very honored to present to Mrs. Muriel Humphrey Brown the Congressional Gold Medal for Distinguished Service to the federal government and the American people in honor of the great happy warrior, Hubert Horatio Humphrey. So it's all right. Thank you, Mr. President, so very, very much, not only for presenting the medal but that most beautiful message and summation of Hubert's goals and life. When we children and wife and friends planned Hubert's funeral, it was not a funeral ceremony that we planned. It was a celebration of his life. And I find that today with this great tribute, this is a continuation of that celebration of that man's life. He was one man who changed, made a great change in the life of our country. Thank you again so very much. Now I'm to introduce the continuation of the Humphrey family in my son, Hubert Humphrey Jr., Skip Humphrey. Mr. President, ladies and gentlemen, honored guests, on behalf of my mother, my sister and my brothers and all of the members of our family, we gratefully accept the medal honoring Hubert Humphrey. I know if he were here today he would suggest that all this fuss over his public life is just too much and not necessarily, but privately he would have thoroughly enjoyed the limelight and attention being given to him today. I remember well the notes and letters I received from Dad telling me of the visits with the President and with other dignitaries. He wanted to share with his family the infatuation, the love he had for his country, its political institutions and its people. We accept this medal for the people that Hubert Humphrey represented, the people who live in Minnesota, the people from all walks of life and all circumstances of need. And we accept this medal as a challenge to keep Hubert Humphrey's faith with the people and the land of America. Thank you for honoring the courage, hard work, faith and integrity of a man we deeply love. As always, I would say it's the words of Hubert Humphrey that best expresses how he felt about life and politics in America. So let me just close with a quote. He said, I have enjoyed my life, its disappointments outweighed by its pleasures. I have loved my country in a way that some people consider sentimental and out of style. I still do, and I remain an optimist with joy without apology about this country and about the American experiment in democracy. Now I'd like to introduce a wonderful friend and a great artist, Mr. Robert Merrill. Mr. President, please join me in singing America the Beautiful. Oh, beautiful for spacious skies, for amber waves of grain, for purple mountain majesties, above the fruited plain, America, America, God shed his grace on thee, and crown thy good with brotherhood from sea to shining sea, once more, America, America, God shed his grace on thee, and crown thy good with from sea to shine.


This talk is a follow up to a couple of questions that I had during my presentation during faculty day. I had mentioned entity that I referred to as a mantidine induced corneal endothelial toxicity and a couple of people had questions about it so I thought I'd get to it in a little bit more detail here. I'm going to start with the patient presentation. This is a patient I saw actually during the end of my residency at the Veterans Hospital in Milwaukee, Wisconsin. She was a 42 year old female with a history of Huntington's disease that presented kind of at the time with very nonspecific visual complaints and part of the challenge with her was that she was difficult to communicate with to a great extent. A lot of information came from her mother who did not spend a great deal of time with her. She was actually in a managed care facility. So over the months prior to her presentation she had been noticed to have increasing difficulty with her motor skills and these things were just attributed to progression of her movement disorder. Her mother thought that the problems were more related to her vision. She was actually having a lot more difficulty seeing her food was really the chief complaint that she came in with. Her past ocular history was significant only for mild myopia and a history of mild corneal abrasions that were secondary to self-induced trauma from uncontrolled movements. As I said her past medical history was significant for Huntington's Korea and had been diagnosed in 2003. She also had some depression, motion sickness and eczema. The motion sickness was related to some of the medications that she was taking. So she had a pretty extensive medication list here. She was taking everything from Tylenol to amantadine. So me having actually had previous experience with several patients that had amantadine induced corneal endothelial toxicity, this was the first thing that I thought of. She had actually started amantadine about six weeks prior and it had been prescribed to her to help control her involuntary movements. I'll talk a little bit about that in detail later on. The rest of her exam showed that her vision was pretty significantly affected. She was hand motions in both eyes, was not correctable to any better. While her visual fields were full we really couldn't do anything as far as refracting her any better. Her pressure showed that they were slightly low but I think some of that may be an artifact and we were only able to actually get a tonal pen pressure on her. And then her slit lamp exam was significant corneal edema with three to four plus decimates folds in both eyes. There's a little bit of surface irregularity but no forcing dye uptake or staining. The rest of her exam was pretty normal. So you can see these are photos that I took at the slit lamp camera at the VA. It was kind of difficult because she couldn't hold herself still for very long. But you can see significant corneal edema here with those prominent decimates folds. There was no inflammatory changes. There was no gutata significantly on this exam. And as I said the rest of her fundus was normal. And we were unable to obtain a paquemetry reading from an ultrasonic paquemeter. But just looking at her cornea it looked like it was greater than a thousand microns thick. So in thinking about a patient with corneal edema I just put this slide in for the benefit of the residents. We think of the primary things, one of the things we see most often is fuchs or any other endothelial disorder including posterior polymorphous or eridocornial endothelial syndromes. Other things to think about would it be inflammatory changes due to things like HSV. Other causes we see very frequently are iatrogenic from intraocular surgery. Those could include pseudophagic or aphagic bolus curatopathy. And there have been several toxic etiologies that have been identified. One is inadvertent benzoclonium chloride injection into the anterior chamber during intraocular surgery. And another is the use of chlorhexidine as a prep in place of betadine. If the concentration is too high or if it's not irrigated out of the eye it can cause significant toxicity including corneal edema. And then the last one is one we're going to kind of focus on today and that's amantadine. So after I had seen this patient, gotten the history and seen the medication list, this was the kind of, as I mentioned at the top of my list, there was no evidence of gutata in either eye, no family history of any kind of corneal endothelial dystrophy. And the patient, you know, temporarily the start of amantadine correlated well with the patient's corneal edema. So amantadine is also known by the trade name Symmetril. It was first introduced, approved by the FDA and marketed as a treatment for influenza. It's really fallen out of favor for that. I think that it's not really shown much efficacy for the treatment of influenza. Other uses though, it's among the neurology community, it's actually continued to increase an interest in its application to patients with movement disorders. Primarily it was introduced for patients with Parkinson's disease and used to treat Leva, Dopa-induced dyskinesias. But it's since been applied to other forms of movement disorders including Huntington's Korea and Tardythian dyskinesia, it's also been used to treat multiple sclerosis patients. The proposed mechanism of action for the antiviral properties is that it interferes with the viral protein M2 ion channel, which is responsible for uncoding the capsid. But more interesting for movement disorders, it's believed to act on the presynaptic membrane and enhances the release and inhibition of and re-uptake of dopamine. It also up-regulates post-synaptic D2 receptors, it has anti-muscarinic properties and it's anti-glutaminergic due to the non-competitive antagonism of NMDA receptors. One other thing that I came across is that it's also been studied recently in patients that have traumatic brain injury or spinal cord injury as a neuro-regenitive drug. There have been several reported side effects as far as ocular side effects including transient vision loss, hallucinations, ocular gyric crisis, medriasis, and then some well-reported corneal side effects including subepithelial opacities, superficial punctate caretopathy, and then epithelial and stromal edema. One of the first case reports for amantadine and corneal edema appeared in cornea in 1990 by Blanchard. It was a 64-year-old woman who had been on long-term treatment with amantadine for influenza-like syndrome and she presented with diffuse corneal edema in both eyes. She stopped the drug, was prescribed basically Miro drops, and had complete resolution after 10 days with no recurrence. Now, there have been several other case studies that have looked at amantadine. This was maybe one of the largest epidemiological studies that was done. French and Margot looked at the Veterans Administration database and tried to correlate the finding of corneal edema, Fuchs dystrophy, those two diagnoses with patients that had been prescribed amantadine at any point. This wasn't really very revealing. They found that 0.27% of the patients that were diagnosed with corneal edema or Fuchs dystrophy had been on amantadine. Pretty low incidence of corneal edema with this medication, at least according to this study, I think subsequently several people had pointed out some potential flaws with the design, but still a lot of people think that this is a fairly rare occurrence. This is an interesting study because the authors in this paper actually looked at some of the histopathology of the patients that had what was believed to be amantadine induced endothelial toxicity. They had three patients that presented with various histories. One had a history of MS and had a previous corneal transplant. Another patient had MS and presented with a complaint of blurry vision. Then the third patient had been on amantadine with their history of bipolar disorder, which there's also been a couple of case reports. I forget to mention that amantadine used as a mood stabilizer as well. All these patients had significant corneal edema marked absence of gutata and otherwise unremarkable exams. In two of the three cases, the cessation of amantadine actually quickly resolved the corneal edema with no recurrence. The third patient actually had irreversible corneal edema and required a repeat corneal transplant. The corneal button that was obtained from that procedure was examined. Significant exam findings on that in the histopathology were pleomorphism and polymega thism of the endothelial cells. There was widespread denuding of the endothelial cell layer and actually retraction and lifting of the endothelium from the underlying decimates membrane. Another significant finding was the posterior collagenous layer was bared by the retraction of the endothelial processes. They actually provided a couple of nice electron micrographs that show that this is kind of that endothelial denuding and retraction of the endothelial processes on this slide. This shows the exposure of the posterior collagenous layer and the retraction of the endothelial cell processes. There have been a couple of papers that have looked at irreversible amantadine toxicity and this one is not actually one of them. This is actually a patient that was originally not recognized to have amantadine-induced corneal endothelial toxicity. She presented with a non-glutate corneal edema and underwent decimates stripping, automated endothelial care deplasty. This patient actually had recurrence and I was actually involved in the care of this patient as well. This patient had recurrence of corneal edema after the decimates stripping was performed and subsequently was found after a little bit more extensive look at her medical history that she was placed on amantadine. She was a patient that had, I believe it was schizophrenia and she had had tardive dyskinesia as a result of her anti-psychotic medications and that's the reason that she was placed on amantadine. It wasn't until the recurrence of corneal edema in the transplanted eye that this was discovered that amantadine was stopped. Unfortunately, the corneal edema did not improve. A repeat desec was performed and at that time vision did improve and corneal edema did resolve. So just to summarize, amantadine induced corneal endothelial toxicity, the mechanism is not completely elucidated. It's believed to have a direct toxic effect on endothelial cells. As I said, based on the VA report, it looks like the incidence is very low but a lot of authors that have written on this subject have questioned whether or not it's underreported. We don't know in those patients that have presented with non-gutate corneal endothelial dysfunction whether or not they just had transplants done and if the amantadine was stopped subsequently and the edema didn't recur whether or not this could have been missed this diagnosis. There's a little bit of argument among a couple of authors that I saw in some of the journals that said that in most cases this should be a reversible phenomenon but with cessation of the drug but there have been several cases reported of irreversible corneal endothelial edema that have required transplantation. And this patient, I don't have the endothelial cell counts, I couldn't find them, I have them somewhere in boxes in my apartment but I believe we got endothelial cell counts on this patient, my patient from the VA and it showed a reduced endothelial cell density in both eyes. So just as a note in the presence of non-gutate corneal endothelial edema it should prompt investigation into adverse effects from systemic medication and particularly amantadine. Any questions? This is actually something that I tried to set up at the VA before I left but the VA IRB is a little bit more extensive than I expected it to be. There's actually one report that I didn't mention here that kind of answers a little part of that at least I think and there have been a couple of pediatric case reports in children who were put on amantadine for influenza who developed diffuse corneal edema. We're talking successful measurement by pechimetry of a thousand microns and I think there are at least two cases that I found they responded very quickly to withdrawing the drug and it seems to be dose dependent. Most of the cases that I encountered report a very close temporal association within weeks of starting the drug on set of corneal edema. There have been a couple of case reports that show that it may take months and some patients for the corneal edema to present significantly. And that's what I tried to set up at the VA. Any other questions or comments?


Welcome to film my run. My name is Stephen Cousins. Today we're going to do something a little bit different something I very rarely if ever do and that's analyze the data in some of my runs Particularly what we're going to do is look at the difference between running on a treadmill and running outdoors I've got four marathons to look at two of them will run on a treadmill indoors two of them will run Outdoors a lot of people say that running on a treadmill is more difficult Some people say it's easier, but most people say it's more difficult to run indoors on a treadmill There are a variety of reasons for that one of them could be the indoor temperature lack of air flow Makes your heart rate higher for example perhaps Or a poor quality treadmill with a poor motor which makes you work harder or seemingly harder Your perceived exertion is harder on the treadmill to get the same speed or to cover the same map different Distance in the same time So lots of reasons why people think running on a treadmill is harder What I feel is that they should be the same so your treadmill running indoors needs to match your running outdoors in terms of the effort that you make over the same distance for the same time So if you run 10k in one hour Outdoors then you should do 10k in one hour indoors for the same level of effort So your heart rate or your power numbers or your perceived exertion should feel the same Indoors and outdoors. I know that's not the case for a lot of people, but it should be So I'm gonna have a look at three four marathons now So let's start with a marathon from March 2019. So over a year ago year and a half or so ago This was a treadmill marathon the Watopia run festival on Zwift And I did it in a time of three hours 29 minutes for 42.2 kilometers Now training peaks which is what I'm using here gives you what's called a TSS score That's a training stress score. It's basically how much stress your body was under or how hard you were having to work to achieve that result And also what training peaks does is it measures your fitness based on number of different Measures and an algorithm that judges how fit you are So my fitness level at this time March 2019 was 52 So just bear that number in mind 52 and a TSS score for this run of 217 Let's just look at the heart rate and the power and the speed So my speed for this run was 12.1 kilometers per hour My average heart rate was one five one and my power average was 240 so that's for an indoor run on in March 2019 now, let's look at an outdoor run one month later I run the Paris marathon and I run the Paris marathon in two hours 33 minutes But I've actually covered a little bit more ground almost a kilometer more in terms of distance covered So you can argue that the distance and time for these is the same one kilometer in about four four and a half minutes 329 and 333 for slightly more distance so we can argue they're the same I'm about the same level of fitness. Maybe not quite as fit 48 as opposed to 52 not a not a big difference really to be honest and I've had to work a little bit harder. So maybe because I'm not quite as fit I've had to work a little bit harder in Paris according to the TSS score. My body was under a bit more stress However, if we look at heart rate My heart rate is actually the average heart rate is lower so one five one average heart rate for the indoor run and One four seven average heart rate for the outdoor run But then if we look at power Average power is two three nine as opposed to average power in here to 40 Hardly any difference at all really So perhaps I was working a little bit harder indoors and maybe that was to do with Warm the heat of the indoor environment who knows But maybe maybe there's some other factor that we don't know about there are a lot of parameters that affect Why your heart rate might be high? But really there's I mean it's four beats per minute difference It is a little bit higher, but not massively higher. Is it it's not huge the maximum heart rate goes higher Certainly sprinting at the end here to try and get in probably in under three hours 30. Maybe I pushed my max heart rate there So those are those two runs then we come to this year Now this year. I'm a bit fitter So we look at June this year for an indoor run Now I'm much fitter than I was last year look 48 fitness 81 fitness But I'm a bit more tired so fatigue is 61 For the Paris Marathon, but fatigue is a bit greater. So I'm a bit more tired going into this run But I am also fitter going into this room Now if we look at heart rate one four six heart rate So actually the same the same heart rate For the same run In all three pretty much the same average heart rate Unfortunately, there are no power numbers for this run. I didn't obviously wear my stride foot pod For this run. So we're gonna have to work just off power speed average speed for this is 11.6 So slower than these two runs the Paris run and last year's treadmill marathon But I did achieve a slower time So and I've worked slightly harder as well two three nine as opposed to two twenty and two Seventeen TSS school, but it is based off heart rate. So that can sometimes affect things can be slightly different So if I am fitter than I was in the previous two marathons and my heart rate is on average About the same. Why is it that I wasn't able to achieve? The two thirty marathon time. Well, I think it's down to heat. We're running in June indoors in this shed here, which is affected by the heat It's very hot in here. There are some big windows even though I had the doors wide open and you can see here I was maintaining a high level of pace All the way I was managing to keep about 12 kilometers per hour all the way until about 35 kilometers in when I cracked I broke and My my pace basically plummets and my heart rate Comes down with that as well So I basically give up here and the last seven kilometers is dead slow Just grinding it home. So if the heat hadn't got to me Perhaps I would have managed to maintain That all the way to the end and perhaps if it had been outdoors that would have been the case who knows So let's go on to the marathon that I did 48 hours ago the virtual London Marathon my fitness is a lot greater 103 Because I've been putting a lot of miles in over the last few weeks You can see that my effort level two four one Is higher than last year's marathons to twenty and two seventeen in terms of power And you can see my average power is higher to five five average power compared to two forty to forty and two three nine on last year's indoor and outdoor marathons So what can we say about this? Well, I am fitter. I'm able to run faster For the same heart rate one four eight heart rate here one four six one four seven and 151 slightly higher there But for pretty much the same heart rate I'm actually able to run quicker because I am fitter over the same distance Power numbers are greater. I'm stronger And able to do that outdoor marathon in a quicker time Just for fun, I've changed the TSS score instead of reading from power. I've changed it to heart rate So this is the virtual London marathon from 48 hours ago TSS score of two three six for a three twenty-four marathon Indoors a TSS score of two three nine for June's Watopia run festival in three thirty seven so Slightly harder work to get the London marathon done than that one, but I did do it slower and it wasn't as fit and Two three nine for the Paris marathon. So again, we're pretty much two three six two three nine two three nine and two six four So my fitness level of 51 in March 2019 has made me work a lot harder to get a sub 330 marathon on the treadmill Then it does for me to get a pretty much comparable because remember I've gone a bit further in terms of distance I didn't have to work as hard to get the same marathon result outdoors For this about the same fitness And then we jump to the next indoor marathon I've had to work exactly the same the same level of effort has gone in to this marathon as To the previous marathon the Paris marathon, but I've got a slower time And we've attributed that to the heat of the day in June So I've worked just as hard Same heart rate But I've just collapsed with the heat at the end of the run And that's why we've got a much slower time or seven eight minutes difference in time Because of that last bit of work there, which wasn't quite as good So I think what I'm beginning to conclude From looking at this data is that perhaps it is a bit harder on the treadmill than it is Outdoors certainly for this these marathon examples It looks like I'm the same level of fitness last year pretty much for indoors and outdoors And the only thing that's making it different because the times are the same for the distances so 333 for 43 kilometers or 329 for 42 kilometers And it's the same time essentially and the only difference being That we seem to have worked a lot harder 264 an average heart rate 151 with a max heart rate of 176 I seem to have worked a lot harder for that time indoors Then I have for this time outdoors Over the same that you know, it's the same distance the same time And if we go to the two more recent marathons, it seems like I've worked harder here For a slower time Than my marathon a couple of days ago only slightly harder two three six HRTSS for a time of three 24 But I have worked a little bit harder and got 13 minutes slower time You can see my speed is slower, although my heart rate is similar And that's because we're indoors and I haven't been able to cope with the heat indoors And maybe if I'd been outdoors that wouldn't have happened this slow down in the last 7k wouldn't have happened So again, we're possibly arguing that it is a little bit more difficult to run a marathon on a treadmill Indoors than it is to run one outdoors Also, I think it's just worth noting that I run on the treadmill at zero percent So I'm not a lot of people will add one percent incline because they feel it makes it more realistic To outdoor running but on the evidence that we've got here that perhaps actually treadmill running is more difficult I'm running at zero percent So I don't want to make it even more difficult for myself By raising the incline by one percent It is just as hard if not harder to run on a treadmill indoors than it is to run outdoors Let me know in the comments what you think do you agree with me Do you think that I've made any errors here in my assessment of the power or heart rate? Measures so so that's it guys. Thanks very much for watching. I hope this hasn't been too tedious for you to get through There'll be another film my run epic marathon coming soon I currently editing the Buell water marathon so that should be out soon take care Thank you very much for watching and we'll see you again. Take care. Bye. Bye


Hi everybody, I have the pleasure to introduce you and Tonyo Terceiro Debbie and Ruby developer with the talk with the tutorial functional testing of deviant packages Everyone So we heard today about to talk about testing I'd like to get an idea of where you guys are coming from Here's our hand if you maintain on devian package Uh, if your package Now keep your hands up If your package has auto package that supports drop your hand Wow, then You are the right crowd for this talk Uh, automated tests are great and we need more of it Uh, I hope I won't find anyone to disagree with that Of course, it doesn't solve everything So lots of types of applications. You still need manual tests But if you have a safety net safety net with automated tests, you can Spend a lot of time handling other things other than doing all manual testing and then This is the topic are the topics we are going to cover here. So I'll talk about a little bit about the history of this Effort to increase the testing coverage of deviant Uh, we are just going to discuss the specs for Declaring the tests that you Want to have run for your package We are going to discuss the tools you can use to test locally And to reproduce exactly the the same setup as you have in the deviant continuous integration And then I'll give Several examples of packages that have tests and that use Several techniques to test their features So Auto package test is actually not was actually not invented yesterday So it was the first very first upload was made in 2006. So it's going to be 10 years next year And took quite a while to get some coverage in the archive and then in 2012 there was a Proposed depth What was it was that page So it described Automatic as installed package testing So it's in depth others More or less. So there you have a pointer to the actual location of the spec which today is in the In the auto package test which repository And then the the idea behind that the depiate specification is to describe a standard interface So that package can declare their tests Uh In a way that can be tested against a real system So the idea of auto package test in depth h is not to test Not it's not to do build build time tests is to test the packages. It's it's installed. So It is possible to run upstream test suites, but usually you want to test the actual package that Is With its binds in users slash being Which which is library codes if it's python or ruby or whatever installing this in the right place So you don't have to mess with Library load paths and all that kind of stuff you want you want to test the actual thing That's users are going to use And then there was this idea floating around that someone somewhere Should run a auto package test on the whole archive And then in january 2014 so year and a half From now, I decided to It was a little earlier than that, but on january 2014 I had something working Which I called levin continuous integration. It was a very ugly The the htma htma was very ugly. It was very Very basic but then One year and a half Later we had two GSOC students last year. So one of them worked with me in the Enhancing the ui and make it ready for having For testing multiple suites and architectures. So today we have support for that We don't do it yet, but the ui is ready for doing it and then also Got a lot of my hacking time And then today we have ci debiannette Which is It's quite nice. It shows There in the left the latest results. I took this screenshot yesterday And then you can subscribe to the feed Which is actually very quiet if you Take it into consideration the number of packages in the archive So a few entries a day because It only notifies you when a package changes state So if you use it Use it to pass and now fails then you get an entry in the in the feed But if it's Always failing or always always passing you don't get Noise saying that pass pass pass or fail fail fail So it's quite nice to watch this feed to Get a sense of what's going on so when When I break something ruby then I get lots of ruby-full fail and so on And now we also have today The same thing for ubuntu So there is this ubuntu actually they run AMD 64 and i386 and then A few of their suites Really Yes, yes, they Yeah, the question was they really use this as a gateway to the archive So yeah, they they do use that data from there to Drive their britney instance And then this is the The numbers of Number of packages that we have Being tested by by dev NCI. So in the very beginning was 190 I think so it was very Very small number and today we are at 4,000 and 300 something So you notice that we cheated a little bit there in in may 2015 when we uh We actually white listed a bunch of ruby and perl packages Which have all of them have a very similar structure and you can run The tests in them with Common code so we you don't have to we don't have to update every single package.  So we have a great spike there On packages tested and you you see that most of them pass their tests So it seems that since the very beginning there was a small percentage that uh always failed but Uh, we increased a lot more of that of Passing packages than failing ones. So Is that so if if if there's anyone here that works in a packaging team for programming language packages and They probably can be also white listed to run all the tests with a single command I I would imagine that python ones at least could you could have a A reasonable way of running tests on all of them I have no idea about other language So we have today a coverage of 18.5 Percent of the source packages in the archive. So the the test suites are source source package base Although they are supposed to test the installed binaries Uh, and then we started with less than one percent in general. So we It was reasonable Increased given the the amount of effort that takes to I mean, it's not so difficult as I hope to show you guys here, but It's not trivial And this is how your package page will look like for a given architecture. So you have The whole history there of test runs with logs and everything You have the artifact which are more details about the the environment where the test was run And then the the logs also have lots of information Oh, ci data today is also available on the devm maintainer dashboard page on udd It's available on the dpo. So if you want to see the ci status for your package, you can just go to the dpo page and also the package trackers list the ci results I'm pretty sure the new one does I don't remember about the one So in the future, we uh, we are almost ready to migrate to a distributed setup But we'll be able to put a lot of cpu power into running tests. So it will reduce the The lag between an upload to unstable and actually getting the run the test run And uh, we also This will also Enable us to test run for testing and maybe other architectures So today we just run amd64 Now talking about the specs again dep dot debon dot net your friend. There's a link there So there are two basic elements of of the specification one Is in the source tensor of your debon control file declaring that the package has a test switch So you do that but adding a test suite field with Auto package test value you can use other values. So the The program ruby package that I mentioned they use auto package test dash pkg dash language That's what uh Helps us identify how exactly to run those tests But if you are writing your tests specifically for a package you want, uh This This header here. So use it to be xs dash test suite, but you don't need that today. You can just use the suite The package ingest you already supposed that so It should be good by now And then you you have an extra control file in debon tests control in which you list your tests So I That's what the head the rest of the spec is about So you you can just declare tests So in this case you are declaring three three tests and each one has to be an executable inside debon tests So it's just like that. So it can be anything you can You can build c binaries with those names if you want you can but the most common case is just writing shell scripts or In another interpreted language And then it has to be executable and if Exit with zero the test pass and if exit with no zero the test fails It's simple like that You if you if you have uh A simple way of running your tests you don't have to write a one line shell script So you can just declare it directly so you use test command And then whatever you want Then you can also have extra fields you can use depends So in this case that symbol means All the binaries of the source package So when the test bed is prepared All those binaries will be installed plus the ones you declare. So in this case, I want uh, I want to have all my binaries installed and also some tests too. I'm going to use to run the tests Uh, you can also Have multiple tests with different, uh, characteristics. So you just use multiple Stances in the control file. So in this case, I have One test program that needs a given test too, and then I have some smoke test That doesn't need anything besides the binders. So if you don't say anything The depends is assumed to be all the binaries. So the at depends Column at is the default. You don't if you just need your binary. You don't need to say anything You can also need your build dependencies to run tests So in this case in the case where you want to run an upstream test switch, which uses x unit frameworks or something You can just Use that and use your by your build dependencies And then you can space five restrictions on the tests. So it's uh, additional requirements I'm gonna give you a sample here. You can say that this test breaks the test bed I don't know something you do there in the test script What's the season in interest state that's not going to work when you run a second test on that on there? So For instance, if in this case you have you only have one test But if you have two and you have breaks test bed there You you would instantiate a new test bed for each test. Otherwise, you just reuse the same one I was wondering if this means that the test bed has to break or that there's a big chance that the test bed, uh, breaks Uh, it means it might mean both So the point is when this test is done You don't need to reuse that you don't want to reuse the test bed And also if you are running these tests against your real system, you want we want to skip it So if you if you run anything outside of a virtual machine or a container The visualization driver will skip that test So if you're running on your main system as root, especially Then that's going to this test going to be skip it. So you don't break your working machine One more question on that topic How are those Machines reused so Is there a chance that there's clutter from some previous one in the one for another package later? No No, so in I will get there, but auto package test supports virtualization backends So you can choose if you want to run that test in a kvm machine or in a lxc container Or in a sch root session so usually unless If you say three tests with the same dependence is everything equal they will all run the same test bed But if if you if you use different Stances then each one will get its test bed and if you Declare breaks test bed and if you reset after each test, so you always get a fresh environment You you can also say that your test needs root So since the test scripts are just anything You can you can even test preceding so you you can Declare declare a dependency line saying that to not stall your binaries and then you can call Apps from the test to install those binaries and you can proceed before and test that they proceed in work Works so you you can use test as root if you need By default this is a common Gotcha, you if your test outputs anything on the standard error Swim then it's considered a failure So if your tests Uh Sends anything to File the script or tool you want to declare a low standard error Or you want to redirect standard error somehow somehow Then you you can specify Which level of of isolation you want from your host system? So if you declare isolation container Then only things as isolated as a container or more We'll be able to run those tests So if you if you want to mess with system service like stopping services restarting services and stuff like that You don't want that to run in a ch root Because that's going to cause problems so you can use isolation container. It can also Use isolation machine To say that I can this can only be run on a virtual machine or something even more isolated So you want to do that like if you want to load kernel modules if you want to I mean test things you usually things that are related to to the kernel You can also say that you need your recommends Which will not be installed by default, but you can say that you need it Now talking about tools The very simple tool that probably everyone has already installed is sag t which is part of dev scripts It will assume that it's being run from the root of the source package and you just run the test there So it has a few limitations. So if if it finds any restriction that it doesn't Know yet it will skip your tests But it's useful as a first step. So You if you maintain packages you already have it installed you just Create the test definition and run s ad t from your source package and it works And then you you have the The full thing which is ad t dash run from auto package test It's a little more complicated So you you first say the input options For ad t run. So you can run the test from the current directory You can run from dsc You can you can run from changed file. You can pass additional binary dabs You can do lots of stuff. So It's the main pages are useful read and then you use yes three dashes And then the virtualization arguments which says Which kind of virtual environment are going to use to run those tests? So there that you can use So the most common case which is equivalent to what s ad t does is Dot slash and the so run the test from the current source package And on the new virtualization Driver, so that's going to run only a real machine It's not going to be it's not going to run as root, but it will run on your real system. So So in my opinion, this should be the default. So if you just say ad t run, it should should just do that I reported a bug Briefly before flying here I hope We are going to sort that out And then you have And you can have the more complicated use case so you can run This is how more or less how how more or less how the psi does things so it It uses s s ch root and then you say the name of your s ch root doesn't have to be this it can be Anything you want and then in this case, I'm passing a changes file. So it will both run Read the test definition from the source package there and then it will use the binaries there to test So if if you want to test an upload before Build before actually doing the upload you can just pass the changes file it to Use your binaries from there Over the ones in the in the archive You can also use this do the same thing with lxc And q emu and ssh. So the ssh driver Assumes that you have some mechanism to instantiate virtual machines on the cloud or in some Somewhere that's going to be magically creating virtual machines for you and then to ssh into them and run the test there So there are requests for p builder and docker support. So if you care about those things you Can write help write the drivers for them Now, let's look at some examples and cheap tips and tricks So I will show some examples here. I forgot to open the terminal in advance. So let's Think white backgrounds better to see right So this is spin point, which is The presentation program I'm using here. So it has a very simple test definition Can anybody see that is that the font is big enough? Okay So it has a simple test script called smoke test and it uses its own binaries file and ssh unit 2 This is my first chip ssh unit Is very useful. So it's a testing framework like you find in any language, but for shell scripts It's very useful. You can do everything that you do with other tools And then the actual script Uh, what I used to overcome that uh, standard error restrictions just I always just redirect sunday or two Uh standard output and I don't need to care about uh that restriction So you have here the test functions like you would do in any other language So pinpoint since pinpoint is a graphical application. It's not very practical yet to actual test the actual To test the actual user interface. So I'm here. I'm testing the pdf output feature So I just create a pdf and then test that it's a valid pdf. So here I'm testing the output of file dash dash my type dash dash brief On the pdf actually returns application slash pdf. So I'm sure that if this test pass the pdf output function, an it is not completely broken And here is the test below is just a a test for a corner case When you have an empty background definition, it's used to crash And then uh to use test unit you just source it from the bottom of your script So you create the Functions which begin with test Something and then you just source s h unit at the end and then it runs. So if You can run with s ad t here s ad t hides the output from you. It just keeps that spinning Thingy there and just gives you the result Or if you want to run with auto package test, it will give you the full output of everything It does a few a few checks in the beginning. So it's checking if that I have the dependencies so since since I'm running the test on my host machine It will not be able to install Things because it's not being run as root. So it it checks that the dependencies are actually available So if you're running on the in your local machine You have to make sure that you actually have the binaries the corresponding binaries installed Otherwise your tests are going to run against the binaries from the archive So it's pasted everything is nice so Tip one s h unit two very nice if you Want to write a test with shell scripts Now rails tip two I already said so just head directing the standard error And then there is So the test definition is also very simple just a single script called new app So this script is going to create a new rails application and do some test some basic rails functionality like Adding a new table and in the corresponding model object and then running the test on that To make sure that everything works And in this case, I'm I'm saying a low standard error It needs recommends because by the full rails applications need actually need the The the full application that rails generate the template one actually needs the recommended packages And then the script so the tip here is how you Interface with auto package test. So you you always have an ad ttmp variable Which is a temporary directory that's private to that single test. So you can use that to To change directories into for instance, if you want to make sure that you are running your test in an empty directory Or you can store things there and And so on so I just create a new application Change directory into it Run some commands And if everything Passes then I'm fine. So in this case it already helped me a lot with rails has a has a huge Dependency chain. So if something in the middle breaks this script already It's very simple just creating very basic applications running the test Already helped me to detect problems with the dependency chain so Another thing I want to show you is this handling here. So if you Check for the existing existence of ad ttmp you can know whether you are running inside auto package test or not So in this case if I don't have one already I create one So I can just Run this test script quickly from inside the source package If you run everything for me And I don't need anything else So this is useful if you have uh, I have 10 minutes okay, it's useful if you have uh Several test scripts and you run you just want to want to run one of them So if you handle these things you can just do it uh red mine Is another package That has an interesting setup The control file is actually a little more complicated ah So I test uh all the cases for all the uh the supported uh databases So I have one test for In this case I use test command because I use a single test script with arguments So I in this in this first in the first tensor there, I'm using s sqlite 3 and I'm using Apache 2 Passenger as uh as the The web application connection How how the the web server is going to interact with the actual application And then I have one for postgres one for my sequel And then another one we ask you a lot again, but using a different apache integration module So I can test all these cases so uh, this means that uh Before I upload a red mine. I need to I need to wait a little bit for these things to run So each one is going to install a full red mine salation from scratch on a new backend and Last time I look at took like seven minutes to run because it's everything from the beginning I won't bother with the details of the script but the point that you can also Past parameters so you can have the same script that does things slightly different with different backends and so uh, I need to show you the script because So you you can restart services This is So I hear I'm here changing the apache configuration and then just reloading apache And then checking that it This is very basic, but also helps a lot. It just gets the The address where red mine is supposed to be and you'll test that the hml returns something sensible So if that was a 404 a 403 or something else or uh Uh 500 error then the test would fail Random ruby package. So if you want to know how your Package that's handled by common you will notice that There is no control file here But the source package declares auto package test package ruby. So The ci environment knows how to handle that and How will it do it it will do it using the output of autodep page? Which is a helper tool that auto package test will use It it's the tool that you will you want to patch if you want to add support for new packages. So In a ruby package it will create automatically a control file for that and auto package that will consume that file Indeed run the test like that. So it will just Do its thing without having to explicitly declare It failed nice I love Live demos All right, I'm running the test against the old binary. That's what happens when you have that so But I do run the test before uploading on my clean environment Just just for you to know So autodep page has support for four types of packages. So there is ruby pearl node js and Kernel modules that use dkms So if you want to add support for new packages, you can you can just look for the commits that added these two Because they are very simple. You just You don't have to touch anything that already exists. You just have to create two new files one to detect whether the The package is of that type and then another to create the actual output for the control file You you can also run look at debc i itself In this case debc i runs its own build time test suite against the installed version so Just call this script here Just a little uh It just makes sure that you are running against Not against the build tree the the source tree because Lots of ruby packages have that so if you are running from inside the ruby tree, it will modify the load path The library load path to load libraries from the source package instead of loading the ones from the system Which is what you want when you are building Otherwise, it's going to run its unit test against the old version that you might have installed but when you're using Autopac says you don't want that so in this case I just Copy every test to a temporary directory Change there and run the test from there. So i'm sure that's not going to load the code in the source tree This test takes a little bit. So i'm not going to run them But then this is one way of running upstream test suites when you If you If that's possible at all, I know lots of test suites Especially for stuff reading c and c plus plus actually require building the whole thing. So That might get complicated, but at least for Interpreted language is usually doable I'm going to skip this one and let's leave some time for questions I'm also going to skip this one this chip is very helpful So if you want to inspect the environment after the test finishes Especially if you're running against a veto machine or a container or a ch route you can pass Dash dash shell which will always run a shell after the test or you can pass dash dash shell dash fail Which will only run a test if the test fail It will only run a shell if the test fails. So you can inspect What's going on there? So, uh, please join the movement to increase even more the test coverage in debian So you can add tests to your package. You can now you can add generic tests for a packaging chain to auto depth 8. It's very easy. You can Talk to me. I'm here until the end of depth conf We can sit together and do it very quickly And you can also talk to me if you want to help improve in maintaining ci dot debian dot net Uh a few acknowledgements. So the injection for curating Auto package test marching pit currently maintaining It it he's doing a very good job with uh several improvements and very responsive Brandon for child and lucas canashiro where the gsoc students from last year. So Brandon did a lot of the ui work and uh lucas did a lot of Since sent a lot of bug repulsive patches for packages that have uh broken tests mostly That's it. I think I remember seeing Just a check that you actually built a package before you run the test. Is that logical needed or Why so If you are running in uh, so suppose you are preparing a new upload And you want to run the test. So you want to run the test against the The bindings you just built, right? So you either Have to install them if you are running on your current host system Or you you need to pass the changes file for instance to get those installed into the ch root container or what have you but What I mean is in the ci? Ah, no, right. No, we don't build the package we test whatever is in the archive So auto package test can use uh the source package as input. So if actually if you run, uh If you do this it to just it to actually download the The source package from the archive in run the test from there So that's what we do in ci. We just we don't build package and uh That that ways is much faster and you are uh testing exactly what you are shipping to users I wanted to know if the test switch field can have multiple values I mean there is this auto package test dash pkg pearl and can we have that Together with auto package test alone for a custom test of Auto functionality I I'm not sure I have to check so what, um What auto package dash does if it if it doesn't find debent test control Then it calls autodep age So if you do have debent test control you want to include what Whatever autodep age would generate for your package there as well You can also disable using the auto tests So if you there's a parameter in an adg run that you say you can say I I don't want to The resource there will be I'm not sure yet, but uh, you can say I don't want This crap that autodep age generates Have you ever tested? dpkg config scripts like config or pre-rm Post install something like that I didn't but it's possible. So what happens is if you If you if you use the default dependency your package will be installed before Your tests have a chance to run. So if you want to do that you can Put like depends dpkg the package So the depends will be satisfied just from the beginning and then from your test you can say I need to test to run as root and then you can there Make preceding or everything else and then Co-app to install your package and then this way you can have code before and after the installation And then you can test whatever you want So is there any existing back practice to integrate this automated testing into an s-build environment? Sorry say that again the last part in the into a whatever integrates into an s-build environment In an s-build you mean automatically running that after s-build for example, I'm just asking of course It's somehow possible, but is there any existing bad practice of Yeah, I'm not sure what I do is that I always run tests against the changes file I just built After I built of the build, but I have a wrapper script that does that for me Okay So I'm just asking because we are actually building on an existing s-build environment and I'm just thinking well Well, what do you mean reusing the ch root that s-build just use it? Maybe could be an approach I think there is a wish list bug against s-build to call Autopackage set after the build, but I'm not sure Okay, so it's still an unresolved question without the speaker. Yeah, I I would say it's doable. I think yes, okay So as the page is out there should be a problem Is there a restriction available to have an x11 server actually a real one? actually running Outside of the test and especially can the ci machines provide this at least for for the tests that require it I don't think so. I I know ubuntu has They do run tests on bare metal. So they do have dedicated Real harder to run like Video drivers testing that kind of stuff, but I don't know exactly how that works And No, what there there is what's there in the the bci is that you can You can have tags associated with a test run and then you can use tags to identify a Back end where you want to run that but that's We don't have a good solution. You can use X vfb dash run which creates a kind of a fake Sometimes not sufficient. Yeah, I know Yeah, I don't have a solution for that yet Is there a possibility to to run tests against well kernels or Tools providing virtual machines within nested virtual machines. So let's say we run one The auto test is run in a virtual machine and inside something is tested which Runs or tests virtual machines Yeah, I don't I don't see why not As far as kvm and kvm. It's actually supported on The host you are running I don't see why not. Is it possible to run these tests before A package gets in the archive Uh That's a good question it would be It would be possible. I think what ubuntu does it they have Proposed suite and then Only after a few checks it goes into the Into the suite that they they use that use the actual development version Get so it's possible I mean We have more questions Okay, so thanks a lot. Antonio. Thank you