Hey guys, good afternoon. Thank you for coming for this session. My name is Sameer. I'm the founder and CEO of a company called R2. What we're trying to do is, you know, use Android and Cloud for the benefit of the masses. So, you know, many companies that work directly with the poor, whether they're trying to provide financial services or healthcare, they operate through field agents and field agents who have a technology barrier, language barrier. So, we're trying to use intuitive touchscreens built on Android, you know, and sort of connect that through the cloud and make things happen. That's a little bit of our house. Now, before I start, just to get a sense of how many of you sort of heard of Jenkins or continuous integration first? How many of you sort of all work with continuous integration? Okay, so that's a large number of people. And so you would have obviously heard of Jenkins, right? And there was a session by Costa earlier, which sort of introduced Jenkins. So, you know, we were also very excited about continuous integrations, right? It suddenly means, okay, now you can build as a process and things like that. Before that, you had a very ad hoc process, right? Everyone would suddenly say, okay, I'm committing changes, wait, wait, don't push stuff to the cloud. I'm going to do this. And then when we got Jenkins, which is this continuous integration system, then the problems suddenly changed. Then it became about, okay, who's going to run it? You know, who's going to sit with it and sort of wait for it to finish and do the next set of steps, right? Not everything sort of takes place only at Jenkins. So I'm going to talk about this new thing called Hooport, which can try and simplify this entire interface, right? And make things a lot more fun for your small team.  Okay, so I'm going to show you how you can make such images also. This is a fun part of Hooport, using Hooport, right? I've also hosted this presentation on this link. So if you guys have laptops and if you want to look at some of the scripts in better detail, you can look at, can I even go to this link? So let's start with the problem, right? So as I'm mentioning, this is the kind of challenge that you face in the small team. Suddenly the person who's designated, in our case, we're just four employees, right? And just because one of us knows a little more about than the others or has some idea of what the others are doing, suddenly it becomes my responsibility to be the build guy, right? And then suddenly I become the bottleneck because now people are committing and they start depending on you, right? And typically you have two sets of processes. One is you take the stuff to your build server and that gives you an output. Typically you store that on S3 or some sort of storage and then you get your cloud to pick up that file and sort of update this, you know, your cloud, right? So there was a nice session on Puppet before this earlier in the morning, right? Now, so this is what we face today at work. Then we came across this interesting deck. It talks about how GitHub uses GitHub to build GitHub. So it's really interesting. You should definitely go through these slides. I think there are sort of 40 plus slides or more than that. Is this clear? Not very clear, okay? So what he talks about is that they work asynchronously. Now, this is a team that's about 52 employees. They spread all across the world and different people wake up at different time and do work, right? And so for them, they have to work asynchronously, right? At a start up, you would like to have this flexibility.  So what they came up with is a simple model. You have the master branch, which is always, which will always hold your deployable code, right? So whenever you suddenly need to deploy, you will take code from your master branch and everyone else will create little, little branches from this. Sameer will work here and and others will work on other branches and everyone can push and then they can deploy, right? So the idea was to make it more democratic, right? Now, anyone can sort of build, anyone can push. Of course, you need to have permissions and place. But the idea was to build a process that allows everyone to push and everyone to deploy. That's really hard. And that's where Hoopod comes in and does a good job. So what does Hoopod? I'll just sort of step to my screen. So this is my G Talk terminal. And you can see over there, it's not really on screen. Sorry, the resolution changed. That's why I connected the project. Is this visible? Okay, I'll use this thing then. This is visible? Okay, so this is nothing. This is the iChat client. It's connected to my Gmail. I'm going to say hi to him. Let me try and increase the point, right? So I say hi to him. He sort of responds by saying hi to your service. And I'm going to ask him how is my cloud doing? It's a simple question. So he's saying cloud doing fine. And I'm going to thank him for his service. So he's going to say you're welcome. I'm going to say hi to him politely. It's a nice speaking game. Right? So it's just a simple thing. It's a program that's running at the back that's simply sort of responding to your commands. But this becomes a very fun way to interact with the system. Right? So this is you what that it is a bunch of things. It's written on load. It's written with, you know, coffee script. And you can make so you can make a script to do anything. Right? So you can simply make commands that can respond using rejects. It basically understands okay, you said hi by thanks and things like that. And then it can run a command. Right? So what I'm going to do is sort of go back to the presentation. Just for completeness is sake, I'm going to talk about, okay, how can you install this? So instructions are fairly straightforward. You need node, you need the package manager coffee script. And then you can install you bought. Now what we've done differently is we've normally you bought is used with campfire or hipchart or one of those things, right? And those are paid services. Whereas we spend a lot of time on G talk. So we thought we'll connect it to G talk. And there's an adapter available for G talk. So we just straightforward use that. And running it is as simple you provide the environment variables, which are your, you know, G talk user name, password and sort of what what needs to run on and you can start this. Then so if you want to know what all it's capable of, you can simply type help and you'll get a list of commands that it can do. Right? So I'm going to go go into this a little later. We've connected this to talk to our Jenkins, right? So I'm going to say Jenkins list. So it's showing me a list of you know, jobs that we have running. And it's saying, okay, this one has passed this one has failed. And you know, these are all passed. Can you see this text is okay? So I'm just going to say Jenkins build. I'm just going to give it this command. And it actually starts a build job in the background. Right? So if you come here, there's this I'm just going to say and you can see that a job has already started over here. Right? And and I can go and I can see so I can see the log if I would like to and then it's doing its work. It's downloading from get in and doing other things. Right? And at the end of this, it you can see it sort of places it on S3 for us. Okay, if you see the last set of lines, it talks about S3 repository and things like that. When our servers, when you want to sort of refresh the code on our servers, they'll pick it up from that repository. Right? Now this is how your regular build process would work. Right? Where one person is sort of sitting and saying, okay, on the master branch, let me go build it. Let me deploy it on S3 and then run a script to run it on to the cloud. Right? But what about sort of your interns or other people who want to simply run tests to see whether they code worked or not. Right? There's something called as parametrize build in Jenkins. That is, you can provide parameters at runtime that can change the way it behaves. Right? So over here, you can see I have this simple project to configure. And it takes a simple parameter called branch. And so you need to supply what branch do you want it to build? Okay, if you look at our GitHub, so our GitHub repository looks like this, we have about three other branches apart from master. So there are three other people who are working on this thing, and they're working. And I can just make this a little bit. Okay, so, so these are sort of each point is a commit, right? This is how GitHub works. This is a beautiful way to define things. So now what I can do is if I'm interested in this building my branch and see how it looks, I can say Jenkins, you know, build months equal to some me. Okay, and I can see sent me a link, right? And you can see that it's, it's just waiting, there's some amount of quiet time it waits until you want to pull it back to view. So again, what this does is it can actually, you know, it's picking up from the same branch, the code, and it's going to run. And at the end of it is going to run tests, and it's going to show you the results of those tests. These tests will fail because I've sort of messed up with my branch. But this is how rest of your team, you know, they may not be necessarily responsible for the end product, right? I mean, the end sort of build process, but they can quickly run that test, and they can understand what's happening, right? And they can get quick feedback, and they can do it in sort of a sanitized environment, like, you know, your build environment, and they can get their own response. This is making sense. It seems something like you guys will use. So to enable this, it was fairly simple. You know, we just simply used we connected it to Jenkins as a script for Jenkins. Okay, so you know, the way GitHub, so GitHub is open source, the code for who bought, right? And they described it as the most, you know, a project that increases their productivity, and also project that decreases their productivity, it depends on how you use it, right?   Say, there are various money scripts like this, mustache me and your name. Hopefully, you should see an image of me looking like mongrel pundit.  I don't remember seeing a skyper doctor. But you could definitely Jenkins itself can directly work with like, as long as you can communicate with it. I mean, if your ports are not blocked, and it's still accessible, it should look shouldn't be a problem. So the way this is working is you're using, yeah, sorry, this is how I look like mongrel pundit. But the way it works is so we will go a little deeper into that. And I can show you how you can write scripts, right? But what we're what we've done is we are hosting Jenkins as a web app. Okay, and on that same instance, we're also storing who bought. So who bought is listening to a particular socket, right? In the previous slide, you saw port 555 is listening on to that port. It's interacting with Jenkins using HTTP protocol. Okay, so they don't need to be on the same machines. Right? There is there are other ways to interact with Jenkins, but we're using a simple HTTP. So a bunch of other things that you can do with this, or you can have a lot of other funny scripts. Well, you have to be careful, some of them are quite inappropriate, not suitable for work.  And then there are some funny codes. Yeah, definitely. So what you can do is you can use that you can write your own script, right? So what I'll show you next is how do you write your own script? Okay, so how many of you sort of work with coffee script before? Okay, very few people, but how many of you work with JavaScript? A lot more people. So sort of coffee script is like another level, a higher level version of JavaScript, you know, does away with curly braces and things like that. It looks a lot like a functional programming language, but at the end of the day, it's JavaScript. So go to this link, just to sort of give you a sense of the variety of love. So this is a, these are all the scripts that are out there that, you know, that can work with who bought, right? And so most of them are sort of funny scripts, but there are a few which are like productivity related.  I can't see anything right now. So how do you write a script for who bought, right? So you saw my first script, that cloud script, right? It was actually a sham because all it does is it searches for the word cloud. And then responds that cloud is doing fine, right? It's fairly simple. It's just three lines of port. This is what's called a ton dog. That is, this is what's displayed when you do help, you know, when I type the help message, it will assure me that, okay, if you type cloud, you'll get a reassuring message saying, you know, cloud was working fine. This is sort of how you export a particular function. This is simple, you know, regex that you're doing, you're just checking the message, you know, whether cloud was mentioned in it or not, and it's case incident insensitive. And then you just send out a simple message, right? Cloud is doing fine. Now this is a fairly simple script to do, right? I mean, it's very as functionality. But this also shows you what all is possible, right? Moment you've done your regex expression over here, now you can do anything else you want. You can reach out to other systems, other services, web services, or you can even run sort of shell scripts, right? So the next thing that I'm going to show you is, you know, an interesting thing that I just wrote today. If I want to get a list of all our servers, right? So we use the right scale, right scale to manage your Amazon infrastructure. 11 servers up and running at the moment. And I'm just getting sort of, you know, a simple nickname for the server and a status that it's operational. And if I click on the link, right, so I click on this link, I can actually go to go there and see my server and see sort of graphs associated with it, right? So this is, I mean, this is the right scale. This is not anything. Now you can sort of see graphs. And the next step for us would be to sort of get these graphs directly into our G-TOP, right? So you directly say, okay, fetch me CPU levels for this particular server, right? Now how do we do this? Right? So what I did in this case was, so this is a regular expression, RS space list that I'm searching for. As soon as I match that, I check for environmental variables that have been set, I use the name, password and account number, and then I run a script. Okay, so the script that is running on the server, right? And the result of the script, I simply put that onto the message and I send it out, right? So those of us who are sort of familiar and comfortable with shell scripting can easily use this. Now what is the side? What does the script look like? So look at the core script, okay? So it's a very simple script. It's taking, you know, the authentication, you know, username, password, and then it gets an authentication token from that service, and then it goes and finds out, okay, how the server is doing, right? I get an XML, you know, it's the response, right? And the response looks, okay, I'll show you the response later with that sort of immaterial. And then I run a Perl script to parse that response, right? And what that looks like is, again, fairly simple. It's just simple XML parsing, right? So those of us who are comfortable with sort of command line and Perl and things like that can set up real complex jobs, right?  So it becomes fairly interesting and sort of the rest of your team can do. Now, coming back to your question about can you have permissions on this? So your permissions would, again, lie on this, right? You can get the user name, you can match against the user name and say, okay, this person is allowed. Sorry? Yeah, so, so Hubot has has only like, you know, two properties to it, right? So it has its own username password from which it signs into your GTO. And the second is it can whitelist certain user names or it can whitelist certain domain names, right? I don't want you guys to suddenly add this guy and then start running jobs on our system, right? So, so we whitelisted and said, only if you have the rest of the stuff, you can sort of build on your role over here. The best way to start on building scripts is to look at someone else's script, right? Just sort of get a sense of how they're structured.  Okay, so you search for a term and space tweet and they'll find you all the tweets that reference that term. So like if today we've been doing root com, right, so we could do a root com tweet, sending me, giving me one person's, yeah, so it works. So, yeah, these are a certain set of links and resources that are placed over here. You guys can look at them. Apart from that, do you have any questions? Is this sort of, what's this useful? Sorry? How different is it from your IHC? Yeah, so I'm not sure about how you extend IRC capabilities. Over here it's much easier to extend the capabilities of this bot. So, if you guys work on Hiroku or, yeah, if you guys host your servers on Hiroku, there is already a ready-made infrastructure for you called Janki. Okay, so it's built on Jenkins by the GitHub guys and they have a nicely packaged thing. Now, I'm pretty sure the same thing would have been possible in IRC.  But, yeah, so a lot of people use this typically in sort of an IRC mode where there's a group chat, right? So where you have sort of group spaces or a group chat room or things like that, right? So, the campfire allows you that, hipchat allows you that. GTalk sort of is more individual. So if I add someone else, so this is the downside of this actually, if I add someone else to this conversation, who doesn't understand what the messages are being directed at him or not. So you have to sort of send, you know, based on his name, you'll have to send a message like saying R2, whatever my command is, right, for it to understand. If there are multiple people around, if it's the only one it can sort of understand your commands directly. Have you guys worked with something similar or what are your sort of pain points when it comes to build processes and what are the optimizations you guys have done because we would be definitely interested in understanding how can we improve ours. Come online then you can directly sort of interact with Amazon API, is it? And Jenkins API. So the idea is to sort of simplify the interface, you know, to make it sort of cuter and easy, may not be really good at it. No, no, you can send images and stuff, you can send, say it has to be text-based data, right, that it can it can be made to do a lot of other things, right, you can call API as a other system, you can, you know, so there's a thing that, you know, you can, someone has sort of done, modified one of the scripts to play music in their system, in their office, right. So someone can say, okay, song, and sort of go and play that song. I mean, there's no limit to what you can do with this. Not yet. But you could do that. So I mean, at the end of the day, a new job creation is a post command, right. So you could sort of write syntax for this. Same. Send anything. No, I like this more because sort of we have only these servers, you know, so our model is we have these servers, you're constantly updating code, and those servers need to be updated. So the build jobs are made the same. Yeah, so you can also Jenkins has the ability to sort of sense change on GitHub.  When you commit code Jenkins, GitHub sort of sends a ping to Jenkins and says, okay, code has been updated on this branch in this repository. And if Jenkins has been configured to pick, you know, just automatically build anything and everything, then you can do that.  Yeah, that's what I was saying. So yeah, you can go and manage Jenkins and you can add plugins to this. Let's open up. So you can add plugins, typically, but you could use it with any other. But, but the important part is that system needs to have books that it can expose to Jenkins, right? Otherwise, Jenkins will never know when something's happened on that system. Right. So what would you do? You would put it in a cron or what you would say, yeah, that makes sense. You can say poll every five minutes or an hour or so. But GitHub has hooks that it can expose to Jenkins and say, okay, you know, when, as soon as you commit, I'll sort of let you know that a commit is taking place. Let's say, right? So do you mean even on successful? We may have successful. Yes, yeah. So it has a setting where it says, you know, notify the person who broke the build. I can show you. Email notification this last part over here. It says send separate emails to information support. So I mean, that definitely depends on your sort of project. In our case, these are sort of the things that need to be executed. So I'm pre-compiling. I'm removing the pre-compiled photo. This is sort of just a test. But at the end of it, you see an auto test and it stops there. If you can see this. If I wanted to do a bit and also decry. So that Jenkins will stop automatically on its own. Yes, so what we do is, what would I do a lot is decry. So we will be doing the shell and use badminton. But I think it will be most built over the catastrophic. So I think it's quite good. We basically then use scripts to then send a bunch of commands to where we're going to deploy to from the after it comes with detected using badminton. So we then get here. So like if you see over here, these are all the commands that you need to sort of put on to S3. Right. But if it fails anywhere along the line, it'll stop really in shell, but you can add build steps and you can say, okay, what needs to happen? Post your build and things like that. This is all that Jenkins has. Is there anything else? Okay. Thank you so much. Thank you for coming. We finished early. Good.


Soldiers of the press night patrol I'm Clinton B. Conger United Press here my credential What's your business here, Mr. Conger? I don't know Admiralty informed me this afternoon that if I came down here tonight, then boys would see do it that I had an interesting time I was told to say strawberry control. Yes, sir. I think you'd have an interesting time. All right Would you mind signing this please your United Press staff correspondent Clinton B. Conger signed a waiver releasing the British government from responsibility in case of injury or death Then he was taken down deep into a great concrete structure along the waterfront Thus began one of the many exciting adventures of United Press staff Correspondents who bring you eyewitness accounts of world news as it happens Listen to Clinton B. Conger story of night patrol Here you are, sir. This is me. Hello, Clint. How are you, Fraser? Hey, what's the pitch here? Where are we headed for? Holland. I don't want to go to Holland. They've got Germans there That's just where we're going at the same promised me an interesting time But that's after to be just a little too interesting How are we getting to Holland if I may ask motor torpedo boats, you know, we call them PT boats back home Murder I get seasick as a dog just looking at a bowl of goldfish. I should have stood in bed I gripe loudly and fervently but just the same I was pretty excited about the whole thing I was going to get my first ride on one of Britain's highly secret patrol boats that sweep across the English Channel nightly to disrupt German shipping As I stood talking with other correspondents in the underground dock I became increasingly certain of one thing Before the night was out a whole mess of Germans probably would be taking potshots at me I anticipated the prospect with what might be called mixed emotions Power so the door opened again and a Navy officer came in he was carrying a hat. Good evening gentlemen I have a head full of numbers here each of you will please draw one What's the idea the numbers will determine which of the patrol boats each of you is to accompany? So will you draw please one by one? We drew folded slips of paper out of a hat. I picked number one the first boat That meant I'd be in a position to see all of the action But it also meant I'd be in the boat that the Germans would likely concentrate their flower pond My shirt suddenly became very tight around my neck. Who has drawn the first boat? I did Will you come this way mr. Conger Thanks, you're locked I Followed the British Naval officer from the underground sharp room into what at first appeared to be a storeroom of some kind It was pitch-black Smell of gasoline As my eyes became accustomed to the dim light I gradually made out the outlines of one of the British Hornets, which is what they call their PT boats We went aboard the 80-foot craft Vibrated gently underfoot from the throbbing of its three great motors The young ensign met us on the wing of the bridge. Uh, mr. Selfridge. This is Clinton Konger of United Press Very happy to know you mr. Konger and have your board. I'll let you know later whether I'm glad I came or not Well, I think I leave you now mr. Konger. You're in good hands. I'll see you when you get back That sounds encouraging. I'll see you later.  Konger I'm afraid I'll have to ask you to go down below and we'll see All this very hush hush stuff, you know Sorry and all that but there isn't any alternative. I understand. Which way do I go? Evans, take mr. Konger below and see if he's comfortable, he'll have a I followed a seaman down through a narrow folding hatch that evidently was closed off during rough weather And on another short ladder into the cruise quarters The seaman left me there and I sat down thought about life and things about how many bullets there are flying around Loose on the world that a man can get to the way of Cars go all four or not. Take it one and two Take it in both. Take it in two All clear. Stand by. Next station is for getting underway The thin hull of the torpedo mode vibrated a trifle more and I could faintly sense that we were moving From time to time I could hear orders being shouted over head. Evidently something was being done about a huge door I had the least idea of where we were How we were going to get out to sea and I still don't know All I know is that after a half hour or so the boat began pitching and tossing heavily and I heard the officer parking All hands put on your mewests man blessing station So I put on the life preserver that the seaman had given me and I tried to make myself as comfortable as possible on that narrow bunk After a while the motion of the boat humming of the engine slowed me to sleep Mr. Konger Mr. Konger I went up on deck with him It was dark now and I could barely see the other boats in the formation lined up behind us I went up on the flying bridge and you have a good nap, mr. Konger. Yeah any excitement? No, not yet I thought I might as well let you sleep on the way out Very dull really, but we're getting close now another hour or so will be in the thick of it I shouldn't wonder not too thick. I hope And they told us to give you fellas a good show So we're going a bit further in than usual They're quite likely to have a first rate rompers. That'll be nice. Excuse me The quartermaster tell the engineer to stand by for emergency changes of speed. I watched while the quartermaster crawled after Along the narrow deck to the engine room where he opened a little hatchway in the deck Then a head appeared in the opening a quartermaster shouted into the ear of the engineer Ye grin nodded and then disappeared I asked Selfridge about the engine room. It seemed awfully small from where we were standing. It is small Chaps have to crawl around on their hands and knees down there. Oh, there's more than one Oh quiet two men three engines. Greatly bad job there I shouldn't like to have it.  You mean we aren't going top speed now? Heavens no, only about half. We can make 70 knots, you know I didn't know but 70 knots is a healthy 80 miles an hour I had a sudden great respect for these boats and for the men who run them As we raced for the dutch coast I looked at the boat over from end to end hanging for dear life to a railing thoughtfully provided for that purpose The forward end of the boat was high out of the water now The stern was dug down deep as the propeller screwed into the sea Everywhere I looked there were machine guns torpedo tubes The ensign's elbow was a small deck gun All together this was the evil's deadly looking weapon of war I'd ever seen And I was on it hurtling at top speed toward german occupied holland Let's go carefully now mr. Cunger minefields all around us minefields uh hours are theirs We must be getting close. I should say so practically there actually Coats is a mile and a half off in that direction I looked very pointed expecting the blackness to open up with the flash of gunfire nothing happened. I couldn't see a thing Then self-rejorted the engines muffled and an amazing quiet descended on our fleet was vaguely disappointed I'd expected great things and a terrific amount of gunplay Instead we cruised up and down the dutch coast as though we were in a lake back home Seemingly we owned the entire english channel and no one was in a mood argue with us about it Hours went by I was on the verge of going below and returning to the arms of morpheus when self-rej spoke uh-huh A big pardon I said uh-huh. Yes, but why business the convoy jerry's where there I couldn't see a thing. I was about to tell self-rej I thought he was imagining things when the night cracked wide open with a blaze of light The germans turned to search light on us take off mufflers full speed all right brother Hang on mr. Conker I hung on The torpedo boat almost leaked out from under us as the engines opened up Then we twisted around in a complete circle faster than I can tell you about it Search light tried to follow us, but couldn't Then the whole fleet of torpedo boats dashed off in all directions. Self-rej grabbed his microphone. Hello. Hello. Hello. Come in strawberry All for orange calling strawberry come in please. Hello jack strawberry here. Good boy, tom I think we have them rattled you go left. I'll go right We'll come in behind them. Is that understood over to you. Yes, I understand orange. See you later over and off We raced through the darkness at full speed I kept thinking about those minefields all around us But evidently self-rej didn't worry about them because we never slack and speak until we were on the other side of the german convoy Then the mufflers were put on the motors again and we crept slowly in towards the enemy ships Everybody was tense and quiet And then dimly at first I saw a large shape looming up ahead of us An enemy ship Stand by to fire torpedoes standing by sir stand by old guns and train on target standing by sir Quietly we edged toward the german ship She was about a mile away when we first saw her We covered about half the distance when the blinker light flashed at us from her bow They're challenging us sir answer them. What shall I say sir? I don't know they're cool. Send anything Just keep their minds occupied a few seconds longer until we can get in range. Aye aye, sir The quartermaster picked up a blinker tube and flashed an answer to the german ship which questioned our identity Meanwhile, we crept closer and closer And the guns on the german ship erupted into flame and the night was filled with whining billets Open mufflers full speed their collision course all guns fly Now our boat leafed through the water again directly at the german ship right into the teeth of our machine guns in three inch shells I ducked down below the bridge and stayed there crossed behind the bulkhead Occasionally I stuck my head up for a look The enemy ship was really a small one, but she looked bigger than a battleship to where I sat I felt acutely unhappy especially so since we were steering directly from the enemy ship and going like a fat out of brooklyn Selfish stood up with the tracers whistling around him and kept his eyes on the enemy ship I was content to keep my eyes on selfish Healed over to get out of their way I learned afterwards we went right by the torpedoes and kept on going past the german ship I wasn't watching I was huddled behind the comforting protection of the bulkhead A few seconds after we healed over and raced away the whole sky seemed to fall in on us I thought for a second he got us. I wasn't gonna say so I got up off my hands and knees some bridge helped me up. Did you fall mr. Cunger? No, I uh dropped my handkerchief when those bullets started whistling bomb It's a nice bucket you got there. Yes, right be much nicer. However, we're on a place Isn't that? Very not it's just plywood you can spit through it. I um, I think I'll go lie down It was quite an adventure Several heavy german escort vessels came out after us and we decided to call it a night for which I was duly grateful On our way back to england the ensign apologized to me for not having been able to show us more excitement I accepted his apology You have been listening to united press correspondent clinton b conger's story of his experiences as a reporter Abort one of the lightning fast british motor torpedo boats that patrol the english channel conger is one of a worldwide staff of united press Correspondence who are eye witnesses to events that are news We will bring you another program dramatizing the experiences of these men of up in the near future Your local announcer will give you the time of that broadcast in just a moment Be sure to listen and meanwhile listen for united press news on the air Look for united press dispatches in your favorite newspaper. They are your guarantee of the world's best coverage Of the world's biggest news


This suggests that interventions aiming at increasing bacterial richness may be beneficial in improving HIV-1 associated immune dysfunction. This article is authored by Mark Nogara-Julian, Monserokafort, Yolanda Guyan, and others.


Good afternoon. Good evening, everybody. Before we start, I have a quick question. Anybody, if you can raise your hand, you know what CICD is in the room? Anybody know what Jenkins is? We're in the right room. It's good. Welcome. We are the last session of today between you and all the parties which are happening this evening, so we hopefully won't keep you too long. We're talking today about the OpenStack plugin for Jenkins and a decent number of all the sessions that we've seen up until here and a number of the discussions in the user summit and the operation summit and the design summit has been about how to use a CICD environment to build OpenStack and to continuously deploy and continuously integrate against OpenStack. What we actually did is we're taking a slight different view of the picture and using OpenStack as our compute resource to do a regular CICD workflow, which is developing applications. So, please. My name is Naama. I work in the CI infrastructure team in Cisco. In the past few years, I'm working in multiple cloud environments lately concentrating on OpenStack. With my team, we're building tools which will ease or enable deployment, configuration, and testing in OpenStack environment.  In case you use the plugin, we'll show and have questions or issues. So, you have the emails here. And my name is Meish. I am a platform architect for Cisco in Israel. We both came into Cisco through an acquisition of a software company called NDS approximately two and a half three years ago. Where we primarily are a development shop, relatively big development shop, doing media applications for service providers, providing content. Confirm you on Twitter and so on and so forth. My email, of course, is on the front page. A tiny bit about Cisco. Cisco has very, very significant commitment into the actual into OpenStack and to cloud and to the community. We have three main verticals or areas where we try to focus on helping organizations and people use OpenStack or connected with OpenStack. Number one is the fact of building OpenStack, of how we provide contributions into the actual infrastructure of building OpenStack environment itself. The second part is how to use or make things easier to use with Cisco equipment and in general of OpenStack, which has contributed into horizon monitoring and use cases such as those. And the last part, which is Cisco specific, is how to connect clouds one to another, be it from your private cloud to a service provider cloud in the public realm or between two different private clouds with the technology. But today what we're going to be talking about is actually the fact of the left block over there is the building of OpenStack and using this OpenStack as part of your infrastructure. Okay. Our environment is made up of a lot of RPMs and a lot of components. People keep on continuously building RPMs and components from a number of different teams geographically dispersed all over the globe in different time zones. They keep on providing their code, which is checked in continuously through a Jenkins workflow into a repository. And eventually after the whole process goes through a number of iterations and a process that we have, we hope to have some kind of a very well oiled machine, which works very, very well together, and has to be timed and pieced together in a certain way in order for these things to work properly. It's not a very generic product. Our products are very, very tightly coupled and very specific to what they need to be doing. And eventually after the whole process goes through a number of iterations and a process that we have, we hope to have some kind of very well oiled machine, which works very, very well together, and has to be timed and pieced together in a certain way in order for these things to work properly. It's not a very generic product. Our products are very, very tightly coupled and very specific to what they need to be doing. And we had a certain requirement that we needed to do. What we wanted to do was we wanted to make some kind of a CICD pipeline using OpenStack as our compute resources. And why specifically OpenStack? Because the requirement we had was we need to deploy our applications on OpenStack either as a service which we provide to the customer or as an environment that the customer primarily already owns. It needs to be deployed natively on OpenStack. In other words, we don't want to start installing instances or applying instances and start doing the configuration and adaptation afterwards. We wanted to do everything through the native OpenStack tools. And with our CICD process, we wanted to have what we call the minimal viable product, which we knew would be working, would be the minimal product which our customers could use or at least for our testing purposes we could use and say this is something which we know which works in a highly available fashion, in a scalable fashion. Provide these smoke tests or these acceptance tests after the fact to see that everything has been deployed. And it's not only deporting one or two RPMs as you'll see further on, it's a certain very decent process which needs to go through. And of course, if all these acceptance tests had passed, which sometimes they do, sometimes they don't, we would pass it on to the next stage in our pipeline that somebody else can take what we have built as a product or as a suite of products. And they will be able to use that as a fully packaged product which they could continuously or deliver or integrate against on the next stage of our pipeline. So just for terminology, so that we will all are more or less on the same page. We're talking about components, we're talking about RPMs or applications which are packaged. We work with RPMs. Everything is built on a continuous basis whenever a developer checks in the line of code. We ran our stage zero, stage one testing to build the RPMs to see that everything passes. And what we were doing, we were collecting all these RPMs on a continuously basis several times a day or sometimes also several times an hour in order to collect these products which the developers were creating or fixing and maintaining and package them into some kind of what we would call a component. Each product was a multitude of components. In other words, we had component A, B or C which made up the whole product A and we had a number of products which made up a whole solution which we provided to our customers. As a fully packaged solution, we wanted to provide this as we said in a workflow. So we provisioned using heat products. If we go back a minute, the column in the middle, there's products which are combined from multiple components. This is one YAML file that hopefully deploys this product. Having multiple teams create components that are joined into one YAML file, one heat template creates a few challenges. If it's one huge YAML file, one monolithic stack, it's very hard to maintain it. If multiple teams do changes, it's hard to track what changes were done and which changes we want and which we don't. The versioning is also a problem. If we want to roll back one component, one version, then we're stuck with a template which has changes of multiple components. So we have Git and Git Merge is both really great, but it is still a problem. So what we proposed is a modular approach. In the modular approach, we expect the heat template to include only the deployment model to define which resources, networks, floating APs and of course virtual machines we want to create and what components we want to install on each VM. The code itself of the installation and configuration of each VM will not sit inside the YAML file inside the heat templates. Instead it will be put in the Maven repository we use Nexus as a sibling to the RPM together with the RPM. And then we'll have to collect it and somehow orchestrate everything so that when we actually execute the heat template, we'll know what code to execute. We gained two things from this approach. First, we set the ownership of the deployment code to the team that develops the code. So whenever the developer does any change in the component that requires changes in the configuration, he is responsible for changing the configuration code also. And also we gain versioning. The installation code sits together with the component, so it's versioned with it. If we roll back from one component, we roll back the provisioning code of that component and we can use the same heat template no matter what version we use. We do not force this modularity. The plugin that we're going to show will support also one monolithic YAML. And also if you want to put all your use puppet to the provisioning, if you want to put all your puppet scripts in one Git repository, it will also be supported. But if you have the puppet code sitting together with your components, then we'll support these two. So it does make things a lot more complicated, but we had a very complicated system. We needed to put a number of things together, and as Naima says, we needed also the modularity. So we looked at a number of solutions, and specifically we all were already using these tools. We used Jenkins for our workflow. We used Nexus at our Maven repository. We used Git for some of our source control. And we were using, this was the last requirement, was using deploy everything on OpenStack. And we looked for a number of solutions to see if there already was something available. And at the time when this whole project started approximately nine months ago, a year ago, less six to nine months ago, there was no Jenkins plugin for OpenStack plugin for Jenkins. There was no OpenStack plugin for Jenkins, at least it wasn't very, very viable. There were a number of other options which were semi-supported or in better stage. But the biggest problem we had out of all of these solutions, which were, for example, JCloud library, or we were something which was called OpenStack 4J, which was also Java library which we worked in, which was being developed. There was no supported heat. Everybody would start a Novaboot instance or spin up an instance, but there was no body which would support heat. And even more so would not support heat with nested stacks underneath, which is what we have in our environment as well here. And also we needed an end-to-end solution that does also the packaging and putting everything together in addition to the stack creation. And as we said, we at the moment support, this plugin at the moment out of the box comes with support for our Nexus Maven repository. It can also be adapted and expanded to do Artifactory, which is one of the things which we're going to be doing very, very soon. But if anybody would like to contribute because it's going to be an open source project, you're more than welcome to expand it to whatever kind of repository you want and expand the project as well. So the plugin includes two steps or two build steps that we can use to define jobs. The first one we define what our product is, what component it includes, what other dependencies it has, what is the heat template. It goes automatically, so leave it as it is. Then when we execute this job, when we build the job, we generate a product artifact. Then we define the deployment and when we execute the deployment job, it is deployed on OpenStack. Let's see what happens in each step. When we define the product, we define what, as I said, what RPMs are acquired, what are the dependent products. Maybe we have two products, one that deploys a database, which may be used in multiple products, and then another product that uses this database. So we'll define the database as dependent on, let's say, the website product. And then when we'll deploy it, we'll deploy both products. We define also what heat template is used and where to take the puppet scripts from. The puppet scripts which are not put in Nexus together with RPMs. When we build this product, what happens in Jenkins, we first resolve the versions. When we define what components we want, we usually, especially in CI scenarios, say, get latest or get latest snapshot. So first we'll resolve what the version, what is the latest version. We can also use fixed versions, of course. Then we download all the dependencies, the ones that are not in Nexus. We generate three artifacts, a POM file that describes this product, a tar file that includes the heat template and also the external dependencies, which Yam will use if there are RPM dependencies. And also an RPM which includes all the puppet scripts, which will be used for the installation. And then Jenkins uploads these three artifacts to Nexus. We'll see all this in a demo in a few minutes. We then define the deployment job. When we configure the deployment, we configure three main things. What we want to deploy, where to deploy to and what Yam repository to use for the deployment. What product we deploy, what is the version that we want to deploy now, where to deploy, we put all the open stack credentials, all the parameters that the heat stack requires. And then the Yam repository, in order to do the deployment, we put all the RPMs in a Yam repository, configure the metadata and then all the VMs which are created, the Yam installer configured to point to this Yam repository and we'll install from it. And then when we deploy from open stack, we'll first see if the stack already exists in open stack and then we'll delete it using rest commands. We'll download all the artifacts from Nexus. We'll take also the dependencies from the tar file. We'll move all the RPMs to the Yam repository. We'll load the stack. We'll set the parameters. If we have set parameters in Jenkins, we'll call open stack rest call to create the stack and then we'll wait until the stack is complete. No, this is not right. When we do stack creation, we have a few limitations or a few standards we require from the stack in order for us to be able to load it. And again, this is a part where you may contribute and make it more generic. When we clone the heat stack from a Git repository, we expect multiple heat stacks to be in the same Git repository. So we expect a folder called after the product and we expect three or two files in one directory. If the product is called XX in the example, we expect the main YAML file to be XX.YAML, the environment variables to be put in XX and VAML, and then another directory with all the rest of the YAML and bash files and whatever is needed in XX.YAML dependencies. In our proposal, in the modular approach, these dependencies will include one bash file which is a few bash files. One of them will be generic and will be used in all our stacks. And what it will do, and this is the bash file which will be called for every VM we create. What the script will do, it will install Puppet and all its dependencies. And then it will call PuppetApply to apply, oh, it's first, sorry, before that it will install the RPM containing all the Puppet scripts. And then it will call PuppetApply to execute all the Puppet classes that we want to be installed on this virtual machine. So these are the requirements on the structure of the heat stack. When we create the stack, we load these files, we pass the YAML and then we replace the parameters according to the parameters defined in the Jenkins job. And then we call the rest call to create stack from open stack. This might seem slightly a bit about Chinese or Japanese, I don't know how many of you speak of that. But when we get to the demo, it will become a lot more clear. As we said, we have a number of components which are make up a multitude of products which come to some kind of a solution. So each product would actually, I would think of more each component. Or some, it was divided in different ways that some of the components had their own heat template and would have multiple heat templates which were dependent one on top of the other. So as we said, we needed to nest stacks, in other words, get information passed from one specific stack to the other in order for us to deploy a full solution and test with a continuous workflow to see if we were doing things correctly. For example, if you had a web, if you had one, one of our applications would be a web application and the other application would be a database application. We need to pass the parameters between the two different stacks completely to allow them to talk to each other and test the full product in the end. That is something which you'll be able to see in the demo. There's something called a product create step where you can see where we allow, we can define different product dependencies in the actual workflow that it will allow. Know exactly that it needs to instantiate or which order these products should be deployed one after the other in order to get the stack built correctly. And of course, the thing which manages the whole workflow itself is Jenkins. It's deploys then one after the other and how we pass parameters between the two different stacks or the two different heat templates. In this case, two different stacks specifically. We would take output and input parameters. And by doing this, we had one of the requirements was in order for this to work. We needed to define that the output parameters from one stack would be identical to that of the input parameters coming to the next stack. So for example, if I had an IP address variable, which was passed out from my first stack, I would pass that as the same variable. And this value into the next stack and continue the deployment that allow us to do wiring between two different stacks itself. So we prerecorded the demo since we're afraid of network issues, not because we're not sure of the stability of OpenStack and our environment. But this is almost the only cheating we've done. I'll be very frank about the cheatings. Let's try to change them. Oh, yeah, this is better. You guys see or not at all? Yeah? Okay. Okay. So. The other one. This one is. Yeah. Oh, so it's stopped. Okay. So in this demo, I'll, it doesn't know it doesn't know it doesn't know it doesn't. Okay. There we go. So in this demo, I'll show a deployment of a very simple product that includes only. This is the beginning. This is not this. This is not the right. You have the right file. I'm sorry. I'm sorry. Hold on one second. My apologies. One second. Right here first. This is the right one. Can I just make sure? Okay. Forgive me for one second. Okay. Even if you record the demo. There are issues. Okay. Okay. So we'll define a very simple product that has only, it doesn't, does it run? Only one component which is NGINX. We'll start with looking NGINX in Maven, Maven repository. And you can see here that we have two artifacts. In addition to the RPM, we have also a targe Z file that includes the puppet scripts. By the way, we have a puppet Maven plugin that generates this puppet file. We'll upload it to open source shortly. Let's look at the puppet file. It's a very simple class that just installed the package and then it changes a little bit the landing page index.html. It uses our parameter NGINX user, which we'll set. We'll see how we pass it from Jenkins to the heat template, to the puppet, to the deployed application. So this is the RPM and its puppet script. Let's now go to Jenkins and first we'll create a folder just to make things nice and clear. We'll call it demo. It's a folder. And I should click okay. And no, no. The other save. Yes. We'll define our job, which will be called create. It's our freestyle project. And here we add a build step. This is a build step that the plugin provides, which is create product. We can see what parameters we need. First we'll give it a name, which is the artifact ID. We'll call it demo product and a group ID. And we set a group ID that Maven Nexus knows. And then there are three Git repositories. One for the heat template, which is required. And it's supposed to be according to the style that we've seen. Additional deployment script. We don't need them right now, but we can use, if you use bash, you can use this repository. And a repository for the puppet script. You can put here all your puppet scripts if it's not modular. I'll stop it for a minute. And you can also put here only the shared libraries, puppet classes, which are shared between components, and then put you product specific classes in Nexus. We'll now continue. Yeah, this is space. Why doesn't it move? Okay. So we can have dependent product, but we don't need any. So we'll add now artifacts. We need only one artifact, which is we set the group ID and artifact ID. We set the version to latest. We could have set it to a specific version also. And we can add, if we've had more dependencies, which are not in Nexus, we could have added them there. They should be accessible by HTTP. And then we'll package them into the same file with the heat template. We don't have any dependencies here. So that's all we need. Let's now build this job. And let's look at the console output of Jenkins. So we can see here, first, we find what the last version was, and we set and then we increase the version. You can define which part of the version to increase. Oh, sorry. So the next thing we do, I can see from there, we resolve the version of NGINX. It's 1601, as we've seen. We then download the Git repository that we defined. We take the tar file, not the NGINX component. It happened during deploy, but in order to create the Puppet RPM, we collect all the Puppet scripts from the components, in which case it's only this one. We then generate the three artifacts. The tar file with the external dependencies, which will include the heat template, and if we would have had dependencies, they would have been there. We also create the POM file that describes the dependency and the RPM, which includes all the Puppet scripts, one from the Git repositories, and the one that we created from the components. And we finished here this job. Let's look now at the Nexus to see the product that we created. We created now artifacts which are uploaded to Nexus. So we can see here the three artifacts. The POM file, the RPM, and the tar. The POM file defines the product, group ID, artifact ID version, and the important part here is the dependencies. We have only one dependency, the NGINX, and we can see here the exact version, not latest, because this is the product that we want to deploy. So everything has to be a specific, not latest. We should be able to redeploy it even when we have newer versions. The next thing is the tar GZ file, which includes the heat template. Let's open it now. We can see here in the archive, it will require now a few clicks. I think for two more. Now it will work. So this is the heat template. Let's open the main YAML file. You can see the names, the name pattern that we talked about. We'll open our demo.yaml. Oh, no. I'll just go over there. So these are the three. Demo.yaml, demo.env.yaml, and the dependencies. And now we open them. In the YAML file, we can see the standard parameters, resource group, floating IP that we define, and then the web service virtual machine we create. And we can see here that we passed the nginx parameter, the nginx user parameter, which we've seen in the puppet script, is passed now from the heat template to the bootstrap script, which will use it to put it as higher data, which will be used in puppet later. If we would have had more dependencies, they would have been put here also in this star file as a repo folder, and will be used later in the deployment, but that's all we have here. And the RPM file includes the puppet scripts. Let's open it too just to see our beloved nginx puppet class. We'll need less clicks here. So we can see here in the models that this is our nginx model with some more classes that we had in the puppet base, Git repository. So this is what we have in Nexus. Let's go now and create the deployment job. We have now a packaged product. We can always go back and deploy it. So we define a new job, which is called deploy, another freestyle job. And we'll need here a built parameter, because we don't want the version to be part of the configuration. Each time you deploy, you want to deploy a different version. So we define here a built parameter, a string parameter, we'll call it version. And then we'll add the build step, which is deploy product. Three things we have to define. What to deploy, what to deploy it to, and what YAM repository we want to use. So what we want to deploy, we'll put here the group ID and artifact ID. The version will be transferred as a built parameter, as we've seen a minute ago. We're to deploy too. You can see here our Jenkins credentials. You can also test them here. We have the DNS server. Jenkins also creates the DNS records if required. And then we can add advanced parameters. We can see here both the standard parameters. We have the subnet ID, private network ID, and we can add our own user defined parameters. In our case, as you remember, we had the nginx user parameter. So we'll add it here. And we'll set the value to mesh in order to my colleague. We need another user defined parameter since we use a floating IP, so we need the public network ID. So we'll set it here also. In addition to the user defined parameters, we'll now have a few more definitions of what image we want to use, what flavor we want to use, if we have a scaling group, what is the min and max. We'll add now availability zone. The names are added as prefix to the names, some requirement of our environment, but we don't need it here. And we'll set the YAM repository. As part of the deployment, all the VMs will move them to some YAM repository. And then we set the metadata and all the VMs we create will use the artifacts from there. In order to push the artifacts to the YAM repository, we need the private key. So we'll put here the private key. You can read it really fast and then break into our system. Okay, so let's now... This is our mistake. Let's now build. When we build, we have to give the version number. It's a build parameter. We'll set it 25, the version we just created. And now let's look at what happens in the console. So, yeah, my eyes. That's what they used to be. Okay, so the first thing we do is we delete the stack if it exists. In this case, it didn't exist, so we see the status is undefined. We download the product artifacts, the tar file, the RPM, and the POM. We then extract the tar file to get the heat template, to get the dependencies. We download the RPMs. We have only one RPM, the NGINX one, so we download it now from Nexus. And then we move all the RPMs, the Puppetscripts RPM, NGINX RPM, and if we would have the dependencies, we move everything to the YAM repository. We configure the metadata. We load the stack. We override the parameters, the user one, all the parameters we've set in the job, and then we call OpenStack createStack. Let's now look at the YAM repository. We can see now that it started, and we're waiting, we're pulling OpenStack every minute to see when it's done. Let's go now to the YAM repository. We can see here the new RPM, 25, it includes the Puppetscripts and also the NGINX that we moved to the YAM repository. So when we do YAM install, after we configure the VMs, they'll pull the right components. We'll now go to Horizon to see the stack. I know it's an old version. So we can see the demo product. Now in progress, we go back to Jenkins, and this is the other cheating that we did. I skipped here two minutes. But this is it. So in a few seconds, we'll see that it succeeded, but actually it took about three and a half minutes. Of course, if the stack fails, then the job fails. So we can see here that the stack is complete. Therefore, the job succeeds. We'll move now to OpenStack. We can see that the stack is complete. If we go and check the output parameter, I set the output parameter to the URL of NGINX, and when we go to this URL, we can see hello, Mesh, which we passed from Jenkins to Hit, to the puppet scripts, and this is it. Thank you very much, Emma. And of course, as we said, we prefer not to do this live because of all the success records of the demos this morning and the keynotes. I don't think we want to risk it so much. Which we knew in advance, of course. So going back to our presentation for one second, you can find the plug-in over there. Open source to download it. Please contribute, add your... And of course, these links and the presentation, everything else will be posted after the... Usually they'll take approximately a day, whatever else it is, but they'll be available. The presentation will be available, which has additional screenshots inside, describing all the processes without the actual video, so you can see more or less what is going on. The second link is Cisco General GitHub repository, where we have a decent amount of work going on on OpenSec and a number of other projects as well. I would advise you if you have the time to go through to see if there's something you can find useful and also, perhaps, if you would like, because it's an open source, contribute back as well. There's one fix I want to push, so wait one day before you do pause. Okay. Or before you send me bags. One of the things we want to... We will hopefully add to the plugin is support to Artifactory. This will make a few changes in the UI also, because Artifactory can act both as an amazing repository and a young repository, so if you use Artifactory, you don't have to define both, so it will be a little bit easier. So we hope to do it in a few weeks. So we have a few more minutes left over before everybody goes to have a nice evening. If you have any questions, please feel free. It would be better if you can come up to the microphone for a second so everybody else can hear, and it will be caught on the recording as well. Thank you. Hi. So I was been working on trying to put together something similar to this. In particular, we have Nexus also, and I noticed in recent versions they at least advertised that they host young repositories as well. I was wondering if you had actually tried that or seen that. No, I didn't know, but it's a very good idea. Make the Artifactory work even better because it will be the same implementation for both. And I had another question. Is there a particular reason why you packaged the Puppet code directly rather than using, I'm assuming you're using masterless Puppet? Yes. This was our requirement of our organization. People did not want to support our master, and they said within UVM you can do whatever you want, but no other infrastructure to be supported. And this masterless method really worked well for us. You can do on every node whatever you want. You can install multiple instances with different versions. So I think it worked well. I know that the WebEx project also used masterless Puppet. And one more clarification was that the requirement actually was that we were not allowed to assume that it would be a web server where we can pull files from through curl or any kind of download of a file. It has to be packaged in a certain repository as an RPM. You would use the ARM installable to get it. This is why we packaged it as an RPM and not just a TARGZ file. We could do git clone inside the VM, but the assumption is that from the VM you don't have access to any internal repository. And the rules were in our organization, the only thing you could use is the ARM install. No other access. Yes. You had a question, sir? I'm just curious. So now that you have Jenkins and you're able to effectively package your application and then create a new heart and basically VM and then deploy it onto it, I assume you guys are going to the next step where you then run scripts or run some kind of automation to do some testing on it. And I'm curious if there's the follow-up where you then, if the testing succeeds, you delete the VM. Is that something that you guys will be doing for this, not this particular plugin, but something you might be doing already? Right. The R tests which are being written right now to run against the new stack, we decided not to kill the stack immediately after the test, but to leave it alive until the next one. So if any test fail, the developer will be able to access that same instance and check the logs and see what's wrong and rerun the same tests. So there are two options, either kill the stack in the end of the test or in the next run kill it. That's why in the beginning of the deployment job, we check for the existence of the stack and then we kill it. Yeah. I actually suggested this exact same thing in our company where we're trying to do something. I'm definitely going to be... I'm glad. Any more questions? You can actually use the Jenkins to trigger the deletion at the end of the successfully view of the packet of products. So that's one way of validating that whatever we have been testing is actually good to go and ready to deploy for any other environment. Right. What we didn't show here, we have a workflow job. There's a workflow plugin to Jenkins that activates all of the jobs in order. It first creates the products and then deploys it, then run the tests, and then if the tests succeed, it promotes the product. We first, when we upload it to Nexus, we upload it to staging repository. And then if the tests fail, then we drop the artifact. If they succeed, we release it. So these are other build steps which exist in the Jenkins plugin. We just didn't show it now. It will a little bit too many details, but yes. We do the same thing. I don't know if it's good or bad. I'm glad that we do the same thing. It's a pity that we repeat the work. That's a surprise, yeah. So thank you everybody for joining us. Hope it was useful and enjoy the rest of your evening.


Hello, welcome to yet another session of our NPTEL on non-linear and adaptive control. I am Srikanth Sukumar from Systems and Control IIT Bombay. So we have this very nice motivating image in the background. This is of this nice SpaceX satellite which is orbiting the earth and we are sort of moving towards designing algorithms which will help drive systems such as these autonomous. So we are going into the sixth week of this course and sort of with a halfway mark if you may. And we have already started to look at design elements. So we have already started to see how to design adaptive controllers for a first order scalar system. And that's what we have been doing. So last time we had already done the design in the session before last and we had essentially arrived at a point where we had a negative semi-definite V dot for the unknown parameter case. So we had a control log, we had a control log we had designed and we also had an update log for the parameter for the unknown parameter. All right. So the control of course depended on this unknown parameter update. All right. So with this when we analyzed the system using this Lyapunov function that we constructed, in fact the update log was obtained using the set Lyapunov function and we got a negative semi-definite V dot. And what we did last time was to start from this one, this V dot and use the signal chasing in Babalat's lemma method of analysis which we had learned sort of in the analysis part of the course in the first few weeks. Right. And we use this V and V dot in order to prove that well we had already proved that we have uniform stability for the states which are the error that is the tracking error and also for the parameter estimation error. All right. So what we wanted to do next was to prove that the tracking errors go to zero, converge to zero. So we already had a stability design. We wanted to sort of obtain a convergence result because stability is not enough. Right. So that's what we did. We were able to prove in fact here. Yeah. Okay. From step number four if you see we were able to prove that the error goes to zero as t goes to infinity. Right. So in spite of the fact that we don't know a parameter of the system, we were able to show that we can do accurate tracking just by adding a parameter update. And so this is the magic of adaptive control. This is what we did. Excellent. So I hope you sort of would see that there is this rather cool element in adaptive control. Right. So now beyond that we of course want to see if the other state which is the parameter error can also be driven to zero. All right. And we tried to do that. But honestly we were not successful because we reached a point where we said that we found that the product of theta tilde in this function f goes to zero as t goes to infinity. All right. Now if you sort of have something very simplistic thing and so this is where we start let's say lecture 6.4 we're going to mark it here. All right. And you sort of look at this and then if you can make a very simplistic sort of assumption which is probably not a very viable assumption to be honest and take you why very simple. Right. So if we assume that f let's see limit as t goes to infinity f x t t is equal to zero is not equal to zero. Okay. It's not equal to zero. Then there is hope. Okay. Then there is hope that theta tilde doesn't actually goes to zero. Then can hope theta tilde goes to zero. All right. Which is also not a guarantee. Yeah. I mean so this is a rather complicated thing to talk about because we're talking about things converging to zero and I mean as t goes to infinity and things like that. So we're talking about objects in the limit. Yeah. Now this itself may not be a very very viable assumption in all scenarios. Yeah. It may be in the tracking scenario. Yeah. So but notice that your dynamics of the system was something like x dot is a f sorry I apologize was something like theta star f x t plus a control. Okay. Now if I typically if I assume that you know zero is an equilibrium then what you would have is that to ensure that zero is equilibrium typically you would have f zero t would be equal to zero for all t. Okay. For all t f zero t would typically be equal to zero. Okay. If that's not the case then you can see that this right hand side being zero at x equal to zero and u equal to zero at typically at the equilibrium the control is also expected to be zero. So the right hand side will not be zero and it will not be an equilibrium zero will not be an equilibrium. So you would expect f zero t to take on the value zero for all that all time. Okay. Now if that is the case here you see I apologize here I'm requiring f zero t to be equal to zero and here I'm requiring that as t goes to infinity f x t t is not equal to zero sort of looks like opposite requirements. Now if I'm looking at trajectories are right which actually go to zero as t goes to infinity this is possible and there's many possibilities if I'm looking at say trajectories like this is like trajectory that is actually going to zero as for large t for large t if you're the trajectory going to zero then this is t and this is r of t all right then this is difficult right I mean there's a difficult situation where you sort of see that if r goes to zero then as t goes to infinity with the error tracking error going to zero x itself is also going to zero. So you can imagine that as t goes to infinity this by standard continuity type arguments this function itself has to be rather close to zero in fact we go to zero itself right so this is not satisfying yeah so this condition is not going to be satisfied yeah for large trajectories which are going to zero also for the stabilization problem where r of t is actually zero for all time even in that case this sort of a condition cannot be satisfied so it's not very easy to have satisfaction of a condition of this yeah so anyway this is a naive requirement I would say there is a naive requirement so this is the sort of lesson in adaptive intro and what is the lesson is that typically you can show that the state follows the desired trajectory accurately this is guaranteed no problem okay our parameter convergence or parameter estimation is never guaranteed in fact you need persistence of excitation for the parameters to converge to that true values all right so now of course in the previous week we spent a lot of time before we go on to the second order system in the previous week we spent quite a bit of time looking at you know persistence and notions of persistence all right and so what I want to sort of do is try to give a I mean we almost don't in most circumstances analyze parameter convergence in adaptive control and whatever we see in subsequent lectures yeah because parameter identification or any identification or system identification is a complete problem that predates adaptive control and I mean you can imagine if you want to identify for example parameters of a manipulator robotic manipulator you're expected to move the manipulator in all sorts of interesting or you know rich trajectory we find me yeah so that you can identify the parameters of this manipulator so you move it in all sorts of rich trajectories all right that's the idea that's the idea now this may not always be possible because we are talking about doing a control task that we are not talking about just identification in itself okay when you're doing a control task you're expected to follow a certain trajectory right for example a robot has to say move in a along the constant trajectory yeah it has to say move along a straight line yeah and not oscillate all around just to identify the parameters that's not very okay as far as the operation of the system goes and so it may not be possible to do good identification of parameters so most engineers and adaptive control theorists are happy to have good tracking performance yeah and if they do get parameter identification in the process excellent okay however since we did do persistence and we studied persistence of excitation in the last week we do let's just try to look at what we can get out of this so let's see let's see what we have let's first write out the system equations right so what is e dot let's use this problem yeah so e dot was minus k e plus theta tilde times fxt right i think that was it right this is it minus k e plus theta tilde fxt this is what was e dot and what was theta dot theta tilde dot theta tilde dot is just minus theta hand dot which is this guy okay so let's write it off so theta tilde dot equals minus theta hand dot equals minus one over gamma times fxt times e so if i write this in the nice matrix form and get e theta tender let's see if this is correct or not yeah one over gamma fxt excellent and this is that excellent this is it okay is equal to minus a then i have fxt then i have minus one over gamma fxt and then i have a zero here multiply en theta tilde okay if you remember the lectures on persistence of excitation you will remember that we had a very similar looking system when we were looking at this result if you look at this result again you look at this particular system where we had these of course these nice requirements a b is controllable a c is observable a had to be a whole weights matrix because then i can have this Lyapunov equation and this phi function had some nice absolute continuity problem and of course phi phi dot was l infinity then we could claim uniform global exponential stability for this entire system okay great great so so what does it mean sorry okay so when too far yeah what does it mean here so this system that we have here is very identical to what we got right now this system so let's try to compare let's try to compare okay so first of all okay first of all what i'm going to do is and is let's assume for the time being that fxt is f of t and that that it is independent of the state and let's make our lives easy and assume that it is independent of the state all right so that's one so i'm going to nicely mark it because this is a big assumption right because here we were talking about dependence with respect to x and all that stuff right but here suppose i do make this assumption yeah so then it starts to match this because it is exactly a function of time all right now let's go back and try to write what these b c and a matrices are so in this case the a matrix is just equal to minus k just a scalar all right the b matrix is just one the c matrix is one over gamma so let me check if i got the signs right yeah c matrix is one over gamma now let's see is ab a controllable pair a and b are both scalars so trivially controllable yes again ac are both scalars trivially controllable observable no problem is a bulwitz matrix yes obviously it's just a minus k k minus k is a stable system so that is to say that e dot equal to minus k times e is a stable system it is in fact bulwitz no problem and finally if we assume further that this is f is absolutely continuous yeah and f f dot is bounded then the e theta tinder system is uniformly globally exponentially stable at the origin if and only if this f of t is persistently acceptable okay that's what it says here fine see if i have all these conditions and we have this we have proved a we have shown ad is controllable we have shown ac is observable we already know a is bulwitz because it's just a scalar minus k if we assume that this phi which is f of t in our case is absolutely continuous and both the function and its derivative are bounded then the system is ugs if and only if phi is persistently acceptable right so that's exactly what you require that f is persistently acceptable so exactly so with and what does persistent excitation mean if you will again recall the definition for persistence of excitation you don't need the signal to be strictly positive at all times right so a signal can be persistent yeah even if it is going through zeros even if it goes through zeros like you know like signals like these which pass through zeros are still persistently exactly okay so that is the nice thing now now if you want to go back to the case where f in fact depends also on the state and not just a function of time right so i will just make a remark here not actually try to prove it because it's a little bit more involved and complicated so so if f x t if f also function of x right then what do you do you write f as you still want to write this as a function of time so you write it as a solution t at some t zero x zero and t this is where you write f s okay and in this case if you go back again we had this more novel result or the more general result it's the integral lemma result right and what did it say what it is help us show it helps us to show that for parameter varying systems also we can have similar results yeah so we can have similar results for parameter varying systems also this was just the parameter dependent output injection and so on and so forth so it can be shown very easily using this integral lemma we did not show but these results are available in standard literature that this sort of a result that you have can be shown also when phi depends on some parameter lambda and can also be shown when phi depends on some parameter lambda in addition to time okay and then this is the result you need to use here yeah so use lambda u p e plus integral lemma okay now what's the deal with lambda u p e and integral lemma see notice that once I write this x here as the solution which depends on time and the initial time and initial state that's if you make a life simple and keep the initial time at zero then this is a function of just the initial state parameter all right just one parameter if you keep the time also initial time also then it's a function of these two parameters now you want a result which is independent of these parameters all right and how do we do that by using lambda uniform p e because that is what gives you uniform lambda uniform global exponential stability that you get all the properties are independent of the parameter so basically you start creating the initial conditions as parameters of the system because they are I mean once you fix an initial time and initial state it doesn't change for the entire duration of your run your simulation or your actual hardware expect right if I if I choose say three seconds when I start my robot and I say that my robot starts in the x y plane at say point at the point one comma two in some Cartesian order then it's fixed for the entire duration of my hardware right that is in fact a parameter it is not there's no time varying there's no time dependence there so we have reduced this problem to a problem of looking at parameter dependent system and this is where the integral lemma and lambda p e we said that we introduced these more complicated notions and we were sort of worried or maybe you know some of you might have thought and why am I taking this expression into more complicated things and what's the purpose and so on but this is the purpose I think most real scenarios you will have functions that depend not just on time but also on state so what we do in those cases we simply write the state yeah as an evolution right as a solution and then it depends only on these parameters which is the initial data of the system and then this is an issue data is the parameters and we just want to give property which are independent of these parameters or uniform with respect to these parameters all right so that's the idea and so that's the idea so in this case you see that under conditions such as persistence of excitation and uniform persistence of excitation or lambda uniform persistence of excitation you can in fact move things like you know exponential stability that is you can get convergence of both the tracking error and also the parameter okay great so usually it's it's not very evident here but not if not if f is just a pure function of time but if f is also a function of the state just like we have here and your state x is trying to track the trajectory r it should be obvious to you that if the trajectory r is rich enough okay so this condition sort of reduces to if you look at this lambda is out of condition this condition yeah yeah this will sort of reduce to you know of t being sufficiently rich yeah so reduced to r of t being sufficiently rich or containing sufficient number of frequency components so when we do simulations and tests that's what we do yeah we want to see r with sufficiently large number of frequency components which is what makes the r of t signal rich and if the r of t signal is rich then the x solution is rich because it is eventually going to track the r signal so lots of oscillatory signals then you have oscillatory x solutions and you have much higher chance of identifying the parameter but like I said like I said already before this is not guaranteed in real applications in a lot of real applications the trajectory is you want to track or not super oscillatory with tons of frequencies and all that you want to do nice simple things infighting that's what you're designing controllers for yeah you don't want them to do crazy oscillatory movements yeah you don't want your robot to be doing this this this just to follow a straight line so you want to do relatively simple things and therefore adaptive control theories do not care about parameter identification yeah so if you want to do just system identification it's a separate question yeah um if you have the freedom to do system identification if your parameters are not going to change with time then absolutely okay you can do your system identification and then feed these parameters to a normal controller yeah but if you do uh not if you do not have the luxury of doing these kinds of experiments for example if you are uh you know one ton spacecraft which you cannot do you know so many oscillations of and move system identification experiments of and there's a chance that these parameters may change in uh space conditions or high temperature conditions or very low temperature conditions then you have no choice but to resort to adaptive control we don't necessarily get parameter identification but you sort of are guaranteed to get precise tracking and excellent so good uh so once we uh sort of understand um what's happened in the schema phase and for the scalar system we want to move on to using our wisdom the same wisdom that we've already learned to try to design adaptive controllers for the second order system so we are just taking very like baby steps one step at a time right so and and please don't think that these are um very trivial steps most mechanical systems that you see around you can be written as second order systems yeah but the thing is here we assume that the system is scalar that is state is in fact a scalar value uh but for mechanical system these may be vector value but that does not change too many things yeah that does not change the fundamental ideas that go behind the adaptive control destiny all right so uh please don't worry uh about you know things being very trivial and very basic you it's very important that we get a handle on this first order and second order scalar systems very well so that we understand how to do the design what the issues are and so on so what is the setup the setup is that you had this system you had this right hand side already we are just adding an integrator to it you're just adding an integrator layer to it okay so if you make a block diagram i would make the original earlier system and then i'll add an integrator it's as simple as that yeah so you have x1 dot is x2 and then x2 dot is given by this yeah so the control appears here in the second layer control appears here in the second year excellent so and as i said the states are evolving in the real number space the function is taking uh function f takes um again the states and time and use you some real numbers theta star is the constant unknown parameter and use as usual a scalar value the standard second order scalar system this is the setup okay what is the aim the aim is now to design a control such that um the first state yeah if you think of it as a mechanical system sort of motion is the position right you want the position state to tag a smooth bounded trajectory okay so we do not specify the second state that is x2 state because it is intrinsically related to x1 state by the derivative therefore i cannot specify x2 arbitrary and therefore if i want x1 to follow r x2 has to follow its derivative okay x2 has to follow its derivative there is no two ways about i cannot have instead of r and r dot i cannot have r and 2 r dot yeah i cannot have r dot by 2 i cannot have any variation i cannot have r dot square i cannot have any arbitrary uh condition on the trajectory that x2 forms okay so to make it simple we refer to these such conditions as matching conditions that is because these trajectories have to somehow be consistent with the dynamics of the system so x2 design has to equal the derivative of x1 designing okay so the desired value of x1 has to um take the desired value of x1 you should get the desired value of x2 this is because it's dictated by the dynamics here all right great so what did we look at today so what we looked at today is we already had completed the analysis for trajectory tracking we were successful so of course we're very happy but we could not prove anything about parameter convergence so we sort of tried to leverage what we had learned in the last week to see under what conditions we can get parameter convergence and so we saw that if you have only functions of time appearing in the dynamics then you can use persistence of excitation and if you have functions of both state and time fxt then you have to use parameter dependent that is lambda u persistence excitation yeah and then we sort of gave the setup for the second order scalar system that's we are going to do adaptive control designed for in the subsequent lectures so this is where we stop today thank you for joining


A glorious story of New India is being scripted through the guidance of our ronorable Prime Minister Shri Mareendra Modi. Holistic connectivity infrastructure is being developed across India under PM Gatishakthi, National Master Plan. Next generation infrastructure or multimodal connectivity. And Tamil Nadu is also witnessing the transformation. Chennai, its capital city, has been experiencing rising air passenger traffic as it attracts tourists to its ancient temples, beaches, and world-class healthcare facilities. To cater to this, the government of India is utilising the land resourcefully at Chennai International Airport that is built on the least amount of land amongst six major metro airports in India. A new integrated terminal building constructed at a cost of 1,260 crores is being inaugurated today. As part of the airport modernisation project, old terminal T2 of 30,000 square meters area that had limited capacity is giving way to a more spacious multi-level building of 1,36,000 square meter area. This will take the airport's annual passenger handling capacity from 2.3 crore to 3 crore. In future, the airport capacity will soon rise to 3.5 crore passengers per annum. Over and above the existing capacity of Chennai airport, the spacious new terminal building will serve 3,200 passengers during peak hour with 100 additional check-in counters, 30 more frisking booths, 108 immigration counters and 22 elevators and escalators. The airport provides flyers with quick, hassle-free and comfortable experience. The new terminal takes design cues from rich Tamil culture while its roof is inspired from sari pleats of the Bharatanatyam dance costume. Its flooring showcases traditional kolam patterns, murals and wall arts installed here depict local art and culture. The modernised Chennai International Airport is adding another chapter to the inspirational saga of New India.