Thank you for being here I'm gonna talk today about The changes in the process that we established at GitLab on our way to continuous delivery But this talk is titled this way Only because it kind of fits within the track. I prefer the name Kubernetes the prequel Let me introduce myself first. I'm the engineering manager for the delivery team I've been with GitLab since September 2012 so that's seven years and I got hired as a backend engineer and through my tenure at GitLab I changed positions multiple times I was responsible for the omnibus package installation methods and so on and then recently I moved into the position of Managing a delivery team whose sole responsibility was figuring out how to migrate GitLab.com and all of our release management processes to Continuous delivery To just give you an a bit of an idea of what we are going to be talking today about I'm going to give you a history overview of how release management will evolve at GitLab How the whole team that I'm leading right now and the whole process got created and How we while we were doing things started changing things just to make it more interesting and finally the results of all of that and if there is anything I want you to get out of this talk is That there is no shame in trying things out Seeing how you can change your processes around how you can leverage your legacy tools Even though everyone screams. This is the best thing. You should be doing this. You should be doing that So to give you a bit of an idea of How we ended up in a place where we started from 2013 Onwards actually from 2011 GitLab has had Monthly release on every 22nd of the month. We haven't missed a beat in all these years From 2013 when we formed the company and had more than one engineer We had a rotating release manager role. This was an Engineer who was responsible this month for writing a blog post tagging a version pushing this out to the public even tweeting and in the first couple of years it was only three of us because we were at five people at that point and That meant that a lot of actions that we did were manual mostly because we didn't have the tools so I Would just log into a machine Build a package manually upload it to S3 manually Copy the Shah Put it on the blog post and release it. That was the release process as we started getting more and more of back-end engineers hired They started getting into that role as well. So What happens usually when you put an engineer on a problem? They find a way to get out of it. So what they did was Automated some of these tasks But that still meant that a lot of the tasks that we had were follow this documentation execute this script do this do that all of that manual because the release manager role Was actually a rotation. So all of the knowledge that gets built up during one month just gets wiped clean for the next one and The idea was also to make sure that we improve things by having fresh pairs of eyes looking at the process Now We actually had One near miss with our release. We almost missed the 22nd deadline that we had Back in December 2017 we deployed one day before the release which was unheard of I think that was where the alarm started going off that we lose a lot of knowledge by Rotating constantly and I got tasked mostly because I'm allowed month. I got asked with Investigating what happened? How would how can we improve this and as I was collecting data it became apparent that We actually do need to spend some effort in either formula team that is going to be Leading this change making sure that we don't have semi-automated things anymore, but automated or We are never gonna get better at this. So in July 2018 I got two engineers and At some point someone at the company said hey It's great that you're doing all this automation and kind of release goes well with kubernetes and kubernetes is this awesome thing So how about you just do kubernetes as well, right like it's easy They added two more engineers and said you're gonna be the delivery team So automate the release and migrate github.com to kubernetes github.com has millions of users has a lot of traffic and that actually is Task on its own. It's a huge challenge. So whenever you see someone screaming kubernetes I hope you're gonna remember this like this bill Burt fake comic So I accepted the task The team accepted the task. It was a great challenge So we set out to see what are our requirements for everything that we need to achieve So github.com live system. We can't have any downtime So everything we do we need to make sure that github.com stays up We cannot move the timelines 22nd remains the same Engineers need to release codes because this is our lifeline So no delays You should migrate github.com in the next three to six months to kubernetes Changing the whole platform So no time to do this great But one thing that actually stuck with me the most was the question is our Engineering organization ready for continuous delivery. It's great when you're using all the greatest tools, but How you use them is really really important So this was the biggest unknown for me like at least in the other three items I knew that I can't change that but can I change the last one? so What do you actually need to do to prepare your organization for continuous delivery? first of all your Development needs to completely shift left. That means that before things get merged into your main branch Things already need to pass all the testing all verification security checks Are you are your testing system solid? Do you have end-to-end integration tests? What do those tests tell you? How are you using data and metrics to inform your deployment decisions and Do you have capability to react quickly to any sort of change and Unfortunately for all of these things in 2018 the answer was no That was my face when I realized this was a humongous challenge not on the technical side, but on the process side so Now that I understand all of my requirements and the challenges that I'm going to encounter What is my team? spending their time on a lot pie charts because they don't tell you anything But this pie chart is Created out of the data we gathered over the 14-day period Where the development kind of slows down so that we can prepare for a release? My team spent 60% of their time in the 14-day period babysitting deploys and then 26% of our time was Related to manual tasks or semi-manual tasks that someone had to do writing a blog post or helping writing the blog blog post communicating the changes between people Doing various cherry picks for P1 problems that the developer found by the way if you Trust your developer today understand what the P1 means You are fooling yourself We also had a manual process where release managers had to do some basic QA that's kind of silly so Release manager goes in clicks on the button. Oh button works great. That's check done and get left also had a special thing where Community edition and enterprise editions were built from separate repositories And we had to merge one into another because enterprise edition was a super set of community edition So that took 10% of our time. So if you take a look at the whole thing in to in 14 days in two weeks we my team did nothing but Sit on the computer and watch well paint dry. Yes in this case so If we change the 80% of our whatever we were doing during this period We would be able to make sure that We have no release deal delays because we'll be freed up to make sure that everything happens in time if we do deploy quicker and Smaller chunks get deployed to production. We will ensure that there is no time or at least we would be able to control that better and If we free up all of that amount of time we would be able to actually start working on the Kubernetes migration that we were set out to do and Another thing that I thought was like a really great bonus here is While we are doing the changes here, we would be able to prepare the organization for the incoming changing process So That is what we set out to do If you take a look at the cycle time compression We set out to go there, but we started going down this route instead so one of the items that we observed is Everything was tied into this simple process How developers behave How the product behaves? We had a seventh of the month which was our feature freeze date at this point we would branch off from the mainline and We would have a slower moving branch from which we would do deploys and prepare release from This reinforced a really great behavior where developers would kind of pile around that seventh because I have time 70s in seven days and Then on the sixth in at midnight. They would panic merge things Because they know that if they miss this deadline, they have to wait for the next month But if they get it under this deadline, they have good two weeks to fix any problems that happen now We are creatures of habit, right? so I thought what if we and I didn't think of this by the way a lot of companies do this What if we speed this up like do the same thing, but just more frequently right like if it hurts So this is the same system But instead of doing one branch We create three every week. We create a new one developers get this Similar system of all right. Well, I have some time to fix things, but I don't have much time So I'm gonna think twice do I want to spend time panicking in like fixing things quickly? Or am I going to like make sure that things are actually operational before I merge things? It also gave us a bit more time to Make sure that Whatever we deploy this week We can be certain that by the end of that week The only new thing that is gonna be bring problems is whatever was created With this new branch that is going to go in we also had great help and that is We got to use the tool that we built so I Mentioned get love calm get love calm is one of the biggest instances of get lab in the world, but we use Get lab To build get lab and then we use another get lab to deploy get lab one Other thing that we had is an advantage was we had access to all of these developers that were working with us because If we don't get something that we need they won't be able to deploy their thing. So they have quite a lot of Excitement when we came to them and asked hey, can we improve this feature? How can we get this done better? So Some of the release toolstiles that you saw the 26% of the time we automated by just taking it into get lab CI Triggering things through the API and using the schedule pipelines. So if we need to create a branch It's set in the schedule pipeline. It triggers every Sunday evening Any P1 item that comes in we automatically cherry pick things into the branch that is currently active we create various issues to track progress through the environments or the q8 us that need to be executed and As I said get lab.com gets deployed from git lab. So we had to mirror some projects between instances and I think Another thing that is worth mentioning here is like get lab CI was the actual Pool to get or rather the glue to make sure that this all pulls in one direction that we wanted and finally get lab chat ops feels like underappreciated here or here, but a lot of the release tasks got automated just because we got a Very easy access to everything we had to do by using it to slack for example We connect get lab chat ops with slack everything is there. It's very convenient. You don't have to change your context So To explain a tiny bit more how this ended up looking So the happy developer as you can see on this side Goes through the whole process right review Making sure that their pipelines pass do some verifications through the review apps that we connected to a Kubernetes cluster and when they're absolutely sure they want to merge this thing they emerge it Usually that means it's out of their hands That's magnifying glass thing and the thing that you see scrolling here is our production pipeline what happens is What happened was we realized that all of the items we had to do were related to moving the semi-automated tasks into CI so developers machine is a machine and CI machine is a machine. So why not just have it there and it automatically logs things and the Release manager doesn't have to make sure that They're looking at the screen for six hours while the deploy is going now one challenge that we encountered here was that The tool we had at that point Was we already outgrew it so instead of continuing Using that tool we decided What are the top two things that we need to do to make sure that we can deploy safely? First get the package in Second thing make sure that it's deployed in a certain order. All right. Well, that's easy and we Rewrote the tool we had Rather we didn't rewrite it. We just wrote a new tool using Ansible and we placed our CI runner on a bastion host that had access to the infrastructure That was one of the bigger battles so to speak We had to do because we had to get a sign-up from security to put basically a remote code execution machine in our System We did get to do that mostly because we get a lot of insight in how we can actually make this happen and One of the great things that happened was we now got to connect through all of our environments and Do sequential checks as well. So what happens was Think like when the developers merge Whatever they did we automatically create a new package that package gets picked up by our CR system deployed on staging and we got to put Automated QA in CI as well if the automated QA passes it progresses to the rest of the environments That meant 60% of the time that we used is now out there It just happens. We don't have to do anything with it so the finish line When we enable the system Same 14-day period We did free up 82% of our time See to emerge also got automated and the 0.3% you see there is Sometimes the pipeline fails and we need to check it to see why it failed. That's it. Oops The release tasks remained relatively high still The biggest chunk of this 17% here is The security release we need to do security release requires a lot of coordinating There are a lot of stakeholders security teams development teams marketing and a lot of back porting to prior releases So that remains like a big chunk of our time in May 2019 We still had the old system and we had around seven deploys that month which was standard forgetlove.com in June we cautiously enabled things and we went to 12 I think and then I think I'm super proud about this August this year we had 35 deploys down on github.com that means more than one deploy a day and Did I mention that none of this is in Kubernetes? All of this is using our old legacy system But what happens with this is we bought ourselves time So my team has time to actually work on the migration But one of the biggest changes that happened was in the habits of the engineering organization People are thinking twice When they click that merge button, they make sure that integration tests are done before They click that merge button We have all of the developers on call because once we enable the system the old habits just got exposed and We had to call a lot of people to help us out with why are yous our performance going down? performance going down Why are we seeing the uptick in errors and within Two months of us enabling the system the whole engineering organization or rather the whole development organization went on call Sufficient to say I'm not really super popular there anymore but where I am popular or rather where my team is popular there is Everyone is grateful and everyone is excited that When an issue is found they can fix it really quickly deploy it really quickly and Within a couple of hours of finding a problem we can push this out to production, which is a huge huge change Why is this a Kubernetes the prequel? Well, because with us freeing up all of that time My team migrated one of the services that we had to Kubernetes So if you go to gitlab.com right now and try to do docker pull or docker push That is being served from a Kubernetes cluster We are using successfully our deploy boards We are using our monitoring successfully, and we are using our web terminals successfully because Obviously we had time to play a bit more than we would usually That's it from my side. I wanted to leave you with a bunch of links If you're interested in how all of these developed You can check out the design docs that we wrote you can check out where That pie chart you saw originally came from the gitlab 10 for release report And finally you can follow follow along our progress with our Kubernetes migration And what kind of challenges we ran into when we started doing the code Kubernetes migration Thank you Questions we got a few minutes for questions. So I'll bring you the mic if you want to ask anything So You're talking about how sort of like CI CD Enables cloud native, right? And you're talking about a lot of like Having to put the entire organization on call and that kind of thing, right? How do you get buy-in from Executives to make those kinds of changes So one thing that I think the executives love is dollar numbers So when they see how much time gets spent on busy work something that does not contribute to the actual Goal of the organization and if you transfer that into a dollar amount Expose how you can change that dollar amount to something way less Everyone starts listening Maybe they don't understand what we actually did here. Maybe they do it totally does not matter What matters to them is that a developer can fix a problem within a couple of hours instead of two weeks three weeks a month And then sort of to follow off of that question. Um, how do you How do you implement CI CD without having the engineering team be on call all the time Or is CI CD just a sinister ploy to extract more productivity out of the engineers So I think the the the on call was not made to make developers less productive The on call was developed to create some sympathy To with people who are actually managing the infrastructure and are on the forefront And the idea is not to have developers on call the idea is Teach them how to get themselves out of the on call rotation So think about what kind of problems at scale you need to think about and how to resolve them properly without Yeah, just merging randomly. So what I think is is happening already like within A two month period that that we had developers on call there are changes in habit And they're starting to ask the right questions How can I get access to the production database? To get the to understand the scale of the problem that I'm trying to resolve That is a great question to ask Because now you can provide them the data they can inform their decisions or other they can Understand how to fix problem on that scale and they're not going to get paged or their colleague is not going to get paged And the organization is getting better with it and I really do believe that With time we are going to remove the need for developers being on call Even if they are on call, they're not going to get paged And I think that's a great success This story at this point it sounds like you're still releasing once a month Yes, correct. Is are there any plans to increase the amount of times you release? eventually So there is a difference between what we do on github.com and what we do for self-managed release We already hear from well all of you that Having a release one once a month is great, but our organization can't update that quickly so We don't necessarily want to change that cadence, but we still use the same tools to To release to self-managed as well. So what is actually happening right now is We are getting tried and tested product Earlier and once we actually say this is 11.3 release What customer is going to get is something that already ran I wouldn't say bug free because that's not possible But at least definitely like not as many bugs as uh as you usually have in these type of processes. So The system enabled us to ship faster to github.com with more confidence And that actually allows us to make sure that we are not going to have more releases for our self-managed customers But actually less because we don't have to create more patch releases and we get to focus on Doing the new features that we want to ship


Welcome back to Kids Fun Science. My name is Ken. Today's experiment is the Smoke Waterfall. As always, adult supervision is required. What you need for this experiment is a lighter, sticky notes, rocks, fish tank, and an L bracket, and a cup. Start off with your sticky note with the sticky part facing up. Roll towards that sticky part until it gets to the end and then it should just stick right onto the end of the roll. You want to make it a little bit smaller than a pencil so you want to be able to fit into the whole of the bracket as you see here. And then you light it with your lighter and the smoke will come down the sticky note and pour down like a waterfall as you saw there in the cup. Here I'm in a fish tank because when there's air or movement in the air, it makes it harder for the waterfall smoke to go straight down as the current will move the smoke everywhere. By having it in a contained fish tank as I do here, you can see the smoke is going completely down the rock waterfall and all the way to the base of the fish tank. So the science behind why doesn't the smoke rise, right? When the paper burns, it breaks down to millions of gases and water vapors. Normally, these molecules are heavier than air. The flame excites these molecules making them lighter than air, which makes the rising smoke you see in a normal fire rise up. So in this experiment, all the hot smoke molecules are being trapped inside the paper tube, which prevents them from rising. The trapped smoke molecules are giving time to cool off, which releases the energy that allows them to be lighter than air. The cooler smoke molecules fall instead of rise and escape out the paper tube and creates a smoke waterfall effect, which is pretty cool. I hope you enjoyed this experiment and thanks for watching. So guys, if you like these videos, there's some right there.


That's very powerful inside. I just walked inside into the space where all the brahmacharis are chanting. There's something special about their chants. It kind of hits you right here. It's kind of energizing. So what is it? What's a brahmachari? I mean, we understand it in Hinduism. Brahmachari means denial, and you often said it's not about denial. It's about gaining something without denying it. So tell me about it. And I have to say, they're both men and women in there. They're not like monks. All the women have been put up in a nunnery and the monks are in some monastery. They're all there. For example, somebody is climbing out to climb Mount Everest. He prepares for years, probably for a lifetime he prepares. And then he's in the harshest of climate, denying himself every pleasure of life, of comfort, of family, of society. Everything that normally people want to have, he denies himself all that to climb the mountain. But do you really believe he's denying himself or is he making another possibility which you can never imagine real in his life? So that's brahmacharya. The word brahmacharya literally means, Brahman means the ultimate. Tharya means the path. You are on the path of the ultimate. Or in other words, you are somebody who is not willing to settle for small things. So anybody who has created anything worthwhile in their lives have naturally denied themselves a lot of simple pleasures other people are having. When you're making a film, maybe you're not eating well, maybe you're not sleeping well, are you taking your afternoon nap, sitting with your wife, enjoying the sea stir, having dinner with them, with your family, are you doing all this?  Isn't that true with every man or woman who has created something worthwhile in their life? The same with brahmacharya, they are on the path of the ultimate. Because of that, their attention is not diverted to other things that people think are valuable. So what's the ultimate? See here, if you look around, there's brick and stone. Or rather mud and stone and trees and plants. This is creation. But now if you look at this, from the same soil this tree is coming, throwing out, right now there are no flowers but it throws out wonderful fragrant flowers. From the same soil a mango tree will come and throw out mangoes. From the same soil you have come and making movies. From the same soil I have come and I'm a guru. So there is something here which can make this mud into a tree, into a fruit, into a filmmaker, into a guru and into a billion other things. Don't you want to know what it is? Don't you want to experience what it is? And if it was somewhere in heaven, I'm not interested.  There is something here which can transform a rice grain or a banana or a piece of bread into a human being. This is the ultimate source of creation. Don't you want to know what it is? Don't you want to experience? Don't you want to know the power of it, the beauty of it, the joy of it? So these are people who have had a taste of that and now they want all of it. They don't care whether they eat or not.  They have had a taste of the ultimate. Now they want it all. They won't settle for anything less. Okay, now I understand that till you've experienced it, it's an experiential sensation or more than a sensation. Talk about the ultimate.  You've had a taste, now you can't stop. So your work is to drive people mad. Yes. Right, okay. Because if people slip through their life in the name of comfort, in the name of security, they're wasting their life. They must go all the way. This is not my desire. It's their desire also. Because today you have a desire to fulfill this. If that happens, you want the next one and the next one and the next one. If you carefully look at your desiring process itself, you will see you're not willing to settle for anything limited, isn't it? There is something within you which doesn't like boundaries. There is something within you which is longing for the ultimate always. So this is not my idea. This is not my philosophy that you should seek ultimate. You're anyway seeking ultimate in unconscious ways. I am saying if you seek in unconscious ways, you'll always remain frustrated and the chances of finding it are remote. It's better to seek what you want in a conscious way, in a focused way. So brahmacharis are those people, they're also seeking just what you're seeking. But they're seeking it 100% consciously. So what is the ultimate? Is there other words? I mean I'm looking for words. What is ultimate? See whatever we look at, we can only look at the way you understand it right now, okay? If I say something which is not in your understanding, now I'll be compelling you to believe me or disbelieve me. I don't want to go into obscenities. I think it's obscene to demand that you must believe me. So right now in your life, do you see that when you stand here, what you call as myself, actually you gathered over a period of time? In terms of your mind, the huge heap of impressions in terms of your body, it is a heap of food that you've eaten. So you gathered this. What you gather can be yours. Can it ever be you? No. Then what is the you beyond yours? What's the… So you're existing here without even knowing what is me. So this me, right now we will go this way. This me is the ultimate. This me is not a simple me. This me can make a, you know, a piece of bread into a human being, a banana into a human being. This is not a simple thing. This same me which can make a banana into a human being, also created the whole cosmic scape. It's not a different me. So the choice is just this. Either you can exist here just as a piece of creation or you can exist here as a very creator himself. That's the ultimate for now. There's more to it, but for now. So why does everybody expect a guru to perform miracles? Yeah, I mean this is the whole thing. At one point if I go and say I met Sadhguru and I had this amazing experience. You must tell them he is a real miracle because he eats a banana and he makes it into a human being. You must tell them what better miracle do you want? See, they have not understood the magic of eating a banana or anything for that matter. So what you're saying is everything is a miracle in any case. See, either, either there are two ways of looking at life. If you have eyes, if you look at it closely, everything is miraculous because there's not one single atom in the existence that you can fully explain to it's miraculous. Another way is everything is happening, the whole physical existence is happening between cause and effect. Everything can be explained if your perception is deeper. Either nothing is a miracle or everything is a miracle. In your mind if one thing is a miracle and another thing is not, you are in ignorance. Yeah. So even miracles are intellectual ideas that are boxed, separated. See, anything that you cannot understand is miraculous for you. If you did not understand how electricity works and if I just touched that wall and light came on, you would think either I am the messenger or the sun or himself. See, right now I can take you there and show a great miracle for you. You will see mercury solidified. Mercury is to be liquid. According to modern science, you can solidify mercury only at minus 32 degree centigrade. I'll show you mercury, which is solid. You can measure it weight to volume. You will see it's pure mercury and no other substance on the planet which can weigh as much as mercury so you can't fake it. You cannot fake it. You will see just a 23 inch tall ellipsoid which weighs 682 kgs. Nothing else on the planet can weigh that much unless I imported something from somewhere else, which I have not. This is just pure plain mercury. But it's solid in room temperature. Is that miracle enough? But it's not a miracle. I can't explain the science of it. Now, I know this is not what it's about but for us mere mortals, it's very fascinating sometimes when I hear that in the Brahmachari way of life, just the physical abilities, the energy levels that these people have, all the Brahmacharis have and what I've heard about, there's that mountain right there and which would probably take me what, six, seven hours? No, no, no. You could do it in three, three and a half hours. They'll do it in 35, 40 minutes. Because I've seen them. You know, while I've been here, I've seen some of them other than the volunteers and the meditators but I've seen the Brahmacharis, they never sleep. When I go to bed, when I'm forced into bed because I stay awake a lot at night, they're still awake. And then when I come in the morning, they're like as fresh as anything, teaching us yoga, working, walking, working, working everywhere. And then I try to play football with them and they're the most energetic football players. I mean, okay, so they're younger than me but I've seen football players, the kind of energy they exhibit without sleep, without rest in this Brahmachari. And then I had, you know, I saw one of them eat and I said, what are you eating? And he said, well, you know, it was just soup and a few grains and that's all they were eating. So tell me, what is it? Tell me about that energy that comes from this way of life, this the way of lifting your energies. What does it do to you physically? See, the energy that you have right now is not all about the food that you eat. Sunlight, air and water you drink is actually contributing for more than sixty percent of the energy that you have. I think there is a whole misunderstanding. I know if I say this, it would be very, very controversial because humanity has invested so much in the other direction. We are always thinking of energy as nourishment, food nourishment.  And every day there's something new. I'm not saying that's not needed. Definitely that's the chemical ingredient of the body it needs to be taken care of. I'm not trying to belittle the nourishment. Nourishment is definitely needed. But that is not going to determine the energy levels that you have.   You're always thinking about being energy conscious, isn't it? Now you are driving an ambassador which was giving you four kilometers per litre. It was okay. Now it is a crime. Because you understand this is not the way to live. Now today you want 40 kilometers out of a litre. So it's a question of energy efficiency. So a brahmachari is into various types of sadhana. Certain internal maintenance of the system, the way it is managed is nothing is wasted. It is very eco-friendly.  You don't sleep, you don't eat. It doesn't matter. If I don't sleep for three nights, you will still not see any difference in me I'll be carrying on. This is not some superhuman. See, first of all we need to understand this. This is not about being superhuman. This is about realizing that being human is super. Being human is not a simple thing. It's a tremendous thing. So right now all the symbolism you see Shiva always has snake on top of his head. This is to indicate, see, snake is seen as the fundamental energy. Kundalini is symbolized as a snake in India, in the yogic systems. So tell us why, because in the West, snake is something threatening and almost evil. It's because it has this kind of sometimes sexual connotation, sometimes evil connotation. The first aspect is, the snake, Shiva is not holding at his feet on top of his head. This is to indicate that his energies have reached the peak. Okay? That is the essence of asceticism, that his energies have hit the peak. Energies have hit the peak means his perception has also hit the ultimate. That is why the snake on top of the head and the third eye are symbolic.  Third eye means your perception has hit the highest level of perception. Everything that can be seen, you have seen in the existence. That is mainly because you are energized in the highest possible way. If your perception has to evolve and enhance itself, the most important thing is your energy has to evolve and enhance itself. So the whole sadhana that the brahmacharis are doing and all the yoga practitioners are doing is to evolve, refine their energies in such a way that their perception is enhanced and the third eye opens, third eye opening does not mean you will have a crack in your forehead. It just means that your perception has gone beyond the dualities of life. You are able to see life just the way it is. Not the way it is necessary for your survival. You are able to see it just the way it is. So about the snake, see, even according to the Christian, whatever, what do you call the theology, Adam and Eve, first man and first woman were there. That is the story.  They didn't know what to do with each other. All right? So the snake came and made them eat the fruit of knowledge.  Tell me. Yeah, I would say that's divine. And by eating this fruit of knowledge, life got initiated on this planet. The whole life process became possible only because of the snake tempting Eve to eat the fruit of knowledge and share it with Adam. Only one who is out and out against the fundamental processes of life can and will say snake is agent of the devil. One who initiates life has to be agent of the divine, isn't it? So here we always saw the snake as the agent of the divine. There is no temple in India without a snake. Somewhere there will be a snake to indicate this is the agent of the divine. And above all, Shiva keeps it on top of his head, not at his feet, not somewhere. A crawly creature on top of his head because to indicate that this is what initiated the life process. Religion and spirituality, you keep saying it's two separate things. Religion is a... See, we do not know what's religion in this part of the world. We have no religion here because here in India, for example, there are 36 million gods and goddesses, okay? As if that's not enough. See, now I'm creating a new one. Everybody's excited. Sadhguru is going to create a new goddess. He's going to give birth to a new goddess. Everybody's excited. Any other part of the world, they would have killed me. Isn't it? The new goddess. That's the whole thing but here, we clearly understood God is our making. Everywhere else, they propagated you are God's making. No. The way you think about God, the way you project God is essentially our making, isn't it? Yes, there is a fundamental force but all other aspects are our making, isn't it? So we made it in every possible way we like it. 36 million, I feel is not enough. I feel every human being should make his own God the way he can relate to. The problem is people want to organize this and run a huge setup of power. So, talking about one and rallying the whole world behind that one God has become an important thing. If all of us had our own gods made by us as we can relate to, you could never organize it. Yeah. So God becomes your fingerprint. Each one different. Yes. And above all, see in this culture we never held God as the highest. God is only a stepping stone. He's only a tool for you to reach your mukti or your ultimate liberation. Freedom is the highest goal, not God. Can we sit here for a while? Because you have more energy than I do. Okay. So, Sadhguru, you said this many times. The highest goal is freedom. Now, people understand freedom in different ways. The two words, the three words that you used, liberation, so liberation from what? Freedom. So, everybody, as you yourself said, it's contextualization. Freedom from what? And when you say ultimate freedom, so does that mean that you get into freedom that has no context? So, therefore, it becomes ultimate to use the word duality. I know these are… these… I'm struggling. I kind of sense them, but I cannot see them clearly. So, I want you to talk about that, these words. See, generally the word freedom before 1947 in this country, if you said freedom, people only thought if the British go away, we are free. That was our idea of freedom at that time. I'm sure those last couple of decades, Azadi just meant English should go away. Yeah. So, English went away. Now, are we free? Not really. Politically free, yes. But in no other way we are really free. So, freedom may mean different things to different people depending upon the context of their individual life at a certain moment. But essentially, do we understand that we are living within certain limitations of our own life? Leave the political, economic and other aspects of freedom.   But do you also see there is something constantly longing within you to be little more than what you are? So, you're seeking freedom. But you're seeking it in installments. So, if I bind you, if I put you, if I imprison you in a small cell, five by five, let's say, you will fight for your freedom. If I release you into ten by ten cell, you will feel wonderful. But within 24 hours you realize this is not it, this is not freedom. If I release you into larger compound and say you are free, do what you want here but you can't go out. Once again, the moment you feel the boundaries you want to break that. So, however big the boundary I set for you, the moment you are able to feel the boundary. When you don't feel the boundaries, you think you're free. But the moment you feel the boundaries, you want to break it. So, there is something within you which cannot stand boundaries. Or in other words, there is something within you which wants to become boundless. What can be boundless if you look at it? All physical forms must have a boundary, isn't it? The boundary, a defined boundary is the basis of a physical form. A physical form can be small or big but can never be boundless. So, there is something within you which is longing to touch a dimension which is beyond the physical. So, there is something within you which is longing to be spiritual. So, spiritual does not mean looking up or looking down or praying or going to the temple. Spiritual essentially means you are in an experience which is beyond the limitations of the physical. That means you're in a boundless space. This longing is always there. In every human being either you're approaching it in installments or you're going consciously for the ultimate. So, why in my mind is the ultimate so blurred? Why can't… I mean, I can imagine, you talk about cage and you talk… the moment you talk about measurement, I can see it. You say you're imprisoned by this galaxy. I can somehow say, okay, it's a galaxy. Now you talk about being imprisoned by nothing. This concept of being boundlessly free sounds brilliant. So, just talk to me about it. How do I aspire to something that I'm unable to imagine? That's why you never aspire to something that you're unable to imagine that you do not know. If you want to become free, you never think of freedom because freedom is just an idea. It doesn't exist. It is just an idea which has no substance in it. When you think of freedom, you only look at the ropes which are binding you. You cut these ropes which are binding you. There is some sense of freedom. Then you realize there's another set of ropes binding you. You cut that. There is freedom. But then you notice there's another set. It doesn't matter how many sets of ropes are binding you.  There may be… the number may be big but still it cannot be innumerable, isn't it? It cannot be infinite. Only freedom can be infinite. Bondage has to be limited. The very nature of bondage is limited, isn't it? Limiting is bondage. So bondage cannot be limitless. Only freedom can be limitless. This is not all semantics and ideas. In terms of life's experience, you don't waste your time talking about or thinking about freedom or the ultimate. You look at what's binding you now and see how to transcend this. This is growth. And what… what actually is… So I'm looking at myself. What's my most limiting factor? I mean, that you could probably say most people. That's what they're bound by there. Let's say, I mean, I am a filmmaker. I can go anywhere in the world. I can travel. I can do anything. I have, you know, I… Yes, but if I… if I make you walk around for one hour, you'll have to rest. You have to go to sleep. That's a limitation. How would you… What kind of a filmmaker would you be? If you could be like me, stay awake for five days as they stretch, don't bother about eating, sleeping, nothing. You can just focus on what you have to do. What kind of a filmmaker would you be? Maybe you would have all the actors dead.  Yeah, well, I've been accused of that. But then my film becomes my limitation. So, boundless… because I get very passionate about filmmaking and people eat. But then it… at one point it becomes a film and the film is over. This idea of complete…  We're just talking about tangible things so that it's easier to, you know, see it.  Suppose you could just be on, hundred percent focused for three days at a stretch. What would be the quality of your film? I want you to imagine. It would be absolutely brilliant. It would be something else than what you're doing right now, isn't it? Yeah. So why is that possibility not being explored for now? So if you explore that, then you will see that's fabulous, but there's still other limitations. But those limitations are not relevant now because we are still in these limitations. Yeah. Okay, so my body is my own limitation. That's what you're saying? My mind is a limitation. There are also possibilities. Your body is your possibility. Sadguru, what I'm asking you, I know you're not saying it because you think it sounds like madness, but let's go into madness. The Promised Land, you know, let's talk about it. Because we've talked about this, we've talked about in the context of Shiva, we've talked about in the context of absolute freedom, we've talked just in terms of absolute, we've talked about you have to overcome duality. What's there? What's there? I know it's difficult, but if I was going to convince my friends to follow this path… It's not difficult, it could be… In your present context, it could be scary. Okay, it could be scary. For example, when we say Shiva, most people, everybody can look at Shiva the way they want it. That's why all these many forms, Shiva is a Bholenath, an innocent child, you can play with him whichever way you want.  Why all these things is that everybody can relate to it as they want. But essentially what the word Shiva means is that which is not, that which is creation, that which is not is Shiva. So what is it that which is not? Why are we looking at that which is not? Well, that which is not is the basis of creation. It is it is the empty space of the existence which is the womb of creation, isn't it? So we say Shiva is the basis of everything. This is just the ultimate physical sciences. Let me not use the word ultimate. The highest level of physical sciences expressed in a beautiful dialectical format. So Shiva is that which is not, that which is still and that is not. Out of that pops out creation because he came in touch with a certain energy which we are referring to a Shakti. Whenever Shakti just passes by boom, he roars.  We call it he rises as Rudra. He roars and he creates his first form which is an ellipsoid. From that springs all creation. Is there one hundred percent in agreement with this today? You're saying it's a dark energy. Shiva has always been referred to as the dark one. So what you're referring to as Shiva is just that which is not. We are not talking about a god who is sitting up there. But why the form? The form is because the first yogi is also called Shiva because he realized Shiva. What is perceived is one who perceives it. We don't see any difference between the two. So when we say Shiva one moment we'll be talking about the first yogi the Adiyogi whom we worship for all the knowledge he's given us and that aspect of Shiva which is limitless, boundless, nothingness we are seamlessly moving from this to that because that is what a dialectical culture is. That's how life is. Life is constantly flowing from logic to illogic masculine to feminine, feminine to masculine. People who think their lives out they're always in conflict they're not able to figure what's the nature of life they're not able to experience life because they're trying to stick on to one thing. This is like you stand on the right bank of the river and you think the left bank is wrong. If you try to eliminate it you will eliminate the river and the right bank. Life is happening not separately. It is all happening in you know it's all entangled in each other entangled is not the word it's intermingled with everything and everything. So the whole culture evolved like that we will be uttering Shiva in this sentence if I say Shiva I'm talking about the boundless nothingness. Next sentence I say Shiva I'm saying yogi it's for you to figure what I'm talking because if I go on explaining this and that then I'll be separating the two consciousness. In my perception the one who perceives and what is perceived as not different he becomes a yogi the word yoga means union he becomes a yogi because he's experienced the union of the existence he's become one with the existence so we don't refer to the existence the basis of the existence and the one who is known the union of being with existence as two separate identities though they are two. So one possibility and as you say it's just one possibility is that the ultimate is nothingness. It is. What Buddha describes as shunita is that the same thing? See Gautama goes about using negative terminology because the moment you use positive terminology people will fly off into imagination wild imagination will happen if you say God people will imagine a man sitting there sitting in his lap and all wonderful things happening to you all the nonsense. So Gautama very consciously is constantly using negative terminology. He says shunya, he says nirvana he says anatma these are all his words anitya, anatma these are all complete negative terminology the western cultures use positive terminology because they are so terrified of the negative but the positive means nothing without the negative isn't it? So the yogic culture uses both of them together seamlessly we move from this to that, that to this Shiva is the most wonderful man at the same time he is the most horrible man he is an ascetic but he is a great householder he is a drunkard, he is the peak of alertness he is intoxicated, he is always intoxicated but he is the peak of alertness because this is the way existence is it is neither this nor that both of them are in play all the time so in the yogic culture we use both positive terminology and negative terminology Buddha used only negative terminology because he is afraid that if you use positive terminology people will fly into wild imagination which is which is a wise thing in a way this is a question I am asking on behalf of people who are unable to speak to you directly okay the economic collapse has taken place the world has gone into a tizzy I have just lost my job I have a mortgage that I can't pay I have three children that I can't educate I have a wife who is saying what is going to happen to our life and my pension has all disappeared as I thought that in future I am going to get this pension my beautiful life if this is United States my American way of life has just collapsed and now you are talking about contradictions you are talking about nothingness you asked for it now I am asking you what would this mean to me when I am so confused and so my everyday existence is how am I going to feed and still have my house and feed my children what would this do for me why would I go this way today we as a generation of people never before the survival process of a human being was as well organized as it is today if you have the money if you go to the super store you can buy everything that you need for next one year and not step out of your house it's that organized never before was this possible everyday you had to go out get your food for that day not knowing where the food is it is too organized to a point that we are becoming lethargic we are becoming unrealistic with the life processes so the economic collapse I don't I think it's improper to call it a collapse the economic reality from the lie of a bubble we have come down to reality okay this is a this is Chinese wisdom okay I'll tell you so some Americans were complaining about the economic collapse and what's happening so a Chinese man took him and took a chair put it in front of a mirror see do you see there are two chairs said yes then he removed the mirror what is left only one chair so this is what you're complaining about always there was only one chair if I put up two mirrors these two one chair will become ten chairs if I become five mirrors they'll become hundred chairs but that's not the point the real thing is there's only one chair there's only one planet there's only that much resource there's only this much rest is all exaggerated nonsense okay I don't think we should wait for it to collapse if we are wise we must pull it down ourselves I was being interviewed by one of the national channels just before the economic collapse when India share market was touching 21,000 something and that day I was in Mumbai they wanted to interview me they said what do you think about this I said it's unwise to have share market running at 21,000 I also keep general you know attention towards what happening in the economic spheres if you ask me India share market should be at 14,000 but we would like to make ourselves little bigger than who we are to attract foreign investment this then so if it's running at 16,000 16,500 sufficient exaggeration to impress the outsider but if you run to 2021 it is not good and this is like you want to go to an interview today you are only five feet eight you want to wear two inches heels and go it's all right you want to look little taller than who you are suppose you wear three feet heels and go you run stills very soon you are going to fall on your face they are in like it okay and it collapsed within about twenty days after I spoke it just dropped to 11,000 whatever 9,000 I think so you are walking on stills you are bound to fall little bit of exaggeration human beings want to do to impress each other it's all right two inches three inches heels all right if you go for too much it's not fair to yourself or to the people around you isn't it but the higher goals that you are talking about let's just talk about that in the context of the individual experience that human being is something the human being was caught in all the mechanics of economies we can say to him or her that you can solve that before you even think of higher goals spirituality all we are saying spirituality and higher goals will actually help you and he will say how let's look at that see right now what you are talking about is certain upheavals in your physical situation one thing you need to understand is do what you want even if you are a super human being outside situations will never happen one hundred percent the way you want them never ever even if there are just two people in the family still it doesn't happen one hundred percent the way you want it isn't it and not a single person in this whole planet has ever happened one hundred percent the way you want them not your spouse, not your children, not your boss nobody so outside is always going to be like this some things happen your way some things don't happen your way but the inner reality of your experience is hundred percent you is only one ingredient that's yourself at least this must happen the way you want it isn't it so if this was happening in a peaceful and blissful way within you economic high or economic collapse wouldn't mean anything to you you just have to adjust a few things around you that's all it doesn't change the quality of your life as such because you're anyway blissed out always you're blissful by your own nature if there is money you will drive there's no money you will cycle which is more healthy I mean you will do whatever the situation demands without any great struggle about it right now because you've not fixed the interior you are trying to fix your interiority by fixing the world around you which is juggling even when things were going fine I'm sure you were still stressed when you have a job you complain of stress so shall I get you fired will you be happy will you walk on the beach joyfully no now you will be depressed because you are fired so whichever way you suffer this is not because of the situation because you have not learned how to conduct this human mechanism how to keep this being that you have not paid attention to nothing happens your way in the world that's not the problem the problem is you yourself are not happening you want yourself to be if you did happen the way you want yourself to be for sure you would keep yourself blissful and wonderful isn't it outside would be just a play you conduct it to the extent you can conduct it never before another generation of people have known these kind of comforts and conveniences you are living better than your father your grandfather anybody unless your grandfather was a Maharaj or something you are living better than your father and grandfather but you are still complaining because it's not happening the way you think it should happen so if it happens any better than the way it is happening right now your economy you will have no planet left I am glad economy has collapsed because ecology will improve when economy collapses


Hello everyone. Today we are going to see document preprocessing or creation of the logical view of the documents required in information retrieval. So, this lecture has been split into two parts. So, today we will see in this lecture we will see two steps of this and then in the next lecture the remaining steps. So, learning outcome for this session is students will be able to preprocess a document and generate the set of index terms. So, let us understand what is information retrieval. So, information retrieval deals with the representation, storage, organization and access to that information item. So, user will be given an interface where user can enter the query and can access this particular information or the collection or the documents and so on in which she or she is interested. So, how to do this process or what is the background required that we are required document preprocessing. So, documents in the collection are represented using the set of index terms or keywords. So, these keywords can be extracted from the documents automatically or by the human subjects, but currently you know that the collection is too huge or the database is too large we cannot do it manually of course, it is going to be done automatically. So, whatever may be the process what it is providing us is the logical view of the document. So, here in document processing now which will be used as an index terms not all the words are equally significant for representing the semantics of the document. So, generally these are the noun words or the group of noun words which are giving you the most which are used as a representative of the document content. So, preprocessing the text of the documents in a collection is nothing, but to determine the terms to be used as an index terms. So, let us understand the steps. So, here which words will be used. So, if you are taking the set of all the words in the collection to index documents there will be too much noise. So, what we have to do is that we have to control the size of the vocabulary. So, what is vocabulary of course, this is the collection of the index terms ok. So, it is the preprocessing is nothing but can be viewed as in control the size of the vocabulary, but what happens generally while controlling the size of the vocabulary some user knows that this is the keyword or these are the documents which contains this particular information and when this keyword has been given as an query user is expecting that documents to be retrieved, but these documents sometimes are not retrieved why this is so, because the to control the size of the vocabulary search engines add some indexing steps or the additional steps in indexing. So, due to which these documents may be removed or there is index terms may be removed and that is why user is surprised with the result. And now it is therefore, most of the search engines are using all the index terms all the words in a text as an index term. So, that user will get the relevant result. Now, this preprocessing is having 5 steps. So, lexical analysis elimination of the stop words, steaming, identification or the selection of the index terms and then the construction of the structure such as thesaurus. So, in this lecture we are going to concentrate on this first two terms or steps that is lexical analysis which is nothing, but separating the words and then treating this digits, hypers punctuation marks and case of letters. So, specially over there and then the elimination of the stop words where we are going to filter out the words with very low discrimination values for retrieval purpose. Steaming is nothing, but remaining words or the words which we are after removing the affixes are going to be taken as an index terms. Then of course, the selection of index terms or noun words are frequently carry more semantics than the adjectives, adverbs, verbs and articles, prepositions so on and add then construction of the thesaurus. So, this is how the logical view is created or the set of index terms are created. So, document is containing text and structure. So, for the text retrieval model we are interested in the text whereas, in structured retrieval model where we are specifying structure also in the query structure is going to be recognized. So, here output will be structure and then text will be preprocessed. So, if you can see at any step the engine or the algorithm can decide which words has to be used as an index term. So, here after removing the spaces accent what is remain is taken all together as an index term this is called as a full text index terms or some of them can go up for removing the stop words then grouping it into then or make the group the noun groups after that do the steaming and then give the automatic or manual indexing. So, any step can be skipped and taken this index terms will be generated or created. But also let us understand how this lexical analysis will be done. So, lexical analysis is the process where the conversion the stream of characters into the stream of words ok. So, identify the separators or the recognition of the separators. So, generally the space is the most recognized separator. So, space commas can be used or the hyphens can also be used for separating the words over there and get the words as an index term. So, the most the major objective of this analysis is the identification of the word in the text over there. So, but here sometimes we need to consider the special cases like digits hyphens punctuation marks and case of letters to separate out the words. Let us understand one by one. So, generally numbers are not good index terms without tsunami context they are inherently vague. Suppose I have given a query select the data from in between the year of 2020 to 2021. So, what I will do is that 2020 and 2021 will be taken as an index term for searching, but the data which we I am interested in may be the since we have used 2020 and 2021 as an index term I may get the other data also ok. So, this is very how we should save, but it is not the case that we can skip the numbers we cannot save it. Because for example, sometimes I want to use a credit card number, mobile number, pan card number and so on for retrieval purpose. And then if I am not storing them as an index term then I will not be able to retrieve the data. So, this is one case. So, there are advanced lexical analysis procedures which performs the date and number normalization to uniform the format. Second case is the hyphen. So, as we have discussed hyphen can be removed and then their words will be produced. So, in the state of art this hyphens will be removed then there will not be any difference, but consider that the mother-in-law. If we will remove the hyphens it will lose the context or b dash 49 if this is a specific number of an item then what will happen or if we are. So, we have to decide whether at which point we should consider the hyphens as a separator or hyphens as a part of the work. Same is the case with the dots or the punctuation marks. So, we are going to treat it as a separators in sometimes this punctuation marks or dot is going to be the part of that work. So, we should not remove it. Consider that we are using that we are searching for the programs if it is in Java program or C program dots will be used. So, if we are going to separate out it will lose the context. So, that is also going to be decided whether to keep it or whether to remove it. Now, generally what happens when we are separating the words it will be either converted into the lower case or the upper case. But consider if you are using that Unix commands then it is going to be the mixture of that it will be in the proper case or it will be the mixture of the upper and lower case over there. Then converting into the one case will not serve our purpose or if we are considering this bank and consider this example. So, capital B means generally talks about the organization and when we are talking about the small letters we talk about the river bank ok. So, converting all the time into the upper case or lower case will not serve our problem. But there is no fixed rules that this should be considered at this moment only there are many limitations. So, there is no fixed algorithm given to this lexical analysis based on the application we have to decide. So, at this moment pause the video and what we have learned lexical analysis based on that try to get the set of words. So, you can decide your separators and then you can write down the set of words. So, here in this I have treated the separated separators as in comma spaces and then these are the words which has been generated. So, data retrieval consists mainly of determined and so on. All the words has been generated as a tokens or a separators. So, if you are using any language for implementation you can use the tokenizer class or you can use the NATK for processing it and you can generate these words. So, once these words has been generated the next step is going to be elimination of the stop words. So, which words will be considered as a stop words? So, words which are too frequent among the documents in the collection are not good discriminators. So, where which occurs in the 80 percent of the documents in the collection is useless for the purpose of retrieval. Why this is so? Because all the documents are going to be retrieved and that will not be giving you the fixed document or the documents in which we are interested in. So, such words are referred to as stop words and these are normally filter out as in potential index terms. Generally articles, prepositions, conjunctions are the natural candidates of this list of stop words. So, here of course, when you are removing all these words the words will get reduced. Let us see with the example also. So, it will provide the compression of the text. So, this can be this can be extended to as it is written article preposition and conjunction over there. But what happens now consider that queries to be or not to be in this if I am going to remove this all the stop words what will remain? Only the be. So, it is not going to serve our purpose. So, this is the one reason why this all the search engines are adopting full text index so that most relevant documents will be given to the user. So, this is what we have seen in the lexical analysis. This was the generated tokens after removing these prepositions articles and so on. These are the remaining terms which we have going to use for the further process. So, this is what we have seen as the first two steps. Next three steps will be seen in the next part of the lecture. Thank you.


So, well, thank you for coming. Hola, everyone. Did you notice that there are free talks at the same time we were distributed in name? The Arup-item folks have some sense of humor, for sure. Well, first of all, I would like to introduce myself, as it was already did. So I'm a software developer and a maintenance guy. It's focused telecom Poland. My main toys are G event, Twisted and Celery. What we are doing at the company is to providing a business-to-business telecommunication solutions. Right? We host teleconferences. We sell Pepe Xs, but our main product is a contact center. So nowadays, you have multiple ways to communicate with each other. There are social media. There are SMS text messages. Well, there are emails and there are phone calls. So a contact center is a way to unify all these, to gain a much better contact with your customer. So, throughout this talk, I will show you the principles of implementing distributed logs in Redis using Python and show the usage in a real-life application. So does anyone know or does anybody doesn't know what Redis is? Okay, there are some hands, so I will quickly refer to that. The Redis is an OSQL database. It's basic key value storage, but the key can be only a string, but the value can be more complicated data structure. For the sake of this talk, most important are string and list. They are pretty much like Python counterparts. And also Redis has a simple support for a message passing. So first of all, I will introduce some case study. Then I will show you how to implement correctly distributed binary semaphore, so-called mutex, and later on semaphore. And finally, I will give you some tips and there will be some time for questions. So first of all, this is a loose definition that I came up with a few days ago. The Gido on his talk yesterday gave a great example of locking and keeping safe the reference counters. The other case would be that we have a short database connection between threads in Python, and we would not like to other thread sending some SQL comments while the other thread was trying to carry out the transaction. So this is basically what lock is, it's to protect a shared resource. So the case study will concern a contact center. Hope nobody expected that. So in the contact center, work people, of course, and we refer to them as agents. So an agent can answer or originate a phone call, email messages, text messages, or carry out chat conversations, and this kind of stuff. Well, and my advantage of contact center is that it automates all of this kind of work. So it keeps the agents busy. So they are not looking at the floor or sailing, okay. But to be more familiar, I have replaced agent with giant W that stands for a worker, and we are tasks. So you could say that this looks like a task queue. Who doesn't know any task queue solutions like Celery, RQ, Germans? Are there anyone who hasn't heard any one of them? No, it's good. So basically, you know the concept, and this is the problem we are trying to solve. Yeah, it looks similar, you know the stuff. You can apply a working solution, because it will be great, right? Wrong. There are a couple of things that you have to worry about. First of all, a worker is an autonomous creature, and it can choose whether to work or not. He can simply log out of the system, and he will not be available anymore. And this is a normal situation. This is not a failure of system or some kind. You can't force somebody to stay logged in and work, at least not in Poland. The tasks must be prioritized, and this is a heavily desired feature from our clients. So yeah, I don't know if any other distributed task queue is like Celery support this. As far as I know, in Celery, you can partially achieve this by routing to different queues, and that's all. So it's also not sufficient. And the last, mostly mind-blowing, is that one worker can handle multiple tasks simultaneously. And from, I don't mean concurrently, but simultaneously, so in parallel, at least from a point of view of task manager. So this will be viewed that way. Of course, Q1 cannot carry out two phone conversations at the same time. But why not write an email and speak to somebody? That's fine. So our solution depicted as a whole looks for some centralized task manager, and it's written using Twisted. And the Redis will serve as a logs server and will hold them all. So the first thing I would like to say is about binary semaphore. It has only two states. It can be locked or unlocked. To lock it, I will refer to this as acquiring and to unlocking as releasing. This is the same as the standard Python library. Who has a background with multiprocessing or multithreading library? Yeah, that's quite a lot. And who was on the Monday talk on this? Yeah, that's quite a few things. So, and it will be stored as a Redis string, and all calls to acquire it or release will be non-blocking from Python. So the first use case is about keeping state of a worker, whether he's busy or idle. So meaning that lock is released, it's the same as idle and so on. And the lock will be acquired when task manager will assign a task, and it will be released afterwards. So it's over. And the second case is about preventing race conditions. The one I mentioned that a worker would like to log out sometimes, and I don't know, for a break or something, that he can do it using this mechanism because it can acquire a lock before a task manager do. Or it will simply do not release it after finishing its previous task. Okay. So since this is stored in this Redis string, we would use two Redis comments, a set to store a lock, to set the string, and a get to retrieve it. What we need is atomicity. Every single comment of Redis is an atomic one. So no one else can disrupt your data. So the first draft, we have a Chrome disk, something like that. We're getting the lock, checking if we can lock it actually, and then do it. But there's a problem, because between get and set, there is some instruction running, and this as a whole is not atomic. So someone else might have changed this in the meantime, and this is a problem. And it's not acceptable then. So we'll introduce some kind of Redis transactions. This is not the real transaction in a sense of SQL databases, but it allows you to execute a bunch of comments in atomic ways. So we issue the comment watch, and giving the key name. So this will be carefully watched during the transaction. And finally, if we cannot do anything, then we unwatch it. So this connection will be free of side effects later. The next step will be to introduce the transaction denotions, multi and exec. These actually will denote a transaction, or the things that should be done conditionally, even if a watched key has not been changed. So I guess that's quite simple stuff. Okay, the important part that task manager sometimes would like to know whether the state changed. When we lock the worker, he will find it eventually, but when we do not, when we are freeing it, it must know when the worker is freed. So we can actually perform some action, like, for example, look for the next task for it. And to receive the notifications, we issue a subscribe comment. But to be more precise, on the Python side, so this is a task manager, as I stated is written in Twisted. There is some kind of magic, like ready subscriber will handle the subscribe and actions for them for you. And this is the event driven. You see the very ugly method name, message received, which is written camel case, and by overloading this method, you actually define the behavior for this. So the second part will be about distributed semaphore. So the semaphore is different from lock in that way, that it maintains an internal counter. So it means that it can be released or acquired many times. If we define a semaphore with counter two, we can acquire it twice and also release it twice. But the original counter cannot be exceeded by further calls. And this will be stored as a list and the calls to it will be blocking. So the use case is that we can handle multiple tasks at the same time. As I stated before, the usual lock is not sufficient and the use of a bunch of binary semaphores would not be sufficient, too. So this is similar to string operations, but now we are using our push, which stands for right push, so it will append an element to a list, and the error pop, which stands for blocking right pop. So it will pop an element from the list. And our push will serve as a release operation and blocking right pop as an acquiring operation. So when there's nothing there, and it means that semaphore has been acquired maximum number of times, the command will block until there's something there. In the Python part, we do not change anything because it reacts only on events. Eventually you can implement your checking whether the semaphore is free or not. And there are cases when you might need a state of semaphore after changing it. You could use previously showed a multi-exec block, but that's not always the case. The alternative is to write a simple Lua script. You see that it's very much like Python in this case, except this word local, so it means it's allocating a local variable, so it will be garbage collected after this block. And this script only releases our lock and return a length that we have left. Okay, but there's a warning. Multi-exec blocks and Lua scripts all are atomic in Redis. The problem is that blocking invocations in atomic way has not sensed this time. So it would mean that the Redis instance has to be blocked forever to serve this one. So making bear-pop inside multi-exec will return you a nil, so it's none in Python client. And invoking such a Lua script, Redis will respond to you with an error. Alternatively you can use an air-pop, so it's non-blocking feature. It will return you either nil if lock cannot be acquired or a value if it has. Some final remarks before you rush and implement these solutions in your software. Please care about starting conditions. Make sure that the locks are properly instantiated before using them. And study carefully control flow in your application to not introduce too many locks or too less. And also please work up a restoring state procedure. Something goes wrong. There might be cases when the state in Redis is not finally consistent. So you might want to fix that some way. For example, we persist a state in SQL database as some kind of history of activity of given worker, so we can restore a state properly after some kind of failure. So well, that's all for me. And now it's time for some questions, if you have some, of course. Thank you. Hello. Have you considered using setNX for the lock, the setNX command? You mean setNX, right? Yes, yes. For lock. Yes. Yeah, we have read about this, and this is described also on Redis documentation as far as I know. But well, this is an expiring command as far as I know, yeah? You can use it as a non-expiring command. You don't need to set an expiring. Yeah, but somehow it didn't fit our needs. I can't remember now why, but no, it wasn't sufficient for this kind of locking. Yeah, but that's an alternative for sure. Okay. Have you tried to look at other implementations besides the built-in lock implementation in a Redis client? You mean besides Redis, right? Yeah, besides the official Redis client. Well, ideally, it would be to write your own locks server in Python, but the case is that some code base we have and the most is written in PHP, so it has to be something outside. Ah, I see. Okay. Hi. When you're talking about distributed semaphores and locks, are we talking about, so are the locks in one location, so in one Redis instance, and the workers can be distributed, or are the locks in semaphores actually distributed? Yeah, that's a good question, and I hope that someone asked that. Well, basically, we use a single Redis instance for serving our locks due to different limitations. We do not need more resources, but distribution you can achieve is by either sharding Redis instances, and this is done automatically by using Redis clusters starting from Redis 3.0, or replication, which is also in this kind. But when I prepared this, I meant distribution by, yeah, the workers are distributed, and these are different kind of resources that might achieve a lock, sorry, acquire. Does this answer your question? Okay. Any more questions? Have you tried, have you thought about using ZooKeeper, because I'm not entirely familiar with it, but my colleagues at work use it, and it's supposedly that, which is synchronizing few distributed instances of anything to only get one resource at a time. Well, we didn't, and I have read about something like ZooKeeper just yesterday on OREALLYSTAND, but I will surely consider that. So thanks for a suggestion, but no, I don't have any experience with that. Me too, but it seems like a good idea. Yeah, if it would be simpler or better in some case, why not replace? Yeah, because it's used supposedly to, you know, select the name node in a Hadoop cluster, for example. So this is also a really distributed system, and only you have to think about it and synchronize that, so ZooKeeper is supposedly for that. Yeah, I will investigate this for sure, and if I came up with something interesting, I will surely give a talk at the next occasion, not this year I think. Okay. Thank you. Any more questions? Well, I do have one there. Have you actually ever seen anyone taking on a phone and writing an email at the same time? Yeah, myself. Do you manage to actually make sense? Well, you know, there are some studies that shows that a human that is carrying out a few simultaneous tasks is completely reduced by its brain capabilities. So there was a comparison in the book I read that he behaves as if he had a brain of a six years old child. So, you know, I don't find personally that multiple tasks at the same time are a super killing feature, but that's what the business wants. Yeah, yeah, because they respond with some delay, so yeah. Maybe. So, any more questions? I guess we could then thank you again, Sebastian.


Today's episode is sponsored by Squarespace. Kodak, you know what to do. Fuji Presto 1600. Maybe you've heard of it before, but probably not. I don't know. How could I know? I'm just a guy talking to a camera in an office right now on a floating ball of rock in what's most likely a simulation. Based on its scarcity online, you would think that I'm making this film stock up. But nay, it is how it sounds. Fuji Neopan Presto 1600 is a black-and-white 1600 speed film from the good old days when Fuji's tagline wasn't Fujifilm.  I received four rolls of Presto 1600 from somebody over in Japan. Expan Japan, in fact, which is their legal birth name. I recommend checking out some of their work. As you can imagine, it's mostly Expan stuff. Squished, squeezed, and compressed to fit on the best platform for panos. Instagram. Anyway, eager to see what all the hubbub about this stock was, I hopped in Caleb's brand new Porsche that he bought with his big-time Hollywood VFX producer money that's totally changed the kind of person that he used to be. I don't want to run this lady over because that would be murder.  Inspired by all things panoramic, I decided to only be fitting to shoot this roll on the Expan myself. So I loaded her up and decided to shoot these flaming hot power slappers at 1250 ISO to account for the expiration. Speaking of expiration, there are a lot of variables here. Black and white film generally holds up pretty well to the forces of time. Better than color negative and slide, actually. Ah, damn it. But also high ISO film degrades faster over time. But then again, supposedly this stuff has been kept very well over the course of its lifetime. So, in reality, your guess is just about as good as mine. After stepping out in our matching lightning McQueen Crocs and immediately getting standing salutes from everybody nearby, we tucked into fashion alley to hide out and look for some compositions that would work well on black-and-white panoramic. First impressions of Fuji Presto 1600. It's pretty contrasty, for sure. I guess you kind of have to expect it to be. High ISO film is really pushing the limits of what's achievable through the format. In my limited understanding, high ISO film requires specialized machinery. And oftentimes some film manufacturers would just get around this by using developing tricks. For example, Ilford 3200. That's about as fast as you can go these days.  But a lot of speculation online will have you believing that Ilford 3200 film is actually a 1000 ISO film that uses specialized development to kind of fake its way to 3200.  But I can definitely see this being the case for Presto as well. My results were grainy and thin on shadow detail. Very thin on shadow detail to be honest with you. That could be a result of it being an expired film or that could just be how this film is. I don't really know. Either way, if you do shoot it, protect for the shadows. However, this film isn't all darkness and negativity. There is some good stuff. It still is high ISO, which means you can grab shots in super low light with a super slow camera.  This shot is an unfortunate victim of shadow detail loss. This stock just doesn't hold on to them well. Our subject here gets lost to the shadow mush behind them. Here's an interior shot that was especially metered for the shadows and it held out pretty well. The highlights aren't too far gone either. Perhaps 800 ISO is the better speed for these roles at the moment. And look, if you get a few shots back that would have been absolute heat, but the darkness once again took over, just like the darkness within all of us, then just tell everyone that you intentionally exposed for the highlights and they'll most likely believe you. Anyway, after running into whatever's left of daft punk on the streets, I started thinking about Presto's natural competitor in the wild, HP5 pushed to 1600. I would personally say the look of HP5 pushed to stops at 1600 is a better overall move, but that's only because the Presto that I was using is in fact expired. Ilford 3200 shot at 1600 ISO could also be a good option, but I've never tried that one personally. Anyway, with the fashion police still hot on our trails for having more drift than anyone on earth, though I am excited to see what our new alien brethren are going to drop, I shot this and I consider it quite good. I'd love to have seen more info in the shadows, but hey, you know what they say, the living we learn and my doctor says I should be dead in about three and a half weeks. Anyways, we continued onward through fashion town, learning about all the different types of threads and phantoms I guess, I don't know, I never saw the movie. I took this of the UPS guy and the USPS guy, a forbidden love story. One of the unique characteristics of this film stock is that it was really made to be shot anywhere from 400 ISO to 3200 ISO. Apparently it featured decent push processing capability if you wanted to take things to the next level, not me though, based on what I've seen here. Apparently Presto is a stock that pulls very well too, definitely a less common practice these days, but still useful nonetheless. Based on my results, this film is very contrasty. I'd say pulling it one stop would even out the tones a little more nicely. We eventually came upon this vibrant and colorful neon sign, which was perfect for sterile, lifeless black and white. This is a shot I was looking forward to seeing the most from the role, a woman reading a book at the bus stop in front of the theater. Did it turn out how I expected? No, but maybe I put too much pressure on it in my head before I saw the final result. Maybe it's a good shot overall, but I just need to back away from it for a while. Being an artist is tough. I now understand what this guy was talking about with all of its unbridled complexity. Anyway, with one or two shots left in the role, I ignored the blatantly obvious shots that would have been good and took this instead. If there was any light or detail on the subject here, this shot would have worked out well, but I'm not that conscious as a photographer or lover. We can just throw that in there too. But speaking of love, you're going to fall right into it with today's sponsor, Squarespace. Are you an artist struggling to get your work out into the world and upon the eyes of the people? Aren't we all? Having a strong portfolio online is step number uno towards advancing your career in whatever line of work you're hoping to enter. Otherwise, where is everyone going to see your hard work? I use Squarespace to host my own photography portfolio because getting it set up was easier than pie, thanks to their simple drag and drop user interface. Start your website off with 100s of professionally designed templates that you can choose from and simply build from the ground up with ease. And if you know how to custom code, there are even modules for that. And if you run into any snags during the process, you can check out Squarespace's award-winning 24 seven customer support to get you back on track in no time. So what are you waiting for? If you're ready to build a website, you can start a free trial today at squarespace.com slash grainy days. And if you use the code grainy days at checkout, you can get 10% off your first purchase. In the end, who is this film stock made for? I don't know. But I do know one thing. Fuji used to dream big and this film stock was a product of that. Is it my favorite black and white film stock? I wouldn't say that, but it's a fun one. I might experiment with shooting it at 800 ISO in the future. I have three more roles of it, and I'll definitely be saving them for something special. Until then, I guess just hit that subscribe button for more makeup and beauty tips.