Did you know that there's actually a way of figuring out which hair color and hair cut will look best on you based off of your astrology sign?   I don't know. Basically I'm just gonna go through each astrology sign, name some characteristics of that sign and then decide which hair cut and which hair color I think would look best on you.  Let's do it. Aries. Aries is out there? What's up? Aries are known to be ambitious. You're brave.  And you're optimistic about the world. You're just like, hi, I'm ready for it. Let's do it.  But you're also reckless. You can be a little selfish and a little short tempered and that's okay. I understand. You can be a little intolerable about certain things.  It's giving me a lot of red. It's giving me copper. It's giving me a wine color, a berry color. I need a bold color on you. It needs to be strong. It needs to be sexy and it needs to feel bold and independent. I want edgy haircut on you. I want something crisp. I want something clean. I don't want it too shaggy. I don't want it too layered. I want something that's like, urgh, bah, bah, bah. You know what I mean? We're gonna do a bob. We're gonna do a longer bob. We're gonna do some bangs but everything needs to be like cutthroat.  You know what I mean? Let's just get that thing in there.  And when it comes to styling your hair, I'm getting a vibe of straight hair. I feel like you need a nice flat iron just crisp. I want it clean. I want the lines. I want the symmetry. I just want it to all come together and be this magnificent like, bah, of hair in front of me. And it doesn't have to be red, red. It feels like there needs to be a lot of warmth in your hair. A lot of that fireiness should come out in your hair. Tortises. I mean, torus. You're intelligent. You're hardworking. You're dedicated. And you can definitely be stubborn at times but it's like kind of cute a little bit.  Your ambition is just out of this world and you enjoy things that are luxurious and cozy. You're also though not fond of authority.  Your color is giving me platinum blonde because you want to put that hard work into your hair too. You're like, it's okay. I can be at the salon every four to six weeks to touch up my roots. I can do it. Don't worry.  You're icy white. Woo, your hair is healthy. It's giving long and just full. It's giving extensions. I'm seeing you have a lot of thick, nice hair but it's not actually your hair because again, you're willing to put in the work and the money that costs. I'm picturing a nicely layered cut. Everything is flawless though. You have the curtain bangs but the curtain bangs are like whoosh. I've never seen anything so perfect in my life type of bangs and it is looking clean, free of dead ends and it is shiny. Or you have some tight curls. It is big. It is giving rounded. It is giving a boom impactful or even super wavy. You style it with a loose beach wave but the beach wave can never be frizzy. Your hair is always kept pristine. Gemini, you're extroverted. You want to be out there. You want to be talking to people. You want to be meeting new people. There is literally never a boring moment when you're around Mr. or Mrs. Gemini. You're flexible though. You're like, you want me over here? I got it. You want me over here? We can do that. I'll figure it out.  However, you can definitely be a little indecisive.  Ooh, those are tough ones. Let's work on that, you know? And based off of that, I'm getting a clear vision already, oh, oh, I am getting it. You have rainbow hair. Like you just want to stand out of the crowd.  Your haircut's giving edgy. There's something going on there. It's an asymmetrical cut.  You got a bang going this way. You got this going that way. It's a bull cut. No, it's a Mohawk. No, it's a, a mullet? Whatever it is, it's attention grabbing and it's iconic. And it just screams, I'm here for a good time, not a long time. Cancer, we're on to you. This is your time. You can pick up energy wherever you go. You can feel the vibes. You, bad energy. You, good energy. You, leave my sight.  Whew, that's a fire combo.  But you love a night in. Nothing sounds better to you than just cozying up on the couch with a nice soft blanket, watching some TV, something chill, I don't know. You want to stay in. You can be known to avoid conflict, which I get. You definitely need your time also to recharge your battery. And you can be a little distant because of that. It's giving me very ash toned, cool color. It's giving blue. It's giving gray. It's giving silver. It's giving that kind of gunmetal metallic colors. Like I want something fierce on you.  It's giving me easygoing. It's giving me long. It's giving me a little bit of like need to get a haircut. A little too long type of vibes but still it's looking effortless. It's looking cool and the style is whatever you wake up with. It's kind of just the chill, messy, but really sexy hair. Lilo, what's up Leos? Leos are known to think of themselves as quite a celebrity. You think you're all that and maybe you are. You're known for your loyalty and your consistency. You're dedicated friends and lovers.   Can I have some of that please? You just refuse to accept failure which is like the best trade ever. Like even in the darkest times you're like, no, we're gonna be fine. We're just gonna figure this out. But you can't have a bit of an ego and a bit of a jealousy problem. Sometimes. Not everybody though. Very strong opinions about things that you have opinions on. And it's not easily changed and because of all of that, you are definitely giving me bold highlights like moneypiece, bright in the front, dark in the back, impactful highlights. And these highlights can be not only blonde but also be different colors. Like let's say a pastel pink or a red. I'm thinking you have just that really edgy hair. You want those crisp lines.  Even like a bang that's a different color than the rest of your hair or the tips of your hair color neon yellow, something that screams I'm unique and I wanna stand out. You're giving layers. You're giving mullet. You're giving bob with bangs. You're giving anything with a nice crisp line and hard edges. The Virgo.  If you want everything just in the right order and everything placed in the right spots and then you'll be happy.  You can take in any kind of information all around you and just organize it. You make everything into like a clear concept.   And being a perfectionist also comes with its downsides. It's a little bit a lot because things aren't always perfect.  You know, your mind's always racing. It's always trying to figure out what else it can do better and you are definitely giving me dark hair, black hair, brown hair, espresso, really overall just striking dark hair style. You're giving me an easy going haircut. Something that's longer, maybe to the sort of collarbone area. You're giving me light layering in your hair and maybe a curtain bang or maybe no bangs at all. Maybe just some face framing layers. The styling is easy going, it's effortless. It's kind of whatever you wake up in and then kind of do a little justhing. That's what I imagined for you. Libra, you got a sense of style because you got that curious mind. You want to try everything.  You're also very kind and very thoughtful. You make for an excellent designer, stylist, decorator, critic, you're also a great companion. You just are really there for people. And God damn, you're easy to get along with because you solve those conflicts like it's your job. You're like, eh, conflict, no, no problem for me. I can figure that out. I gotcha. Sometimes though, you can struggle with indecision and you just can't make up your mind.  But you can also be a little bit controlling. Don't get mad at me. I didn't make this up, all right?   I don't even know why that's like a bad trait. Let's not forget to mention patient and unreliable. A little bit, you're in there.  With all that, take it into consideration. You're definitely giving me a pastel pink. I couldn't even say anything else. It's just pastel pink just keeps coming up in my head right now. I just see that like dusty rose, that pastel pink, that snappy color that's saying, I'm a designer.  It's giving shoulder length. It's giving layers. It's giving wolf cut sort of vibes, textured, messy style. It's feeling like you curled your hair and you shook it all out and sprayed some hairspray and dry shampoo on it and called it a day and wore that for the next four days or five days or seven days.  Oh, is it not my favorite sign? Scorpios, where are you? Your boy is a Scorpio, indeed. We can be quite mysterious.  To be honest with you, it's like actually kind of weird. This is not about me though. Once we make a decision, there's very little room for negotiation. We have decided on that and we are going with it and we're sticking with it. We're modern. We like fresh ideas. We want to progress forward and once we build trust in somebody, we are loyal for motherfucking life. There's no getting rid of us. We live that luxe life. Everything we buy needs to be top quality. We just like to find our things in life, I'm sorry but we definitely have control issues and when things don't go our way, it gets a little upsetting. We have that intense drive but sometimes that leads to like self-destruction. It kind of makes us a little bit angry when we don't succeed in the ways we want to. And Scorpios, what am I gonna give you? I am gonna give you a bold color but a natural bold color. I'm thinking black or platinum. Devil and angel, vibes. Like you just need something that's gonna pack a punch but you don't necessarily wanna be the center of attention. You wanna blend in but stand out if you will.  It's giving me like this, like you're walking around with your ponytail moving side to side. I see that for you. Stay sexy. Such a terrier, what's up?   You're just imaginative.  You wanna become the expert in everything. At the party, you're the one talking. You're the one meeting new people. You are born to explore. You wanna see everything, feel everything, know everything. And it's sickening. But you're brutally honest and sometimes that could hurt people's feelings.   You're giving me like that deep plum extravagant purple. You wanna make sure everybody knows that you are the creative one.  And I see like a shag cut on you. I see something flowy and cool and unlike any haircut that anybody else has in the room. But maybe you also have like a deep dark root with the purple. Something that's making it look lived in and cool and fresh and modern.   Capricorn, say these words with me. Visionary, leader, trailblazer. That is you.  Can't even handle you. You love positions of power and being the leader of the group. You even absorb the responsibilities of others because you're like, I can do it all. It's fine, just give it to me. And you have a huge heart without being overly sensitive. But sometimes you project your values onto others and it can get a little bit messy and you have a hard time just appreciating like the goodness going on in your own life as it is. And if it doesn't matter in five years you just can't be bothered with it today. Like just leave me alone. You're giving me like an underlayer moment like there's a peekaboo going on. It's everything subtle and natural on the top but then boom, pink underneath. Boom, we get the skunk stripe. We got something going on but it's included with that natural hair color as well. And it's a little bit more on the subtle side but it definitely packs a punch. You definitely have some kind of color going on there. That just shows like, I'm in power, not you. You're constantly getting haircuts. Your hair is extremely healthy, extremely shiny and beautiful but the cut is practical. It's something that's longer, long layers, no fringe, too much of a fuss for you. But you have a little bit of face framing going on and you get a nice, gorgeous blowout that has body and movement. Aquarius.   You love collaboration. You love working with other people and you do it very well. But you're also free spirited and eccentric and kind of off the beaten path when it comes to your sense of style. And it's so cool. You kind of pick up unusual hobbies and have a sort of nonconformist attitude towards things.  If you're bored at your job though, you're kind of like, I'm done with that. I'm just gonna move on. You definitely overthink and it leads you to exhaustion because you just keep going and going and going and it's like, you just stop thinking for two seconds. You are definitely giving me the color blue. Oh, like a deep, deep blue or even like an aqua. Like something that is just boom. The color blue is the color of the ocean and the ocean can often center you. You need something that brings you back down to earth in times where you're just overdoing it. And when you look at yourself in that mirror, you see that blue. You're like, I feel good now.  And your haircut is giving nonconformist. It's giving anything that is not of the norm. It's giving asymmetry. It's also giving mullet and a baby bang. Something really short, really cropped above the chin. Something that's really giving like, yes, you can tell it's me from behind because of my haircut. The styling can be really disheveled and really crazy or it can also go the opposite direction and be super polished and really crisp. There's no real telling with you. You never know, Pisces. You're always expressing your feelings. Oh boy, are you. But you're so compassionate and empathetic. You can literally adapt to any situation to get along with anybody and everybody.   But you can be flaky and delusional.  It's a little bit of a mess. And you just become captivated by your emotions sometimes and you can't find a way out. And if another person doesn't live up to your standards and what you think of them, you can feel kind of betrayed and like, what the f**k happened, bro? You are giving me pastels of all colors. Pastel pink, pastel silver, pastel blue, lavender, anything that is just subtle but is really cool still.  It's really important for you.   And your hair color says just that.  It's something that just looks natural and like you didn't really do much to it. A little bit of layers, longer hair for sure. The ends of your hair are not blunt. They're almost straggly but in like a cute way. There's no bangs. It's all long, all around your face and there are no face-forming layers. Like it is just, but it is natural looking and it is really cute, especially combined with the pastel colors.  Like you're putting a little bit of styling effort in on the weekends maybe. A few curls, a few waves, a little bit of crimping going on. Just something that's effortless and cool but still embraces your natural hair. Or your hair is curly and super volumous. Like you got those curls going in every wish direction. Do you agree with what I said about your hair based off of your astrology sign? Let me know if I got you completely wrong or if I got you completely right. Make sure you guys check out my hair care or my hair color line where you can color your hair basically every color I talked about today. Everything is linked below or you can go to xmodohair.com. And that is all for today guys. Thank you so much for watching. Don't forget to live your extra life and I'll see you next time. Bye.


Yeah, it was nice to know about SEO. So that's my VPN, actually. I was working. So my name is Senthil Kumaran. I work as software engineer at Twitter. And thanks all for coming to know something about what's happening on Twitter. Close the VPN stuff. OK. I used to work. I just am one and a half years ago. I used to work in Singapore at different companies and I got an opportunity to work as a Python developer at Twitter and then I took it up. It's interesting. And I have to say that it's one of the good experiences I've had because I tend to work with extremely sharp developers from all over the world. And being a member of the set of developers who are really skilled and really passionate about what they do, that first it helps me motivate to be in the development line for a long time, be as a good developer for a long, and then also helps me to appreciate the craft, which is software development. It's not just a business about the software. It's a craft. It's how you develop the software, how you make the software stable, and what are the complexities which are involved in developing a software which is usable by millions of people in the world all over the day. I can definitely appreciate Twitter has about a good percentage of like 250 million users every day, every second. Facebook has a billion users, actually, per day, which is four times bigger than Twitter.  And all these companies kind of teach us that how do you develop software which is used, approved, and stable across the hands of all the different people and stuff like that. And developing at that scale has challenges. And the team I am in is solving those challenges. How do you develop software at that scale which is fast, stable, and accurate? So that's what I do. And within our team, Python is a major language which is used. Outside of my team, Scala is the language which is majorly used. Scala and Java are the languages which are majorly used in Twitter. And C++ for the core infrastructure, very less. But these are the languages. And a huge development group, like around 1,000 engineers are there who work with that. So with that intro, I will be releasing. As I said, there's a variety of languages. There is Scala, Java, Python, C++ mostly. And back-end services are in Scala. Revenue and machine learning stuff are in Java. Yeah, just a year ago, I saw the actual production ready machine learning stuff. The one which provides you with the promoted tweet or the promoted content and stuff like that. There is a team. The team is called the revenue team. That's what they say. And the revenue team's whole job is to write the programs which make money for the rest of us to work, actually. That's the one which makes the money. The machine learning programs go about doing the job and make the work. I mean, it looks like one of the major, the currently hot field in the industry right now, machine learning. I'm not there into it. But I have seen friends who are good Java developers, good Scala developers, very good in algorithms, and usually from top universities. Yeah, however they are. And they are into this revenue team which write the money-generating stuff. And generation is huge. Millions of dollars per second or something like that. And then thrift. Thrift is an Facebook product. Thrift is for the protocol-agnostic sharing of the code. Like, you write a thrift specification and then you give it to someone else. And he can generate the Scala code. I can generate my Java code. So he can, he can, using the thrift template, you can write your Scala server. And I can have my Java client. And then we both can interoperate with it. So by doing this, we share the thrift. And thrift becomes a protocol-agnostic way of sharing it. And huge lot of code is written in thrift. And surprisingly more. And by writing one thrift, you can generate Java, Scala, Python, other Ruby, all the other languages. So that's what sometimes Facebook, I think it started with Facebook. Facebook and Twitter, what they did was like, each one seems to be expert in different languages. No one is an expert in all the languages. One person may be good in Java, another person may be good in Scala, one person may be good in Python. But in order to have a big company like Facebook or Twitter, you need to have these good focused people into the same team. And you cannot ask everyone to learn the same language again because they'll be losing their expertise. So the solution seems to be like, let's write thrift. Let's generate the languages which you are comfortable with. And then let's go about, let's go about writing the client services over the thrift. Okay, let's write the client which will interface on the thrift server in the language which you choice, in the language of your choice. And let's go about with it. Huge, I have very less knowledge on that. But we deal with it constantly, every day. Okay, infrastructure and tools are written in Python. Twitter before 2011 and 2010 was mostly Ruby. It started off as a Ruby on Rails app. But later when WorldCorp and all happened and then billions of people started using it every day, in order to have the scalability and in order to have the power, in order to have the power of JVM, Twitter you started using JVM. Okay, so when I said all the languages, let me have something in the background so that, so this is my Twitter repository. I'll start counting the major repositories which we have so that, let it go in the background and we can come to it later, okay. Let's come to the pass, like to give an indication of what all the languages that we use. And this is not full to pass yet because I just chose three of the major repositories which we work on. And I needed to give that, to give an indication of how huge the software development is. And once we get the results of it, we can come to it, it's still scanning so let's give this some time. So how the development workflow happens is get this the version control repository and we all work on a branch and then the branch is submitted for review. So by review, there's a tool by name review board, okay, reviewboard.org, which everyone can, it's our open source tool. So the review, the patch which we work on is submitted for the review in the review board and then there will be experts of each branch of the repositories. And then they are usually listed as the owners in the repository. And the person who's listed as a owner needs to approve and give a ship it, actually. And then only when the review is given a ship it, when you submit it, it doesn't merge into the master yet. It goes into a submit queue, which is like a Jenkins instance. So the whole, the submission goes through a Jenkins and then the whole set of suite is run against the entire world of the Twitter to see that the changes which you are making, even a single print line statement change will be running the entire 1000 tests or so for your repo. And after that is run and then passes, it will get merged to the master by the Jenkins itself. This is the, I mean each change, sometimes if you are doing a change in the major library, code library, it takes a couple of hours after you complete the code and then review it and then submit it to go into master. But once it's master, it's kind of like stable approach that we are, we have written a well-reviewed tested code which need not have. And the reason these kinds of things have happened, okay, when I started in the software development industry working at Dell, we didn't have these kinds of processes in the software. We were, they had manual test teams, okay. The test team's job was to download the binary and run the tests against it. And with companies like new web 2.0 companies, there's no manual test teams at all. That industry is gone actually, unfortunately. But because it has been replaced by tools which have such a streamlined process of review and automated testing and stuff like that, which give you a good graphical result of if it's correct or not. So if the submit queue says okay, then the change is master. And then deployment, it's like, it depends on the team. Like immediately it's deployed to the hosts wherever we are. And it's controlled by a decider, okay. That's what we call the decider. So there's a web framework by the name decider which has all the features. So the one which you saw Twitter.com on the website, it has like 120 services, okay. By that I mean that. So each thing, this page is a service. This following tweets and followers are different end points of service. There's who to follow. This is like a machine learning algorithm going on to say such as who you should follow. And then the trends, it's basically given from the search and analytics team which goes and then searches for all the things and then gives it. And this is a trade service. And this one is running on like hundreds and 200s of machine, okay. This is one. And this, this is like the Twitter page consists of like 100 services. And each 100 services, one service will consist of like 10 engineers. And so 10 into 100. That many engineers are there, thousand engineers are there. So this is, and I was surprised to know that the Google Docs, right? Okay, I mean because I knew this because my team member whom I work with used to work on the Google Docs team. And you see that in the Google Docs there is the star button to make it a favorite. You remember that? The star button, okay. The star button. This seems to be the service in Google and there are 30 people working on it. 30. 30. And all smart engineers, by the way. I didn't know nothing of that thing. Just a star. Yeah. So that seems to be, I mean, surprising to know that that seems to be the state of the world right now. That's it, I. Yeah, depending on the team you are and the kind of service, your code is immediately deployed. Application engineers are encouraged to turn experimental features on a subset of users. That is followed by all the web companies, I think.  That's it. And in order to support the architecture of Twitter which is like real-time web used by millions of people per second and then different clients and the TV channels and everything, what we needed to support the development for was a scalable version control system which is Git, but with a lot of engineering energy gone into it. The review process was something which I contributed majorly, like we had a tool by named Git review which is not an open source yet. But that's one. But the scalability of Git is itself is a huge thing. Facebook faced the problem. They went about with trying to use Mercurial. I use Mercurial before using it and I think that it is better than Mercurial. And so some of the engineers also think that too. So Twitter is investing on Git again, like thinking that let's see if we can enhance Git by using standard scalability thing. Instead of putting a report on the Git software on the object model itself, let's shut the servers where the Git servers are installed so that our latency is small and then Git can support better. Which works fine. I mean, I don't have any complaints, but there are people who have a lot of complaints against it. And Facebook seems to have a complaint and then they are moving to Mercurial. I have no idea about it. Build system to support multiple languages. This seems to be interesting because I have no idea about what it is before I join Twitter. If you are a Java developer, you may perhaps be using Ant or Gradle or Maven. Gradle is used by the Android studio for the Android development. Ant and Ant is majorly in the eclipse world and then Gradle is, Maven is another one. I have used none of them because I had never used Java before. I didn't know about what is a building. Python doesn't use that. Python doesn't build it. Python is interpreted. But in order to support multiple languages, it's easier and better to build the software, create a binary executable, attach the dependencies and create a jar so that a jar which consists of all the dependencies together so that you can deploy the jar as a single binary on the deploy. And the idea came about like this. They wanted the whole twitter.com to be a single jar, which can be thrown into a cloud and then run. So that is why I think historically they went powerful with it. And in order to have this, yeah, I mean I ran this account, lines of code count. And just before the three metered repositories. In order to have Java, Scala, Ruby, Python, BornShell, SQL, and a little bit of CSS, some of that SQL stuff. In order to have all these combined together into a language node, into a build, it needs to, I mean there is none, Ant cannot do it. Ant simply will not support all these languages. Okay, Maven cannot do it either. So Google has something called Blaze, okay, their build system. I'm not sure if there is a Google Blaze build system. So here's, building in the cloud, how the build system works, that is the thing. Facebook has Puck build system. Oops. So it's a Java build tool, okay, and Twitter has Packs, okay. So this is our Python, this is our build system we used in Twitter. And this is what I work on, the team which I work on. And it's about compiling the languages and everything which is written in a variety of languages which I showed you. So given this much of information, the introduction, I can dive a bit into the build system and then show you how things act, okay. If I speed up, please don't mind. We can have that thing later. And by the way, all these build systems, Facebook is open source too, Puck, but it's for JVM Java and Android only. Twitter is open source too, it's called Twitter, GitHub, Twitter Commons. It's for Java, Scala and Python, and it's written in Python. I don't know, Facebook's written in Jiphon perhaps, I don't know, I have never looked at it. But all of them are inspired from Google's build system called Plays. And all of them have been developed by these engineers who worked on Google, who worked at Google and who found that that build system to be of very good in building such a good stable software. And they got inspired by that and then whichever company they went, like Facebook or Twitter, they went about implementing something similar, okay. So the parent of it is Blaze, okay, and people still appreciate it for many things. It's not open source. These children, which got folk from that, these are open source. So you can clone it and then look at what it is. The review process, I said like, it's a published and submit, it published to the review board and submitted, that's okay. Okay. And Twitter Commons itself includes a build system. It's something similar to Ant, Maven and Braille, and that's what we are going to talk about. But look at an example, okay. Let's directly dive into an example. I have the code checked out so that we don't have to do it. So I'll increase the font. So I'll read a Python Twitter, Pants, and let me open from there. So this is how the Pants thing is. So it starts with and I'll explain the directory structure from here. The binary which we have, okay. I'm finding it, I'm thinking in terms of how should I give an introduction? Because if I dive in directly, it will not be offline. I mean, you can give the directory names. But let's look at the example of something which we are going to compile. Okay. That's another quick question. In Twitter, the Python process run on Jicon or? Python, Python. C, C. C, Python. So let's look at SRC, SRC Java, com, Twitter, common examples. Ping-Pong is a name and then let's look at a build file, okay. So this ping-pong is creating a binary binary ping-pong which has a base name and which has a main class which is like the main it's pointing to. And it has a dependency called a ping-pong lip. And the ping-pong lip consists of the dependencies which is using the guava, which is like a Java library for it makes Java a little bit more like, it's Google's Java library, actually, guava. And Google is a Google's again Java library stuff. And Sun has provided some Jersey client. Since I don't know much of Java, I don't know what they are for. I'm giving an example of a Java compilation and then I'll look at the, I'll give an example of Python compilation. Okay. And then it depends on other modules like application, HTTP and HTTP modules. All these are separated into modules. And if I go into one of them, like application or HTTP, say, let's go into common application and, okay. And SRC, Twitter, common application, that's where I go. And if I open it, again, this is a build file by name application and it's written as a Java library. And this one says that it provides a jar by name application which is just published in com.quitter namespace. And it has the repo specification wherever it wants to publish it to. And it has its own dependencies, okay. So, as I saw, there's application, HTTP, and then everything else. It will pull in all these dependencies and then bundle it together into form of a Java library. And this Java library can be used to create a JVM binary. Now it forms the sense, right? We wanted to create a JVM binary, a binary which can run on JVM. And the JVM binary will depend on a Java library. And the Java library in turn depends on multiple Java libraries. And this is like this. And in order to create a fans goal build, fans goal, let's say compile, okay. SRC, Java, com, Twitter, common examples ping pong. This will go about compiling that Java library with that stuff, okay. Let's go fans goal bundle, okay. It will create a bundle to Java library from the different stuff, okay. Creating this ping pong that jar and stuff. And you can also have fans goal run. I'll get the full example from here. Let's run this, okay. Fans goal run. And this will run the whatever sources we ran against and let's running it here. And we can see the various end points which it has exposed to. This is just an example which everyone gets used to. So like what are the different contentions, like logging aspects of the library. The developers don't do that, okay. These end points are not written by developers. If you see the code, none of these end points are written. It's inherited from the dependencies. And I'll graph you for the different processes like Glass, Loan, Com, Viewplot, how your performance on the JVM is happening. So these kinds of stuff are all available in the form of libraries by the Twitter and common library stuff. And like the important part and the interesting part was a JVM binary created by a Java library. And now let's look at SRC, Scala, com, Twitter, common examples, but okay. Here it's a JVM binary which is created from a Scala library. So because Java and Scala are by code compatible, you can create a JVM binary from a Scala or a Java library. And you can create this thing too. You can interpolate. And it uses the same build system again. So we have abstracted couple of layers now. Same build system, same commands, same interfaces, same binary, but internally you can substitute one library with another. You can use Java or Scala, okay. That's it. And at last, it's not just for Java and Scala, it's for Python too. SRC, Python, Twitter, common, SRC, Python, Twitter, common, fans. Let's look at fans, let's do it. Okay, and fans is the one which we use. And that itself is written in, that itself is like written in Python, and the source code is in Python. What I'm trying to say is bootstrap. See, I use the fans command initially, right? And so I'm going to look into the fans code itself. And I open the build file. And I see that it's a Python binary by name Py. Just like I showed you, there was a JVM binary. And now it's a Python binary. And the Python binary by name fans. And it has an entry point, whatever it starts to. And it has a dependencies, like what it depends on. And the dependencies, if you look, here it depends on the fans limit, okay? And it says that for what platforms it can build. It can build for current, it can build for Mac OS X or Linux. And the fans live, if you see, it's a Python library. Okay, and the Python library depends on so many other Python libraries. Like the app is our application framework, or the conference is the tool for the key-like thing, okay? It's for publishing. And then it can depend on context-suitable and some other languages, like process-suitable and stuff like that. So the encouragement is to write libraries modular so that your library can be used by some other product. And once you have the dependencies on it, you can build fans in a similar way like this. Fans go, fans src, Python, Twitter, fans. And this will go about building that fans bundle, okay? Just like the Java library, we have a fans bundle. And it wrote it inside the disk directory. See, sorry. Now, it wrote, first previously we created the ping-pong.jar, right? And ping-pong.jar has here, it has all the contents of the class. And similarly, fans.txt has the contents of the PYC files which are to be created for running them. Okay, this single executable can now go compile your Java Scala and everything for, and create binaries, can run tasks, and do multiple things. That, I mean, that's pretty much like, I wanted to give an example of how it feels like to have a build file and compile a Java Scala binary and stuff like that. And it's interesting that other companies are adopting it. Foursquare is a major company which is using fan-spill systems, stumble upon a son of a, and others, like if Twitter engineers go forming some other companies, they kind of like, since they got used to it, they go about using it. So, and common libraries are, since heavy effort is put into it, and it's open source too, it's easy to utilize it, okay? I sometimes don't like some of the portions of it because there's too much abstraction, but still it's good to have a look and then develop something better. A lot of things are featured in media. There's lots which are, if you go here, there is like plenty of projects which are available, like there's Storm, there's Scalding. Scalding is like a Hadoop job running using Scala Phileg is our RPC system, which is one of the code for the Twitter, and then JavaScript stuff. I'm not a web engineer, but there are a lot of web engineers who do plenty of good work in Twitter. I have no idea about that. But that's like a Zipkin is for like tracing and to have a cache with them and crash proxy which is used by Wikipedia as well. And by the way, it's like a Scala library, I have no idea what it does. But plenty, and we talk about commons only right now. Like that there are like so many other stuff. And Twitter got featured in a number of media recently, like the Tweets per second apart. This one was interesting. Here are new Tweets per second apart. On August 23rd, 2012, I guess, 2013, very recently. It was August 12th, 2013. It hit a new Tweets per second, that's 150,000 Tweets per second. It was huge. I mean, you remember just few days back, few years back, there was a C10K problem, kind of web server, 110,000 connections per second and stuff like that. Now look at this, it's 150 Tweets per second. It's huge. And this blog post explains about how it went about doing that and what happens when so many things happen. And Twitter didn't break that, that's why. That's why it's like, and what was the event which was happening which took place was, I think Japan is, yeah, Japan people was the Eric of Castle in the Sky. I have no idea what it is. Okay. So when Japan was watching Castle in the Sky, there were so many people tweeting about it and then it hit the high rank of like 150,000 Tweets per second. It's a mark, how much? When Obama kind of like says that we want election or something like that, that's huge, okay. Or when, what is it again? Thank you such an campaign which happened. That was huge, I guess. And there's a lot going on. And there are other media articles which got featured too. Like this one, the second coming of Java, the Relic returns to all the web. This explains how the Java is used, okay. Java by it, it's JVM. JVM and then other languages on top of JVM. And then there is return of blog, how Twitter rebuilt Google's secret weapon. This is about our cloud infrastructure by name MISOS, okay. It's open source too. So it's an Apache project. Apache.MISOS.org. So making it easy to build resource efficient stuff. Airbnb seems to be another consumer of it. What it does, it is based by the University of Berkeley project. There's a professor by name, Stoica, who's a researcher, very famous in the distributed systems world. He kind of leads it. And this is like, if your company doesn't want to spend money on Amazon web services, because they are costly, but you want to have the same scalability which Amazon web services provide. And same kind of like abstraction which AWS provides.  Well, that's what it is about. Download MISOS, install it on all your hosted machines, and then provide your engineers as AWS instances. It's like build your own cloud kind of thing. This one is like, and so Twitter doesn't use AWS. We use MISOS because it's developed within, installed on all the machines. Our jobs are running on MISOS. And it can be monitored, it can be, and it's like standard distributed system stuff, okay. So that had got a good feature. And that's again, as this article shows, it's inspired from the Google again, okay. Google has done really good stuff in this, in not just Silicon Valley, but all over the world. So there's a lot of technologies which are built very similar, or sometimes when they are building the second time, it's built better than that. This is another example which. That's what I wanted to share. And if you have any questions I can take. I hope it was interesting. A lot of information was presented. And if you have any specific questions, please feel free to reach out to me. Yeah. Thank you. Questions? Are there any questions? Yes, yes, yes. Hence, in the, how is this done in the classroom? No, the underlying compilers are still GCC, JV, Java and stuff, Scala, SBT, okay. But in order to bundle all the projects together, it's like make file. So make file is the traditional build system which we know, make and make file. Make file by itself doesn't involve, make file by inside it involves GCC, right? So in a similar way, these structure the projects in such a way that you can have the organizations, you can have the, you can have the compilers, correct compilers with correct flag getting involved to build the binary. They are not compilers by themselves. Pardon? Python two. But it's compatible with Python three. It's Python 2.7, but past it's written in Python 2.7. It's compatible with Python three, upward compatible. I have no reason, we have no reasons why we are not using Python three. I mean, I guess you would treat different major version changes like a different language. Yes. No, not for, not for Python inside of there. The languages that are compatibility label libraries which are written, I'll show you an example. Let's look at, so look at this code. It's comparatively Python three. If there is a comparatively layer of Python three, we import statements are written in such a way. And if there is a, if there's not a Python three, it's written in such a way. So it's some random thing which I recollected, but the code is written for Python three to be, I mean, that's because the developers who work on this care about abstractions, generality, and everything. If I were to read, I'm not sure I might have done it like that because I think I would prefer sometimes simpler straight forward approach. But some people prefer it more abstract, general, and overly scalable things. So it's both Python three, Python two, and anytime it can be switched to Python three. So this one plays the Google's business stuff that seems to be written in Python. And the build file, which I showed, the SRC common, Java, com, Twitter, common, examples, ping pong. See, I'm opening a file by name build file. But look at the syntax of this. Imagine that this is a function call, and the function call has a name parameter by name name. And this is the second parameter by name dependencies, which is like a list, and each of the elements of the list are a parameter with a function call by name pants. This whole build file is a compiled Python file. And how it works is, it traverses all the build files and then PYC compiles them. So if you, once I compile that find type main data PYC, you will find that there are build.pycs also, yeah, here. There's a build.py, so the build file itself, which has written like a text file for writing your dependencies, those are Python files themselves. So it can be compiled and then made into a single compiled object, which can be integrated. Choice of languages of Python is historical. It was chosen before I joined. Probably, it's a very good idea. I was chosen before I joined, probably because Google's blaze was using Python and possibly because you can write your, you can write your configuration build file as a language itself to support multiple. And since it's a language, you can do a lot more things in the language instead of just writing your configuration file. And bootstrapping, Java has the cost of bootstrapping. Okay, Java processes are very fast, but it has a cost of bootstrapping. Like when you write a tool and then you run it, it needs time to start. Whereas Python doesn't have, so these are some, all technical reasons, like, perhaps, yeah, I think. Is this a piece of the Scala view tool for this compatible SVT? It uses SVT, but it uses Zinc as the thing which are comparable. There's a good article. It's called, is it TypeSafe which is creating the Scala compiler? It's called TypeSafe. Yeah, TypeSafe. So let's TypeSafe Scala Pants, okay, sorry. Zinc and incremental compilation, okay. So this TypeSafe company which creates a Scala, I don't know how they are associated with that, but they are majorly creating Scala tools on it. They mention about using Zinc to increase the speed of Scala compilation. And Pants enable Zinc by default. And I think here, yeah, it's a Zinc has been already integrated with Scala, Maven plugin that's currently being integrated with a new build tool developed by Tipper called Pants. It increases the speed by a couple of points. Major speedup is gained more by caching. Like, once your build is created, like here, inside the Pants.d directory, sorry. Here, there's a whole lot of things which are cached. And then the caching seems to work in effect. Second time, there's no compilation, it's just copied. And if there are multiple developers which are developing, and if I and you use the same module as in the Shared library, and if my Pants build compiles for me, when it comes for compiling for you, it will not do it because it can simply download it from the network for you. So, a lot of tricks in caching and stuff which is used to speed up the process. Yes, not just local, but remote as well. The open source Pants definitely has it. And that's how the Foursquare, which is a 100% Scala shop, seems to have increased their build speed when using Scala. Cool, thank you, thanks. Yeah, sorry. Okay, it's a Shah. The Shah is calculated from the Git Shah at the moment, as well as the path of the... If you see the names of it, I think you can figure it out. Like, there will be various different Shahs which will be computed and then met for different things. It's basically, it's the name, path, and Git version number. All take place together to create a unique artifact cache. And I think that's it. Okay, cool, thanks a lot. Thank you, thank you for listening. Thank you. Thank you.


Do you want to let me know when you start recording? I think we're going, so go ahead. Okay, so let me start the presentation. So, hey everyone, my name is Thiago. I'm a back engineer for the Create team, and I want to welcome you to the first deep dive of the Create team where we are going to be talking about the pull mirroring system. So, let's jump right into it. So, the purpose of this talk is for me to share my knowledge of the pull mirroring feature with the entire GitLab team and to make this deep dive session a reference for everyone that might need to work with pull mirroring in the future. Here's the table of contents. I'm going to give you a couple of seconds to keep this game through it. Moving on, we will not talk about the mirroring through SSH, push mirroring and bi-directional mirroring due to time constraints. So, what is pull mirroring? It's a feature that is available on GitLab starter and bronze tiers. And basically what it does is it automatically pulls the changes from an external repository into a project in GitLab. We make an effort to keep a healthy mirror synchronized with external repository at least every 30 minutes. And a user is also able to update more often by using a functionality that we call update now, which we also make an attempt to handle common failure scenarios gracefully. It's very useful for teams that have a canonical version of their code in an external code hosting service and want to have a secondary version hosted on either GitLab.com or their own instance. So, for example, users can have their code hosted on an external code hosting service and let's say that they want to leverage our CI service, they can set up a pull mirror that is captain sync and it runs all the pipelines that were configured for that project, which is a really common use case for this feature. So, let's quickly do a demo. So, right from the start we have a really basic project that I created on GitLab.com and I have a bare bones project that I created on my own instance and so we're just going to quickly copy the HTTPS URL from this project and go to settings repository and we're going to go to mirroring repositories and right from the start you start seeing a lot of options that are made available to you by using the mirror repository so you can just copy the URL and you can see that this URL can either be using HTTPS, SSH or Git all of them are supported. You can also choose the direction of the mirror so it's either a pull or a push. In this case we want to synchronize into my own local instance not push towards the external one so we're going to select a pull. The authentication method will not be necessary because we are dealing with public repositories people privately probably require an authentication and then we have a couple of other options mainly the mirror user which is going to be the user that will be the author of every event that is created due to the mirroring. There are also options such as overwrites, diverse branches this makes it so that when you're trying to mirror and both versions don't have this both versions have a diverse branch it will override that one instead of just failing and not mirroring at all. The trigger pipeline you already mentioned what it does and the only mirror protected branches so based on the specific set of rules that you put for protected branches you can only mirror those branches it's really useful for instance for something that we call the bi-directional mirroring that I won't get into on this call. So let's go ahead and create a mirror repository and as you can see created a mirror for us and if you refresh it needs to be updated straight away since this is a really small project there you go, it updated 13 seconds ago and if you go to the main page of the project you can see that we now have the contents of the external repository and another cool feature that we have is for instance if I go to the main repository and make a change into one file so if I go onto my local instance and I go to the mirroring settings again there's this button that we make available from the update file so we can click to basically update the mirror as soon as possible so let's click that now what does that do exactly just set the next execution time to now? Yeah exactly it sets the next execution so the next project I will get picked up by the scheduler I will get into into more detail further down the line but basically what it does it passes everyone in front of the queue so that it's the next project being picked up and we can go onto the dashboard and you quickly see that hey everyone the update, the project has been updated so yeah, on with the presentation so there are a couple of key factors that kind of make this future all work so these are the ones like the capacity, the transition of states the state management and determining when a mirror update will be attempted again and I want to share with as well like some key metrics to kind of give you a scale of how this operates on github.com so we have over 50,000 mirrors being updated within the last 30 minutes so it's a pretty large scale so let's first talk about one of the key factors what it is, it's basically a reddit set that contains the IDs of all the projects that are currently waiting to get updated or are already being updated the total capacity is a fixed number that can be configured by any github instance administrator and it is basically used as a way of limiting the size of the cycle queue so that you don't schedule a lot of projects at once it's mostly like a bottleneck that we need to do but if you want to have your mirrors updating as frequently as possible I suggest that you put the capacity a bit higher than the concurrency that Cyclic has configured that will ensure that Cyclic always has jobs to perform and it will in essence translate into more frequent updates if you want, on the other hand if you want to have less frequent updates and you don't want to have Cyclic performing jobs all the time you can lower the capacity and instead of having the bottleneck of the actual Cyclic concurrency being the bottleneck you will instead have the capacity as the bottleneck so the next one will be spent transition why would we pick one over the other? what do you mean? why would we have the capacity be the bottleneck instead of Cyclic? basically as I mentioned so we have a configured concurrency a yellow concurrency for Cyclic and basically there's this Cyclic tube that we have and it's basically managed by the capacity that we have if we make it lower and the concurrency is higher it means that it will pick up every job and we will run out of things to run in the capacity instead of always having project IDs ready to be scheduled at any moment's notice does that make sense? so ideally they're the same or similar to just keep Cyclic busy? ideally they will be a bit higher just to keep Cyclic busy if you make it way too high like if you make it like 5 million you won't make a difference because we don't have that many projects and it will only translate into a bigger Cyclic tube instead which is not what you want and usually we recommend to have the capacity set to about well in the case of kit.com to about twice the size of the actual concurrency Cyclic can take on at any time so there's always a full queue of new jobs waiting for Cyclic to pick up if you lower the capacity to the exact concurrency Cyclic and handle at a time and once a new spot opens up in Cyclic we first have to wait for the update or mirrors worker which you'll see in a bit to come alive again in fact there's another gap in capacity schedule another job and only then can it be picked up by Cyclic which means we lose precious seconds during which we could be updating another mirror so the capacity we use to always keep that queue filled up enough and there's always something new for Cyclic to do instead of having to wait for Cyclic to be open again before we schedule something but you'll see the code in a bit I know that Tiago is going to cover it Yeah, so now let's talk a bit about state transitions and by the way I didn't say this at the beginning but feel free to interrupt me at any moment if you have any questions I'm not currently checking the chat but we will have a state transition take a run so the state transitions so the way mirroring works is divided by five states we have the none, the scheduled, the started, the finished and the failed and each of them are responsible for performing a specific task for instance the scheduled state will be responsible for actually scheduling the worker that will perform the updates the started will mark in the database the point where it actually started performing the update job and the finished worker, the finished state will mark the time where the mirror successfully finished and sets the time when it will get updated again and the same for the failed but instead it will mark the time where the mirror finished unsuccessful this is also really useful information to use as we'll see later on to find mirrors that are inconsistent states with the overall system for example a mirror that is in the started state but that doesn't have the psychic job running attached to it so there's also the state management and basically the mirroring system has three focal points that tracks the progress of the new so it's the database, psychic and the red disk capacity states the database basically holds all the information such as status, the job IDs the timestamp set point at which the mirror started and finished psychic provides all the information about the mirroring queue and we are able to know based on that if the specific job has finished or not and with the red disk capacity set as I mentioned previously we are able to know if the project is we'll tell us if the project is waiting for it to be updated or if it's already been updated of course the objective is to make the system as self-healing as possible but there could be there are some scenarios where the system doesn't recover yet and we have to do it perform an operation by hand and we are planning on doing an emergency test about it soon so for example if a project in the database says that it has started we can check that the job ID is saved in the database and match it with the psychic mirroring queue to see if it's finished already so if you see that the job has finished already and the project is still in the started state it means that the psychic wasn't basically finished and wasn't able to communicate the database with the database in order to transition the state to either finished or failed and this is the case that we picked up and basically handled as we will see later on so another thing is to know when should the mirror get scheduled and for this we developed a formula that basically takes into account the time that it took to update the mirror the last time and the number of retries that were necessary to actually get a successful view out of it so we can move into more detail into the formula we currently have this back off period and the jitter because previously this is a bit of legacy that's currently in our code base but the objective was to spread the load a bit more but as you see further down the line you have the minimum delay and the maximum delay which kind of bound the time stamps a bit the main thing that I want to focus is the retry count which takes, it's a huge factor in when we are actually going to update the mirror next usually the mirrors finish the finished work in two seconds so it's really fast and in this case we are going to have a 30 minute delay which is the minimum delay that we can have and as I mentioned we want to penalize mirrors that fail often from running as frequently as the healthy mirrors so we always want to prioritize healthy mirrors if the mirror is for any reason which a maximum number of retries we are going to put it into a specific state of heart fail where it will no longer get updated and instead make a request to the user to actually solve the problem by hand so this is a basic run down of the workflow so the scheduler will wake up every minute it's a front job and you'll pick up all the mirrors that have the next execution due for an update so previously to the current time that we are on it will schedule all the mirrors until there is no more capacity available at the moment and after each mirroring starts it will basically fetch all the changes from the provided remote URL and update the respective branches and tags with the new information after updating the mirror we will always remove that project from the capacity list and set the next execution time but then we will have like two outcomes you can either finish or fail and if it finishes it clears the reply counter basically sets it to 0 again and if it fails it gets incremented and if it surpasses a certain limit as I said previously it will get a heart fail so this is the basic architecture of both the state transitions and the workers and how they communicate so you can see that we always start with the non-states but we transition into the schedule then it started and then it can either be a failed or finished mirror that will sort of go back into the scheduled state again and in the life cycle update all mirrors worker will get rescheduled every minute to wake up it only communicates with the project's database it requests the mirrors that are ready to be updated and basically schedules a project import schedule worker for every single mirror and then that schedule worker will as we will see in the code base it will schedule a repository update mirror it's the only purpose of that job and then the update mirror worker will actually perform the work of updating branches and the types and all that so far are there any questions I'm not being able to see the chat for some reason are there any questions so let's move on and do a quick code guide oh ok for some reason you can see the chat so let's do a quick code guide I have all the files that are kind of important for the whole new system to work in our web ID the first one that I want to touch on is the project import states so you can see the first thing that you can see is that the table name is actually called project mirror data and this is also a bit of legacy code that we have previously only worked for mirrors but we have since migrated imports and ports as well to this logic and every project will have a project import state so why we decided to separate the new import and ports logic into their own model was because the project's table was getting too big so reads and writes were getting a bit too expensive and we could make this a bit more lightweight as well the second thing the third thing that I want to show you in this specific file is this transition here so the transition from the finished or failed state to scheduled will actually perform this method called add import job that we can see here so add import job over here that you can see that if it's an import and the repository doesn't exist it will actually schedule an import or a port but otherwise it will be scheduled to a repository update which is the actual job that performs the update operation and we can also see the company part so there you go the super basically calls this portion here but it's either an import or a port another yeah so this is basically what it does it returns a job ID and we update the current project with that job ID that will basically keep track make sure that when the job finishes it gets cleared and otherwise if it's in a started state we are going to use this job ID to make sure that the job is still performing well so going on to the part of the project import state let me see so this portion as I mentioned you can see that we decrement the capacity when you transition into a failed or finished state and we increment the capacity when we go to the scheduled or started state and again from a started to a failed or finished state you only see two differences it's the increment we try count as opposed to the reset we try count when we finish and we will set the next execution timestamp and on the failed we only set the last update that when we actually finished we set the last successful update that as well to mark the successful so here I would also like to show you another method that sets the next execution timestamp it's also in the slides but you can see here we actually use this method called base delay here that has the formula that I showed you basically using the current timestamp minus the last update it started at the duration of the last year on the project model there is also which is really useful for debugging practices as well but it's also used for scheduling which is the mirrors that are ready to synchronize and we do this by having a preset which we easily provide the time that we are in now and we fetch the projects that have a timestamp that are previous that timestamp another module that is really important particularly in the core of the entire feature is the githlite mirror module it has the logic of scheduling front jobs managing the capacity checking the available capacity and so on so you can see here that we are configuring the front job of the scheduling worker called update all mirrors worker and it runs every minute we can also see that we have the logic of incrementing and decrementing the capacity that checks the available capacity right now on the ready set so now we are going to move into more the workers and how things get scheduled and how we have a better notion of how the transitions of states work so in update all mirrors worker the only task is to grab projects that are ready for an update and schedule them and fill the capacity with them so you can see here that we save the available capacity into a variable we fetch the projects that are ready to be synchronized here and we do it in batches and we take as many projects as we can as the capacity lessens state and in the end we just perform schedule on import schedule worker to build projects so we just did that after that we will transition into the project import schedule worker which it's only like it's a really simple worker it's only task is to actually transition the project in scheduled state which will be responsible for scheduling the update mirror worker which is actually the one that performs the work which we are going to see right now so the repository update mirror worker basically transitions the project from the schedule from the started state and then calls the update mirror service which is the one that will fetch the remote and update branches and tags from the external repository at the end it will finish or file and we handle those in these methods so we either finish transition to the finished state or mark as file this method actually isn't possible for filing the mirror it sets like the error message and so on so the update mirror service which is the one that is called by the repository update it's a really basic service it basically just fetches the mirrors and then updates the tags with them and updates the branches there's a bit more logic because of the as we mentioned that the protected branches only mirror protected branches or override divert branches so you can see the logic mirror so only mirror protected branches you should skip the branch or not and the way we handle the divert branches because of that feature as well so the other really important worker that we have is it's also a cron job that is called stock importance worker it runs every 15 minutes don't get misled by these 15 hours this is just to as a mark for all the workers that are dependent of this stock importance worker so you will be sure that every mirror job will time out past 15 hours actually attempting to update we are planning on decreasing this number by the way but right now is what we have yeah one thing one thing that you would want to know is the project with job ID we are going to mark them as this portion here without job ID is no longer possible and I am doing an emergency task to remove this portion of the third base so this method is no longer used but this portion here is actually really important what we do is we grab all the projects that are in the scheduled or started state and check for their job IDs and when you get the job IDs we actually use this method called completed job IDs that are like a psychic base we have internal tooling that keeps track of when a job has finished and we can use that to check if the jobs for specific mirrors have already entered and if they ended and we are in a scheduled or started state we should mark them as prior here and it's basically that so yeah back to the slide presentation so are there any questions? I know this is a lot of information to take in so please bear with me and let's start diving a bit more into the troubleshooting so here is an example of an unhealthy mirroring system you can see by looking at for instance like the project mirror updates over here and the mirror updates at least 10 seconds over here you can see this huge mountain of projects like 15,000 projects that are not getting updated at the right time coincidentally the project important schedule worker and the repository update mirror worker are not performing any work during those times or at least very minimum work so in this specific case what happened was that this not important job worker wasn't timing out and that's why it was timing out so we weren't able to actually mark projects as file and that's why after the situation gets solved which is why you see this huge guy in the project mirror updates over here and the schedule start capacity stays the same is because we are not being able to mark those failed projects as failed because it's talking about job worker once again so there's a link as well if you want to check in a bit more detail these were paragraphs they are really interesting and yeah so for troubleshooting what I usually do is I always refer back to the project important states model or the project mirror table and check the last error message, the root write down the job ID, the last update, time stamps and yeah work and it is really useful to kind of graph that information into an excel sheet or something and kind of find pattern inputs it was really useful when I was developing this before so you can also check the available capacity by using the the GitLab mirror method it has a lot of functionality that is useful and you can take a lot of inspiration from that module for debugging purposes and you can combine this method with the project mirror to same time dot now to actually see if there's actually projects to fill the capacity and for some reason we are not filling that capacity or something like that that might indicate that the schedule is not working correctly which is something that happens in the beginning of this feature so you can also check the status of the workers for each mirror in the schedule for a started state as I mentioned the whole the job ID in the database for each project for a started state and those job IDs you can actually use this command called gitlabsitics that is job status job ID and pass it to job IDs and it will basically tell you if they are completed or not you can also retrieve the project ID that is currently in the red set by using the sMembers method it's useful to do projects that could be stopped or really for instance a finished or a failed project should never be in the capacity they should always be removed so if for some reason a project transition into a failed or finished state it might indicate that the service is not communicating well with the Redis without Redis so for clearing data inconsistencies so when the database is inconsistent the whole point of us having this distributed model of knowledge let's say is because we want to keep it we want to make it self-evident and we want to keep this data as synchronized as possible so for instance when the database is inconsistent in site like a project can be in the database starting in the database but site already finished the job we can use stock import jobs or we can schedule stock import jobs which will look at the job IDs maintained by this github psychic status and usually this happens when the actual update worker times out for any reason you can also have the capacity set being inconsistent with the database and site, for instance if the project is finished but the project ID is still present in the capacity set which was the scenario that I mentioned previously right now there's no real self-healing way to solve this, this has to be removed by hand by removing the project ID from the capacity set you can do it by either running the decrement capacity and feeding it the project ID if it's something that you're seeing that it's occupying like the entire capacity it might be worth just deleting the whole thing and starting from scratch this though should be taken with its precautions because what this basically will do, if you remove all the project ID from the capacity set you will basically say to the psychic hey you can schedule more jobs the psychic you unnecessarily so now let me show you what a healthy new system looks like and you can see that it's a huge difference so you actually see that a project import schedule worker and depository update worker are consistently performing jobs you see that the scheduled or sorry capacity doesn't it's not sale so it's not stuck with projects and the two most important graphs that we have which are the project new updates overview and the new updates at least 10 seconds are really healthy, you can see that the amount of projects that get 10 seconds overview of an update from the actual scheduled state from the actual schedule times it's minimal, it's really minimal so yeah, these are also some really useful things that I use all the time so the documentation for the full new system, the state machine actually corrects the documentation it's also really useful because the whole system relies on the state machine working properly there's also some guidelines to troubleshoot full new made by the infrastructure team and I'm planning on doing a couple updates on those guidelines and there's also an issue of dynamically determining a new update interval based on the total number of new so we want to get rid of the lower bound and the upper bound scheduling times and instead we want to make it dynamically based on the information that the system has there's again the project new graphs that has a lot of information and please if you have any questions about new systems, feel free to ask in the VAT channel at any time are there any questions? I think you said that the mirroring by default runs every 30 minutes is that a value that can be adjusted at all or is that set at 30? I don't think so, the minimum delay right now is set to 30 minutes so that's actually a really good question I don't know if you probably should change it because as I mentioned we are trying to make it dynamically so getting rid of the minimum delay and the maximum delay entirely I'm definitely going to discuss this yeah right now the minimum delay is not configurable but like Diego said the intention is and then pretty much always has been to have that kind of automatically adjust itself to the actual psychic capacity and the actual time so that it never tries to update more than it can actually handle but also that means as the amount of if the amount of mirrors grows but you never actually adjust your psychic capacity automatically the time in between two updates per mirror would go down to not overload the system too much but we didn't have time to implement that at some point when we did need to fix the dire situation at the time because we had far more mirrors than we were expecting and we were not hitting any kind of reasonable update frequency on those so we kind of implemented that hard-coded 30-minute limit and we still haven't kind of made or found the time to switch to the dynamic approach we had originally wanted to deal so right now it's hard-coded to 30 minutes but the idea is that on github.com 30 minutes is pretty much the ideal value so even if we installed the dynamic stuff it would land on that 30-minute frequency but on like a smaller instance the update rate could be much higher you would still want some kind of lower bound there so that you don't try to update the repo every five seconds because it happens to be the only mirror in the system so psychic can handle it of course that's not somewhere you want to go either but the idea is not for this to ever be configurable as in just pick a number and github will try but more for it to be github will do its best within the resources you've given it because ultimately github knows and has the information available to it to figure out what number it needs to be updating it so that it doesn't overload but still uses the capacity and the resources to the full to the full possible extent and you also make available the update now functionality especially because it is like you knew that there was going to be a delay and if for any reason a customer actually wants to update a bit more we can also use that update now but right now I think Tiago mentioned earlier that we have the ability of someone can choose to host their project somewhere else but then use kitlab just for CICD and we have an API endpoint that will effectively do the same with the help button does so you can call an API endpoint that will immediately trigger an update which like Tiago mentioned earlier effectively is set in the next execution time which means that the next pick of the update all mirrors work will pick it up to be updated immediately and when you set your kitlab project up to be a CICD project for a project on github for example we use the github webhook which will be called every time someone pushes to github to automatically trigger the update now endpoint on the githlab side of things so that the mirror will sync immediately and the pipeline will run immediately when someone pushes to github and we are actually smart enough to also report that pipeline status back to github so that's really the way of using github CI just like you would any other standard on CI solution like you know circular travis but we've built it on top of this mirror ring functionality and that update now feature hooked up to that github webhook that was very helpful thank you and can you review again how you determine when a mirror is hung can you just go through that again sure so there are the most common case when the mirror is hung it's for some reason the update worker so let me go back to the architecture it might help you so this graph that I showed this diagram that I showed earlier you can see that the repository update is the actual job that is responsible for updating the information on what we pulled from the external repository and this can actually if it can happen that there is a timeout or for some reason the job crashes and we don't actually get to transition the project onto a finished or failed state so we will stay in a started state but it will still hold that job ID but the project already the job already finished and what we do to kind of catch these scenarios here is we have the stock import jobs worker which I showed earlier here so we have this jobs worker that basically catches all the projects that are in the started or scheduled state and grabs their job IDs so let me try to show you again so we grab the job IDs for those projects and we call okay, it's here so we find the job IDs that are already completed and if it's completed that means that we have a project that should be marked as failed because it shouldn't be started anymore like the job finished so at some point there wasn't the communication to the database to transition the project into the next state that makes sense okay, that helped thank you again, if there are like if you have any other questions you will be able to drop by the peer channel anytime, I can help whenever necessary okay, I appreciate that, I may have some clients that will have questions like that so I appreciate that, thank you so, are there any other questions? okay, so if there are no other questions, I want to thank you for listening to this talk and yeah, I wish you all a great day bye thank you