This is why narcissists create enemies. Narcissists are always looking to create enemies by being actively opposed and hostile to people. They're always looking to weaken or harm you in some shape or form to cause you severe physical and mental suffering. As though it's their only purpose in life, as though it's their only reason for existing, which is why the only thing they really know how to do is to assess the risks, possibilities or effects of a situation or a cause of action to make a judgement, forecast, projection, prediction or expectation. And they're very good at it because they've had years of practice at manipulating their targets. So they know how to assess a situation and weigh up the potential benefits of a hunt and analysing the potential cause or risks at the same time. It's like an animal. They operate from their reptilian brain. So they know how to coordinate an attack and they anticipate a response from their prey. Predator animals in the wild have been found to frequently form coordinated hunting groups where they attack in line formation. They engage in collective behaviour, especially under conditions of considerable informational constraints. And narcissists are no different. They run schemes. They operate from large scale systematic plans or arrangements for attaining a particular object or putting a particular idea into effect. They make plans in a devious way with the intent to do something illegal or wrong. A system. A programme of action.  The winner of a competition. The one who achieves a position of authority where they can be in charge and give orders and make decisions because what they're really seeking is unlimited access to power and control. But they're not really as dangerous or frightening as they may seem. They're not really all-knowing and aware of everything. They don't possess unlimited knowledge of awareness or understanding. Even though some victims may see it that way. Because the reality is that the root basis of their disorder is the opposite of everything that they portray themselves to be. It stems from insecurity. Which is overcompensated by selfishness, arrogance and entitlement. When really they're actually very weak and vulnerable. They're easily damaged and destroyed.  They take cues from their target. Which is something that serves as a signal for them to bring in their performance. So they actually have no power of their own.  It's all make-believe. It's delusion. They're not really who they say they are. They have nothing to fight for. They have a false sense of pride from their belief in a separate self. A false character which they use to fool you. And that is what they're defending. Because it is essential to their emotional survival. And it works for them because they compartmentalize their situations and interactions. They divide them into sections or categories. They separate them into parts. And they do not allow those parts to mix together. They separate them into isolated compartments. To where they live in multiple separate and very different lives. Where they may appear to be a different character in each. And one character may be the exact opposite to the other. Because they will continue with whatever is working for them. So if one character has failed to secure the outcome or result that they would like. They will drop that character and then they will rebuild another character somewhere else. And they can switch characters whenever they choose to and act their own well. So you know that they are engaged in unfair and invalid pieces of play. Involving interference with and illegally hindering their opponents. Where they're continuously getting involved and influencing situations that should not involve them. Because they're despicable and contentable. They're dishonest and unfair. They're unsportsmanlike. They have no real pride, dignity or self respect. So anything is fair game to them. They consider you a reasonable target to be chased, criticized and attacked.  And to memorally exploit and cheat a system or a situation. Slightly and dishonestly for their personal gain. And they won't hold accountability for anything. Because they can switch characters whenever they choose to. They can take up a very different way of life somewhere else. And when they do that. They may forget things that they've done or experienced in the past. Which acts as a coping mechanism to a lot of the failures that they've had. Because they can just build a new character and personality. With a different situation and experience. Which is unattached to the one that they had before. Because it's just whatever people are reflecting back to them. And the new people they're dealing with are only seeing this new character that they've created. And this new character is being accepted. It's not being judged. Which is all they really care about. Because they just want to be praised and recognized as being good and important. But when they're targeting new victims they're looking for weaknesses. They're looking for disadvantages or faults. They're looking for something that people are unable to resist. Or something that they like excessively. When a normal person is not arranging or preparing things in that way. They're not seeking to find people's weaknesses. They're not looking for what people don't do well. But this is how narcissists are wired to think. Which is why they can only ever be your enemy. They can only be your adversary and opponent. This person who is engaged in conflict and competition with you. So they will never see you for who you really are. They will only see you for what they can exploit and take advantage of. Which is why they despise power and strength. They hate anything that is able to withstand great force or pressure. Anyone who is mentally strong and has the power to resist attack. Which is why they hate seeing you giving orders making decisions or enforcing obedience. Because that puts you in a position of power. Which is a direct threat to them. So they have to criminalize and discredit you. They have to make your activities immoral or illegal. They have to harm your good reputation. And cause you to lose your dignity and respect. While also discrediting any opposing force that is in a position of authority. Whether it be doctors, lawyers, police officers, judges, psychologists or psychiatrists. They have to discredit them or befriend them. And turn them against you. Because otherwise those people would call them out. They would draw attention to their unacceptable actions of behavior. And if this was effective and commonplace in society. Then narcissism would not exist in the world today. Because there would be nothing that narcissists could do about it. Which is not what they want. They do not want the shoe to be on the other foot. They do not want to be in that position. Which is why they maintain right to the edge of what society deems to be acceptable. Of what they know they can get away with. And they manage to befriend a network with authority figures and public officials. So that they can protect their image of reputation. And it's really the only reason why most of them aren't in prison. Because if you look at their behaviors. They're really no different to what you might find in a jail cell. They're disordered people. They're out of control. They're borderline crazy. And it's because they're very weak, fragile and insecure within.  They habitually seek to harm, intimidate and coerce those who they perceive as vulnerable. And they act unbothered and unmoved. They show a lack of concern and interest. As though they don't feel any agitation, worry or annoyance. When in actuality when they experience a narcissistic injury. It makes them very upset. Because they're very envious of you. They have this deep rooted resentment. A browse for your possessions, qualities and abilities. To where they can never have a break or take a rest. Because they're constantly looking at you. And wondering how they can be successful, determined and intelligent. Instead of being weak, speaking nonsense and lying. Which is why they typically believe that other people are the same as them. They can't understand that you could be a genuine person. And you're still able to be successful. Without having to indulge excessively in sensual pleasures. Such as bad sexual behavior. Drinking too much alcohol or taking drugs. Or dishonest or fraudulent conduct. Or manipulation, alteration and falsification that they engage in. They can't understand it. They can't see how that is even possible.  By invalidating you. By trying to disprove your arguments, statement or theory. And depriving you of legal efficacy. To discredit you. And to make something not true or unacceptable. Which are all just tactics to minimize you. To reduce you to something unwanted and unpleasant. To represent you as less than your true value and importance. To belittle and play down who you actually are. So that they can give control. And it starts off delicate and precise. Where they use indirect methods. So that it's difficult to analyze or describe. And at times may even play it out like it's just an amusing story or trick. And so it's intended to be funny. So that it doesn't make you want to run away. Where they've always got to have a psychological advantage over you. They've always got to do something better or more extreme. They've got to get the better of you. They've got to score a lead over you. They've got to succeed in being a point move or step ahead of you. And at times they may even be lying. Just so they can downplay the struggles and obstacles that you went through. To where you still manage to succeed. Because they want to break your pride. Your consciousness of your own dignity. To where you're proud of a particular quality or skill. Even though it may just be something that you achieve from having to deal with them. But yet they want to take that from you as well.  Anything that you acknowledge and recognize about yourself.  Even though you may have done it all on your own. And you never received any assistance from anyone. They will still find a way to downplay it. As though anyone could go through what you went through. When they're just attacking your pride to minimize your value. They're trying to tell you. You're not who you think you are. And what makes it so effective by them doing it indirectly. Because then you haven't even got the option to oppose their decision or theory as mistaken or wrong. So they're just completely shutting you down. They don't even want you to speak. Which is why at times they will even ignore you. They will refuse to take notice of you. They will disregard you intentionally. They will treat you as a low priority. As though other people are more important. And they will pay close attention to them. And attend to their comfort and wishes. They will be very polite and respectful. But only because they want you to feel an important, insignificant and irrelevant. They want you to feel worthless. To change your position, direction and tendency to who you know you are. And to what you know you've done. Because that is how they succeed in controlling you. By breaking you down and yet still managing to keep you around them. By constantly questioning the way you operate. To make you doubt your own abilities. Because if you fall for it and you still manage to maintain and go through it. And you're still dealing with them. They know they've got you. Which is what they want. Because a lot of times they target you only to hurt you. To conquer you and break you down. And then leave you. That's the only reason why they're even there. And they will often use these tactics to confuse and make their next victim uncomfortable. To make it seem like you're not that impressive, attractive or desirable. Because they're envious of you. So that's all they're going after you for. To strip you down of your worth, value, pride and dignity. So that they can feel like they're this top dog who conquered you. They're more special, important, superior, desirable. Because now they've brought you down. They've brought you beneath them. And as long as you're around them. That's all they're going to do. Because of their envy of you. It causes them unrest where they must act. They must do something to take you down a notch. Because otherwise they can't feel comfortable. It's like in London. Which has been called the asset attack hotspot of the Western world. They engage in these premeditated attacks. Of throwing corrosive acid on the victim's face and body.  To where it may cause lifelong scarring, physical disfigurement. And in some cases permanent disability. Including blindness and immobility. And it's all because they're envious and jealous. They can't keep their emotions in check. Their feelings take over them. Aroused by your possessions, qualities or luck. So now they're out to get you. But they disguise it as love away relationship. Well that's just so they can gain access to you. So they can have a better shot at taking you down. Thank you for watching. If you found this video helpful please give it a thumbs up. Share your thoughts in the comment section. Hit the subscribe button to receive the notifications. If you would like to support the channel. You can donate at payable.me. There's NARC Survivor. You can book a one on one with me on my website. It's NARC Survivor the code UK. Thank you for watching and I'll talk to you soon.


Good afternoon, everyone, and welcome. This is Melissa Arma with the stocks. Why should I'm here today to review this buy? And the reason I wanted to do this is because yesterday when I was teaching the class, I ended up flipping over to the market and we were talking about it and, you know, this trading action that has happened for the last 14, 15 days is extremely bullish. And I've been talking about the fact that I didn't think we'd pull back. And we're not going to before we move higher again, but we are really going to explode. So I thought it would be good to just do a video to remember what I see here. And it didn't even dawn on me until I taught the classes today. And I was like, wow, I just want you to look at this here. One, two, three, four, five, six, seven, eight, nine, 10, 11, 12, 13, 14, 15, 15 days. For 15 solid days, that is a long time. That's three weeks. That's three weeks actually.  For three weeks, the market has traded with, you know, very tight days. That is not normal. That's very unusual. So instead of pulling back, which the only way that would happen is would be with selling or shorting action, which, you know, again, no one's selling here because the market has just decided to go higher. And people that sold already sold when they were into panicking here, the Brexit and earlier in the year. There are some people that have been shorted this market that are shorting it here are still waiting to see where it acts to probably have to stop at B20, which is ridiculous now. Just made a brand new all-time high again. And it's fine, but I want to, in fact, we did it on Friday. This is a new high for the market. Anyways, when I noticed this yesterday, I thought, wow, not only did we not pull back, we actually had very tight trading action. It's like condensed. This is going to break out. It's going to explode. The market is going to explode higher. Any second, any day, any moment. Monday is usually a slow trading day for the market, but it could be tomorrow. And it certainly looks like it's going to be this week. The longer we act like this, the bigger the move up it's going to be. That, I can tell you. So I have no idea when we explode, but we are going to explode over the high. And originally I said, we'll see how we act at 220. That's not that far away. It's only two and a half bucks from here now. So I really think we explode over 220 into the next move. We are going to explode. Three solid weeks of trading in a tight range, an arrest range for the market is extremely unusual. So bottom line is we are set to explode. This is unbelievably bullish action. I know what you can't tell really, except for the fact that it is. And it is because it just is. Because we're not pulling back, because we're trading in a tight range, because of lots and lots of reasons. So I just made a great call here in the market. And sometimes I see things and I feel the need to write it down and record it for myself. And we'll just have to watch and see what happens. But it very well could be this week. But I tell you, if we trade in a range again for another five days for one solid month, we're even going to explode even higher. And we could run right up over 225, we'll have to see. So have a great day, everyone. Email me at melissathestockpush.com if you want more information on the Golden Gap Force. And have a great week.


Bains. Albert Lewin for Davis. Niasz inside the penalty area. Umarniasz! From zero to hero. Umarniasz for Everton. Back in the first-team squad. He delivered the goods against Sunderland in the League Cup. And he's done it here in the Premier League for Everton. With just under a quarter of an hour remaining, Umarniasz gets his first Premier League goal for the Blues. Snidel in. Kenny to get forward now for the Blues. Four men ahead of him. Kenny's ball in for Calvert Lewin. Davis went for goal by 4 for Niasz. It does! Surely that's in! It is now! Umarniasz! Can you believe this story? A man who was not part of his plans last season is very much the story of this. From 1-0 down, Everton are 2-1 up. Thanks to this man, Umarniasz.


The Canadian Northwest, here the adventurous sons of the Old World, came to trade with the natives of the New, and stayed to intermarry with them. This union created a new half-breed race, the Metis. Here for two centuries, the Metis lived and prospered, a law unto themselves. Then civilization moved westward, surveyors and land speculators came, bringing with them the laws of land and property. In 1885, resentful and confused, the half-breeds rebelled. Only a handful of fearless, hard-riding men in scarlet coats, the Northwest-mounted police, stood between Canada's future and the rebellion that was kindled across the border in a little Montana schoolhouse, where two men came to speak with the half-breed schoolmaster, Louie Real. Where Louie? I've got your message, Carbo, but to return to Canada now, that's death. What's up, maybe? Not for us. Fifteen years of exile haven't changed you, Carbo. Or you, I hope. You've got to come back and lead us. The Canadian government will never... You and me and Dorak here, we're going to be the government. We're going to set up new government, Louie. The half-breed coming to back force by the Ormwood. And I've got a gun that shoots a thousand slugs a minute. A Gatling gun. Go near the border. God! Do you remember that you lead by people? You can be the saint who leads them, Louie. All I want is the whiskey business. You realize what it means? The blood will flow like water. You won't notice it much. The mounted police wear red coats. Red coats! What'd you say? Red coats coming through Redcourt, running down streets. Len, he's missed! Quite a crowd. Looks like bargain day on the dust, Jim. Bargain day for trouble. Keep your eyes open, Ronnie, and your hosts are closed. Say it, he's missed! Quite a crowd. Looks like bargain day on the dust, Jim. Bargain day for trouble. Keep your eyes open, Ronnie, and your hosts are closed. Hey, do you suppose the inspector's right and Rielle is back? The inspector is always right. An hour for twenty miles away. Look at all those men. What do you suppose they got the guns? Santa Claus. Pull up here. Ow! What's the matter? Something hit me in the face. A stone. Oh, I see. Hello, Ronnie. I got good aim, yes? Not now. Not now. Ronnie, you don't like me to speak with you, Ronnie? Listen, Louie, I'm on duty. Oh, on duty? Then I don't say a word. I wait till I see you tonight, yes? Tonight, Ronnie? That's fine. Jim, listen, I couldn't help it. I told you to forget that clout. She's poisoned. Jim, I was only... Never trust a blue-eyed squaw. Look, there's the old wolf himself. Who? Dandier Rock.  Stay there till I come. Don't come on a stretcher. Shop now. Go on. Hello, Dan. Hello, Jim. Welcome, everybody. Quite a crowd in town today. A few plenty friends of mine. Is Louie here yet? Louie Reilly is best friend we got. And you'd better advise him to go slow. For 15 years, we go slow. Now we go quick. We give you 24 hours. Here. On this paper, we write what we want. If your government say no, we make no government. I'm making a mistake, Dan. The law is in the northwest now, and you can't fight the whole British Empire. 50 mounted police. Only 50. Well, Canada, only 500. We got many thousands of priests. Chief Big Bear. And Proford got 10,000 warriors. The Indians won't follow Real. Then maybe they follow Jacques Corbeau. Corbeau? If you bring back that whiskey-running killer, you'll deserve what you'll get. We'll want Corbeau. We're getting Corbeau. You don't stop us.  Why you tell me? I'm warning you, Shorty.  Shorty! Shorty! It's a boy. Shorty, the nurse called you. Shorty, do you hear me? A boy. Huh? Shorty, don't you want to see your son? A boy. How did he wait? I got son. I got tell my wife about this. Tim, come in here. I thought you were in trouble for a minute. I was. Sit down. You look busy. My sister's always busy. It's just a bandage for he-calls-legged. Who's still he-calls? Oh, oh, oh.  I will. Thanks. April, how can you deliver a baby? Set a broken leg. It'll look lovelier than a Christmas calendar all at once. Well, you're quite a picture yourself, Sergeant Brett. You look well in a frame. We'll look better in the same frame. Let's make it a family album. Let's water the horses. We've got to get Durox's message on the wire. Right. Then you look at Ronnie even his hair stands at attention. I don't know. There's a contrary streak in the Logan breed. You just know our own mind. You don't know your own heart. I think I do. No, you don't. I've made plans for this, April. Jim, Jim, I've been transferred to Nova Scotia. Nova! That place where the codfish grow? Orders are orders in my branch of the service, too. Oh, oh, oh.  I've been counting on you staying here with me. Don't you think that... Why, I think you're splendid.  But, Jim, you're too much the born soldier to ever be anything else. Now, listen. I have a bugle for an alarm clock and the eggs would click to attention at breakfast. Marry me and see you. But you're already married to the service. You see, I want a husband who wants to sit in some of the fire with me at night and hold hands and popcorn and do and so... all the silly things that you couldn't because they'd be tantal and tender. Instead of iron and steel. Oh, April, I don't understand you. You never understand me, Jim. Until something beats you to your knees. You're lovely. You see, you never even ask me. You just make statements. Well, do you love me? Well, I might, but I don't want to. So I'm going. When do you leave? Next month. We'll see about that. What if I can't possibly act on these demands in 24 hours? But I'll swim well over there in Inspector Cabot and scald half of Canada. What in business? If the Korean Blackfeet rise, they'll be far in blood from here to Bethan Bay. Cancel all leave, Sergeant Brett. It's ready for inspection after a belly. Yes, sir. Well, what is it? A man just rode into the stockade, sir. A man from Texas. He says he must see you at once, sir. Thanks, Mr. Evening Inspector. My name's Dusty Rivers, sir. Texas Rangers. I've been sucked up here following a man we want for murder. Oh, then it tells Sergeant Brett I want him. Here's my rangers' commission. This is my appointment as United States Deputy Marshal. I've heard of your organization, of course. Thanks. We've heard of yours, too. Thank you. You've come at a rather difficult time, sir. Inspector. Sergeant Brett, this is Mr. Rivers of the Texas Rangers. Yes, sir. He's here on a police errand. See that he's fed, deleted, and off at a bath. You're right, hospitable inspector, but I want to get started on my... Sergeant Brett will take good care of you, Mr. Rivers. Gentlemen, this is Mr. Rivers of the Texas Constabulary. Texas Rangers. Seems another criminal has leaked out of the United States and he's after him. You can sleep in that collapsible man trap here, Sheriff. It's not, Sheriff. Good night, Sheriff. Revelli is at six o'clock. Re what? Revelli.  Sleep well, we'll get your man for you. Well, I was counting on getting home by the fourth of July. Ogan, come over here. I want to speak to you. Good morning, Jim.  I know. Sorry. Ronnie, listen, I've warned you about that new bet, girl. My personal business is my own. It's my business to keep you from tying to a feeding little clooch. I ought to wipe your jaw, but...  You're against that girl because she's part Indian. No. Because she's all bad. You'd lose your eyebrows if I married her, wouldn't you? What? Why not? You know who she is. The Jacques Corbeau's daughter. Corbeau... What of it?  Sergeant Brett! Well, what is it? Trouble, sir. Miss Ogan just wrote into the stockade. We've got Constable Spettman and Grove in the wagon, sir. Grove was dead. Where did you find these men? On the Red Dairy Trail. I was on my way to a case. Tenton, can you hear me? What's happened, Tenton? Trains, sir. Carrying a gaffling gun. Gaffling gun? Somebody's running guns to do rock. Did you recognize the men? No, sir. But there was a funny, porn-looking saddle. Excuse me, Inspector. Can you describe the saddle? Did it have a big silver horn? Because I ever saw silver all over it. Silver? What that boy saw was a Mexican churro saddle. Probably written by the man on after. Jacques Corbeau. Corbeau? Sergeant Brett? Have the bugle a sound boots and saddles and form up a trail of plenty men. Yes, sir. You'd better stay at the fort, Miss Ogan. Natasha's not safe. Thank you, but there are several patients in my hospital there. The first team is working for you, Miss April. Inspector's orders. Thank you. I'll hit him up for you, Constable. Get over, boys. My name's Dusty Rivers, ma'am. Dusty Rivers is a contradiction, isn't it? Well, I expect you've never been to Texas, Miss... April Ogan. April? You wouldn't be fooling me, would you? Thanks all the moment, Sergeant. Take your place. Oh, Sergeant, which part of the town do you live in? Oh, Sergeant, to which part of the parade do I ride him? I'm sorry, but you're not going with us, Mr. Rivers. Well, I'm sorry, too, Sergeant, but I am going with him. Corbeau is wanted for murder here. He belongs to the amount of police now. He belongs to whoever catches him first. Nevertheless, you're not going with him. That's all already, Sergeant. Good. Sections, right? Are you all right? Right. Sergeant Brett, hold the patrol. Sections, hold. Miss Miss Ogan, Sergeant. Very good, sir. Captain, the old Mr. Rivers report to me at once. Sergeant Peel, dismiss the patrol! That roll is dismissed! Regine, no patrol is to leave until we're re-enforced by Colonel Audeyne. It will take them a week to reach us, sir. Corbeau will have Big Bear on the warpath before then. And the Blackfeet. The order prohibits sending out a single scout, sir. No. Then I'd like permission to proceed to Big Bear's camp alone and try to keep him loyal. Alone? A particular job, Sergeant, but permission is granted. I'll leave it once, sir. I'll leave at the same time. Where do you want to go, Mr. Rivers? Well, I figure to pick up Corbeau up in Batorque. Why do you think he's there? Well, that's Riel's capital, isn't it? That's where I'm going. Hmm. Very well. Good luck to you both. Gentlemen, I hope you keep your scouts. Dusty Rivers of the Texas Rangers is going out alone to bring back Jack Corbeau. This is a dangerous job. But Mr. Rivers of Texas doesn't mind mixing it with a little pleasure. Miss April Logan's wagon is just ready to pull out for Batorque with Mr. Rivers handling the reins. You know, I wish Batorque was a thousand-mile ride. This speaks too hard for even a twenty-mile. Lady, to me, it's just like sitting on a cloud. Who's this, Jim? Are you riding with us, Jim? No, I'm riding the other way. Too bad, Sergeant. I'll send you a postcard. Just a moment. You can't make an arrest in Canada, so I'm sending a man along with you. Mr. McDuff, Mr. Rivers. Now, Constable, I don't want anybody... Mr. Rivers, you've got somebody. And where would you like me to sit, Mr. Rivers? In front with you or in the back? Well, in the back, Mr. McDuff. Canada's a lot different than I expected. The scenery? No, and all the people. You haven't had much time to tell about us yet, have you? Well, sometimes it hits you at first sight. I mean, you can usually figure a man by... Well, by the way, he handles his horse. But with women, it's different. What did you expect the Canadian women to be like? We're like the scenery. Good to look at, but sort of frostbitten. But with you or not? I've always understood that Americans were too busy building and selling and shooting each other to waste time saying nice things to women. Or we get around to it now and then. Oh, that must be appreciated by your wife. Whose wife? Aren't you married? No, not me. There's a man who never makes the same mistake once. Do you believe that? Well, I did. To a mighty reason. Tell me, do you think a plant or something from up here would do all right in Texas? I don't know Texas. Well, even the moonlight warm and soft along the Fakish River. Oh, Mr. Rivers, have they got fast horses in Texas? Fastest in the world. I bet they cannot keep up with the men. Look, look down the road. All those people. Well, they must be coming from Batash. Refugees, I'd say. Refugees? Well, there's Mrs. Burns. Mrs. Burns? Stop the wagon. Mrs. Burns, what happened? Mrs. Logan, don't go into Batash. They've looted the trading post and driven us out of our homes. Is the rock allowing them to loo? No, it's not the rock. It's your carbo. Carbo? Oh, excuse me, lady.  I've just heard Carbo promise this council the backing of Big Bear. And when Big Bear leaves the Cree on the wall path, the black feet will rise with him. Morning, folks. Who is that man? What does he want here? All right, I hear, mister. Don't mind me. You rock. Who is he? He's of himself from the United States. One talk with Canada government, so I bring him here. I'm a deputy United States Marshal, Mr. Riel. You know my name. Why, sure. Who don't? Here's my credentials. Now, we'd be happy to know if your new government means to work hand in hand with the United States. The leaders of Canada will be most happy to cooperate with the United States. Well, suppose somebody commits a crime in our country and trails up here to hide out. We'll help you in every way. Just as we hope your people will help us. Well, that's fine, because I've got a warrant here for the arrest of a man named Jack Corbeau on the charge of murder. Way in! Way in! The council is recessed. Wait here. I'll be back in just a moment. Thanks, Mr. Riel. What's the matter, Louis? Why aren't you at the council meeting? Corbeau has an American officer here. For what? For you. Well, he wants a warrant for your arrest. What shall I do? In two hours, give him your warrant. Then tell him I'm in Big Bear's camp. He can arrest me there. He wouldn't leave that camp alive.  In an Indian camp? Who cares?  It's going to be ours again. Take the warpath with it. Fight like you used to fight. Not one of those prairie cops will be buried with his scouts. Big Bear, let me speak. Don't listen to the Red Court. Now, I warn you, the Red Court has come from far. Let him speak. Big Bear, that medal you wear around your neck is for the Queens. The great white mother who sent food in you when your people starved.  Men said, Queen, no longer rule us. Red Colts, no more our friend. No. This is our friend. This catching gun. It takes with a thousand teeth of fire. Make gun talk. If I do it, we'll tear your lodge to rags. Squads make no law. Make gun talk. Big Bear, a hold and a tent won't change the British Empire. This man Corbeau is going to hang. I don't want to see you hang beside him. Hold up. Hang it easy, friend. I'm coming, people. Wait. Who is this man? This old Omanoteva. Hello, Chief. Sorry I can't shake hands. Your braves got me tied up a little. Rivers. Hello, Constable. Who is he? Spy. Out your men and horses. This man is no spy. He comes to take Corbeau who killed in his nation. Like Mounted Police want Corbeau because he has killed here. Are you afraid of Red Colts, Big Bear? Their blood will spill like other men's. Listen, Chief. Before you turn this Corbeau fellow over to the Mounted Police, I got an order for his arrest not only from the United States, but from his own Chief, Louis Riel. Very brave of you, Corbeau. Hitting a man with his hands tied. Get out of my way. Don't reach for your gun, Corbeau. I'm paid 72 cents a day to protect dummies like him. Thank you. We have peace. We got no war with Red Colts yet. But there is no take-away, my friend. The Red Colts will go and this man with him. Yoko Ka. Big Bear was a great Chief. His friends will be sorry for him. I'll take that medal, Big Bear. No. No? Then I'll rip it off the chain. Sorry. Keep talking to him.  But its medicine will be gone.  You are not Chief. Unless she gives it back to you again. Of what? Here. You want to join his blood brothers and fight for real? Big Bear is a great Chief. Why, I've heard of him way down in Texas. But how do you know he's the best fighter, Big Bear? The modern police are Corbeau. A great killer of men. This fella's killed in Texas and he's killed in Canada. He must be a great gunfighter. But has Big Bear ever seen him killed? How do you know these items are true? I'll show you if they're true. Well, he offers to show you. All right? You can kill now. Who? Me. Slide my guns back into their hostages. Untie my hands. Then tell this tough killer to go to it. Money's worth. Pizza gonna be. Thanks, Chief. Now tell Corbeau to kill. Never. Go ahead, Corbeau. Go on. Tell him to shoot, Big Bear.  Tell him to start on me. Okay. You're a hand of shooting men in the back. What's wrong now? I won't howl at great chief's lodge with the blood of a dog. Big Bear. This man runs yapping across the trail to save his red-coat friends. But the medicine gun will roll them all in the dirt. I have not seen this yet. Then look at it. Watch. That gun's strong-witted. Shooting holes and tents is not killing men. Big Bear, hear me.  Bring me in your hands, their empty coats, before three suns have set and my graves will take the warp. Before three suns have set, Corbeau will wear iron on his hands. Corbeau will wear iron on his hands. Who is it? Who's there? Heekor. Heekor. What do you want at the hospital at this time of night? It's late. Now I-I help you. Help me? How? Red coats march for dark lake. Half three, they wait and move. They make trap for red coats with no click. Remember. With medicine gun? Yes. No, brother. Keep watch in old heaven. See this person.  So he don't see them. Make what you call. Umbus. Ronnie. But it's Ronnie who is in danger, Louvette. That's why I came to you. You think I like your brother too much, eh? Oh, Louvette, if you like Ronnie at all, this is your chance to help him. What do you mean, like Ronnie? I love. That's why you don't like me. You think I'm no good for him. Ronnie's on outpost at the old cabin near Beatty Basin. If you love him, Louvette, go to him. Tell him to warn the police not to march on dark lake. Where don't you go? I'm being watched. I could never get out of the posh, but you can get through. If you don't warn the police, Ronnie will hate you as long as you live.  Mm-hmm. That's too bad.  Well, there isn't anything in you but hate.   Blood and revenge is all that you know. Not love. All right. I assure you love. What do you want, I tell Ronnie? You go? Sure I go. Tell Ronnie the half-breeds are going to ambush the column at Beatty Basin. Sure. I'd tell him. He won't forget what you're doing for. No. You won't forget. How'd you know where I was? Oh, I just hear some clothes. You know my father, he's put him out on me. He found out I got big love for you. Why didn't you tell me Jacobo was your father? Who do you care who is my father?   But you don't never marry with me now, eh? Once this blow-off is over, I'll marry you quickly. No. That's too late. Why? My father going to kill me, because I love Redcote. We'll be married tonight, or we don't marry never. Tonight? He married a friend. I can't leave here. Why? Nothing happened here. Who are you here anyway? Orders. But you'll be back before somebody knows. Oh, Ronnie. Oh, love is so terrible, won't you? My mom's duty, they hang me, shoot me. Try to understand, won't you? I've got to stay here and watch the riverbed. Father fell a good watcher. He's there now. Maybe you better get in the riverbed with him, eh? If your neck wasn't so lovely, I'd break it. You break my heart. I don't live without you, Ronnie. Goodbye.  What do you think you're going to do? I go. No, you won't. It's the sweetest poison that ever got into a man's blood. I love you. I want you. We'd be in that ocean one hour, maybe less. Ten minutes with the priest. You know where to get back here. Ronnie. I'll leave a note for Jerry. Tell him where I've gone. Oh, Ronnie, hurry. There's the cabin. That's funny. I don't see anyone around. Well, I'd say your eyesight was pretty bad, constable. If I'm not mistaken, that's a red coat over there. Where? Linging on the ground, under those trees. Edwards, give me a hand here. Yes, sir. Turn him over. Quiet. It's Jerry Cameron, sir. Jerry. With two Criaros in him. Get back to the column. Born Inspector Cabot, outpost kill. Danger of enemy on all sides. Gallop. We'd better get back with him, Mr. Rivers. Thanks, but if it's all the same to you, I'd like to stick around while you inspect that cabin. Come along, then. Was there anyone else on guard here? Ronnie Logan. Ronnie. Ronnie! Where are you? Doesn't seem to be around. Must be down in the riverbed. You'd better wait here. If I don't come back in five minutes. You won't find him there. What? Hey, he left a note for somebody. See? Jerry, I have gone to a wedding.  That can mean only one thing. That blue-eyed cluch has got him. Yeah, what are you doing? I'm burning this note, do you mind? That's destroying evidence. Evidence that might put a boy in front of a firing squad. If our whole column rides into a death trap, he'll deserve whatever he gets. I was thinking of his sister. You think it'll help any to make her pay for his mistakes? Somebody's paying for it right now. That's her bear. Spring the wounded in here. Dusty, Dusty, I want to see you in there. Hello. Hey, you always turn up where you can help, don't you? Help me. Let me leave Batash this morning. Dusty, what are they all saying about Ronnie? Well, they're all out of their heads. Do any of you know where Ronnie is? Please tell me. They don't know, April. Sergeant Brett, Inspector Cabot wants to see you, sir. What for trouble? Wounded? Yes, sir. He's in bad shape, sir. Very bad. I'll see him at once. Jim. I'm glad you're safe, April. But I wish you were somewhere else. Jim, I tried to get here in time. I could have saved all this if only... Jim, where's Ronnie? I don't know. Jim, is he... I'm sorry, April. I have to go to the Inspector now. Inspector Cabot, you wanted to see me, sir? Yes. Is there an assault? Not yet, sir. Got us in the Bastion. I have, sir. Have you been deserted? Yes, sir. This is over. Make it your job to get him. Bring him to the Justice. Justice? I'm about to get mine. Take him out from Brett. Yes, sir. Some fool at headquarters wants to change the uniform to green. Stand up to the Red Coast. It's a good rover. Inspector. I'm sorry, sir. You bring it to the men. Soup? If I pour any more soup down them, they'll burst. Sit down a minute, April. I can't. There's too much to do. Sit down before you fall down. Special order says you've got to stop now and have a cup of coffee. I'll get it for you. Dusty, what's happened? Tell me. Did something happen? I didn't notice anything happening. Over and over, I hear them say his name but they won't tell me anything. Not even Jim wants to talk to me. Jim's a great soldier. Where's Rami? What happened to him? What has he done? Listen, I don't know Ronnie so very well, but I know you.  And I know it couldn't be in your brother's blood to do anything that was cowardly. But love does funny things to people, lady. Oh, even comfortable. Is she all right? Yes, of course I'm all right. Rivers, I've got a tough job for you if you'll take it. I'm going to try to get the wounded to the river under cover of the smoke. What smoke? We'll burn the fort. It's your job to get April and the wounded down the river to meet Colonel Irvine's reinforcements. I suppose you're going to stay here and roast chestnuts? I have six other men sound enough to ride.  Well, it sounds to me like you're just delivering seven more red coats. Maybe. I'll speak to you outside. Jim, Jim, wait. You've got to tell me. You've got to. I know it. But let me tell you first that I love you. Always, I guess. But nothing under heaven can ever bring us together after... What do you mean? Ronnie isn't dead. He deserted. He could have warned a column and all those men lying dead out there might be living now if Ronnie had... But he ran away. He ran away with that half-breed girl, Cor Bol's daughter. I don't believe that Ronnie deserted.  I'd expect you to feel like that, April. But when this is over, and I will get him if I have to follow him over the ice cap, you'll know what that means for him. I know your duty comes before everything else, and I know it means more to you than mercy or love.   And don't tell me how sorry you are. Just do your duty and kill. Kill! The battle is over, and Cor Bol has made good for Big Bear. At a council of war in the Indian camp, Cor Bol throws at Big Bear's feet the bullet-riddled coats of the mounted police.  This one belonged to that sergeant who said he'd put me in iron. Medic and guns speak like thunder. Yes. Now keep your promise, Big Bear. War! War! There's the cry of war that echoes through the camp. Seven riders appear silhouetted against the sky. Six red coats riding abreast on the sergeant in the lead. A thin red line of courage advancing to almost certain death. Little man, Lowy, if that sergeant wasn't killed, he will be now put down your gun.  Big Bear, tell your wards chief to spread his blanket on the ground. Tell your men to fire. Kill the dogs. You tell me they are dead. Let us know what dead men want. A cupota. The spirits of the brave are still inside their red coats. They are sacred. Tell your men to lay them on the blanket. No. No. These enemies are in your hands. Pull them down. Kill them now. If one shot is fired, the soldiers of the queen will come like the sands of the Great Salt Sea. She lies. There are only seven.  There's no magic in their red coats. These handcuffs are for you, Corbow. Shoot them, you fools. Shoot them. I said you were iron on your hands, Corbow. Shoot them. Kill them. Too many left. Just mount. No. No. Now take the prisoner. Come on, Corbow. Big Bear. I have brought back the metal from the great white mother. Will you wear it again around your neck? Will Big Bear kneel to the queen and be the chief of his people again? Decree our brothers to the brave. Big Bear will kneel. That will be Colonel Irving's camp just around the bend of the river. I'm afraid we'd never see that camp. You're going to see it and get some rest. I'm leaving. Corbow, go on hunting. Jim has Corbow by now or Corbow has Jim. I wasn't thinking of Corbow. You mean Ronnie? Well, I'd like to have a pow-wow with that little wolf girl he's been running around with. Ronnie's dead. I know that. He wouldn't run away. If he were to blame, he'd come back and face it. Sure he would. But maybe he can't come back. Maybe he sort of lost his head and, well, I believe I could be here for you if you'd let me.  Sure there is. Come back to Texas with me. And if Ronnie's alive, why, you'll find us there. If you want what's left of me after all this, I'll go with you. I'm a coward, Dusty. I won't run away. You're the loveliest and the gentlest lady I've ever known.  I don't want you to say yes I don't want to think. You won't have to, in Texas. Put in the shower, Todd. You take April on down to Irvine's camp and then make me back here. Would you mind telling me what we're up to, Mr. Riddick? No, I wouldn't, if I knew myself. Where are we heading? Well, there's a little creed village up here somewhere. And I got an idea. Whoa. Hey, Todd, look over there. That cliff on the other side of the river. I see the cliff. What of it? You don't happen to see a Gatlin gun, do you? I do know. Right on the edge, too. You know, Todd, if a man could right up there and slip a rope over that gun, he could jerk it clean into the river. He could, if they didn't see him first. Well, I'll take my chances on that.  But how will I know if you run the gun off the cliff? When that gun gets a rolling down hill, everybody for ten miles is gonna know it. Open up, Louvette. I know you're in there. Open up. What do you want? Oh, just passing by. I heard you were here and thought I had dropped in. You look for Ronnie, huh? Well, you don't find him here. He never come here. Maybe. You come to hurt him? Nothing I can say can hurt him any. It's been said already. Don't listen to him, Ronnie. Go back. Hello, Ronnie. What's been said? I want to know.  Does April say that? No. She figures you're dead. She knows if you could even crawl you'd get back there and face him. Don't listen.  Well, she may be right, Ronnie. Listen. I tell the truth. I tell him we'll be married. I lie. And then I have half the time. Look, you see his hands. We try for days to get back. And then he know it is too late. You must believe it. I believe you, Louvette. But he's still guilty. Well, Ronnie, I got to be heading back to Bettorch. You coming? What for? I'm already buried in every grave of the men killed at Duck Lake. Well, they're bound to find you with your help. Don't tell nothing. He never tells Ronnie. Never. I'm not going to turn you in, Ronnie. But April has the right to know what she did is up to her.  Because you're not a coward. Ready, Ronnie? Yes. I'm ready. He never tells. Well, back so soon. What was it you fixed so well? Where is Ronnie? Well, he's going to Bettorch. No. What's the matter with you? Let me go. You can't catch him now? Let go. You're too late, Louvette. Listen. I tell Cree, I tell him to shoot men unhorsed with white faith. What? I want to stop you. Now let's shoot Ronnie. Ronnie. Captain Gower is here to help us get what facts we can, regarding the ambush at Duck Lake. Sergeant Brett, Constable's Cameron and Logan is that correct? Yes, sir. Warning should have come from those centuries. Isn't that so? We found Constable Cameron dead, sir. Two Cree arrows in his body. And Logan had left his post? Well, Sergeant. Well, you see, sir. My brother had left his post with a half-breed girl, Louvette Corbeau. How do you know, Miss Logan? She thought I sent her myself. I sent her to warn my brother of the ambush. She'd never have known where he was if I hadn't told her. None of us are on trial, Miss Logan. We're only trying to find out what happened. I would have known that she'd lie to him. That she was in love with him and would persuade him to run away with her and save his own skin. I don't believe that. And nothing can make me believe it. Sergeant, will you post orders to all detachment to arrest Constable Logan on site for desertion in action? You can't arrest Constable Logan, sir. What's that? You can't arrest him, sir. I brought him back with me. He's dead. Ah, I'm sorry, Miss April. Put him down over here, men. Sergeant Brett, is this Constable Logan's body? Yes, sir. I don't understand. Well, this boy was captured. Take a look at his wrists where the raw hide tongs cut in. The man has been down, certainly. I don't know how he got loose, but when I saw him, he was free. Maybe he could have got back to report, but he didn't. Because there was something that he wanted more than he wanted his own life. What do you mean? Well, I mean the Gatlin gun. Now, the half-bridge had it on a bluff, but it wasn't his approach.  Now, the half-bridge spotted him. Bullets were as thick as July flies, but they didn't stop him. No, sir, over he goes. Horse, gun, and man are rolling like a Texas twister over the cliff. Well, that's where I found him, and that's where you'll find the gun under the river. He never quit. He never ran away from anything because it wasn't in him to quit. Well, I was squeezed under a log so tight I was breathing on the installment plan. Shall I include that in my letter to your Texas commandant? Well, I hope you won't, because I'll be in trouble enough when I get back without Corbeau. Dusty, Dusty, you're an angel in leather. Well, I'd look funny with leather wings. I'm sorry I misjudged a brave man. We all are. Sergeant, she will deliver as soon as she's gone. Like I said, Dusty's gone. He took Corbeau with him.  Corbeau's our prisoner. They must have hit the trail early this morning. Last night, Dusty said, if he should kind of disappear, I should tell Miss April he'd be back for us. Thanks, Mac. Well, I suppose I'll have to hit the trail myself. Come on, Corbeau. We've got a long way to go yet. Where are you taking me? It's a wonderful place. You like it. Come on. I'll tie my hands and I'll ride faster. Yeah, maybe too fast.  Just a minute, cowboy. Going someplace, cowboy? Well, I don't look like I was going far. Come on, April. It's nobody but Dusty taking a little ride. Hello, Dusty. May we ride with you a little way? Well, yeah, yes. Fine. Well, Dusty, I didn't expect to find you alone. Alone? I'm not... I was afraid Corbeau might be with you. Oh, well... Say, Miss April, do Canadians go snow-blind in the summertime? I'm afraid we're all blind sometimes. Well, I'm not. It's talk that some fellow might try to get Corbeau away from us tonight. How's that again? We're only a hundred miles from the Montana border, you know, and no telegraph. Is that so? Dusty seems so restless. Maybe he's thinking of a man who moved in on a friend that trusted him. Sneaks in behind his back like a weasel in a handcoupe, and stole the one thing his friend valued most. No. No, no, I wasn't thinking about that. Wouldn't you say he'd owe his friend a lot for doing such a low, contemptible thing to him? Well, he'd certainly owe him an apology. Well... I apologize. You apologize? Jim's trying to tell you that he's taken something that he thought was yours. You mean... You? Yes, Dusty. If I'm free. She told me I'd be all right if something beat me to my knees but I've been brought down hard. Oh, no, I'm the one who surrendered. I wouldn't change him if I could. Well, I guess I've just had a beautiful pipe dream in the middle of a nightmare. Sure. Sure, you're free. You got what we both want, Jim, and I got what I came for, and I'll say the mounted police are the greatest outfit I ever tangled with. And the luckiest. The Texas Rangers are two-fisted fighters, and one of them is the most magnificent liar in the world. Well, we'll say goodbye. Good luck, Dusty. Thanks. Goodbye, Dusty. There'll always be a part of Texas in our hearts. Come on, sweetheart. Well?  No. No? Come on, sweetheart.


When I submit this talk to RailsConf, it is in the track of we are living in the distributed world. But I'm surprised to find that I'm the only talk in that track. Seems like there are no other tracks that talks about scaling of Rails applications and distributed systems. So I think the reason might be that as Rails developers we are following some best practices so that making our app distributed or scale our apps doesn't seem that hard, doesn't seem that problematic. But this one, this GitLab thing is a bad boy, I would say. It really got some problems and I mainly got to talk about how we fix those problems. So thank you very much for coming to my talk. My name is Minky Pan. I came from China. I work for Alibaba Group and that is my GitHub account and my Twitter handle. Welcome to follow me. So what is GitLab? Well, GitLab is, well, let me say it secretly, it is just a GitHub clone, open source clone of GitHub but nobody likes to say that. So a better way to think it is a Git box that you could deploy on your machine. It is installed on premises. So just a quick survey, how many of you use GitLab in your organization? Two of you, thanks. So GitLab is, if you see it as a black box, it actually exposes two ports. One is HTTP, the other is SSH and HTTP are used on two purposes. You can clone a repository via HTTP and you can push content to a repository via HTTP. And also, more importantly, as a Rails application, it provides rich user interactions with the web page. And on the other hand, the SSH only allows you to clone to Git operations. And in the back end, from a very simplistic point of view, stores this content on Git and that is what makes this thing a monster to scale, very problematic on that part. So if you look closer, it also uses some other stores on the back end. One is Massacre. Actually they also support PostgreSQL because they use ActiveRecord, which abstracts the actual implementation of the DB so it's changeable. And the other is Redis, use it as a queue for delayed tasks and also as some cache. And the other is FileSystem, they use FileSystem to store the Git repositories. So that's the black box. If we open it up to see what's inside, then you could see it's basically structured like this. It's all open source, so you could also download the source code and see it. When you deploy it on the front, and there are two parts, Engines and OpenSSH server, well the reason why those components are inside GitLab is because GitLab has a omnibus package that you can install and they actually depends on those two other packages. Engines is for HTTP and OpenSSH server, as we mentioned, is for the SSH port that opens. When some requests came, for HTTP requests they came to the second layer, Unicorn, is for the ordinary Rails requests, but for the request for Git, like clone and push, it goes to GitLab Workhorse. It's another service written in Go to make it fast. And if it came as a SSH request, it goes to the third part of the second level, namely, GitLab Shell. And on the third level, the third level is called by the second level components. Mainly Rails was responsible for operations on page. And GitLab Git is a wrapper around Rocket, and Rocket is a wrapper around Libgit 2 on the fourth floor. And SideKick, yeah, that was for some task handling. And on the lowest level it is Git and Libgit 2. They utilize both implementations of Git. Libgit 2, if you don't know about it, is actually a rewrite of Git in a way that is portable, embeddable, works as a library. And Go to name Git 2, they see it as the second generation of Git, but with Lib as a prefix because it's a library. So this structure works really great for small teams, but the company that I work for has 30,000 employees. This is from the physical year report of last year. They just published a new one this year, days ago, the day before yesterday. And the stock price went up, it looks good, it's a public company. So let's scale it. So how we do this, well, we first consider about the problem on the front end. When the request came, it's either HTTP or SSH as Rails developers. We are most familiar with HTTP. And on the server, it's actually run as unicorn instances. And that's something we are very familiar with as well. We just put the engines in front of them, set upstream in the configuration, let them point to the unicorn servers in the back, and we are done. But for SSH, how to deal with this is a problem. So I started a project called SSH to HTTP, it's open source on my GitHub account. It basically eliminates all those SSH requests because the way Git interacts with the server between HTTP and SSH is very similar and the request to SSH could be easily delegated to a Git request on HTTP.  There are more complications to this. So I guess that is the reason why GitHub nowadays has set HTTP as a default. When you go to a public repo on GitHub and the Cologne URL, as far as I remember, is defaultly as HTTP URL instead of SSH one, there are actually complications to the architecture that makes the SSH access a little bit slower than HTTP one. But actually in Alibaba we did not use my approach, my approach was this slide, but actually we use this slide. What we did was we are not using engines as the front end. We use something called LVS and it is a feature from the Linux kernel and the specific part of it that we're using is called IPVS which expands to IP virtual server and LVS stands for Linux virtual server. It is actually a layer for switching service, unlike engines which operates on layer seven of the TCP IP stack. It does load balancing on the transport layer. So it supports all communications as long as they are TCP IP. So the difference between HTTP and SSH are illuminated. But there comes as a cost as well because when you go down to the fourth layer, you lose the ability to do health checking with the status code returned by the request because on the seventh layer you could see actually what's the status code of your HTTP requests are and mark some server as health or not healthy. But on the fourth layer you cannot see those. You can only see packets, you can only see the date. And you are rewriting, you lose that ability as well because there's a layer seven as well. And like I said, that comes with complications because SSH protocol involves some security mechanisms that checks with your keys. And if you have more than one machines in the back end, their keys are not the same by default. So when you deploy the application, you first have to copy the host keys across the whole cluster to make the host key the same. Otherwise, when you connect to more than one servers, the client will complain saying that, oh, the SSH key was different, this is a security vulnerability, you got to check it out, and it will not connect. And secondly, if you remember, you could add SSH keys from the client via the web pages when you clone a repository like on GitHub, and same thing happens in GitLab. So when you add your SSH key to the server, it has to dispatch or copy all of that keys across the entire cluster to make every machine accept your key. Specifically, they add a line in the .ssh directory slash authorized keys. And they have to do it on every machine. And we did that via, well, you cannot do that via SideKick because SideKick only, you know, only one machine in the cluster fetches that job and the other will ignore the job. So you have to do it in a way that broadcasts all the keys across the whole cluster. And we did that via Redis PubSub data structures. And that goes the back end, well, the real trouble begins at the part that GitLab stores its repository on the FS. And I want to pause the moment to remind you of the 12-factor app. So the reason why GitLab is such a bad boy, unlike other Rails applications, is because it wireless the fourth rule of the 12-factor app. That is some principle advocated by Heroku, where the fourth rule says back end services should be treated as attached resources, like Twitter service, Amazon service, MySQL service, they should all be configured as a UIL that could be easily attached and detached. But GitLab stores some content on file system. That is the source of all evils. The content they store are firstly, Git repository, and secondly, user generated attachments and avatars. Now we are going to move them to the cloud to make a scale. Well actually, standing at this point, you have a lot of choices. The choice that I am going to elaborate might not be the best. I want to, you know, analyze the options that we have. So if you, when they run into a Rails application that has a similar problem, you could evaluate those options as well. So the first option is some feature provided by GitLab Enterprise Edition is called GitLab Geo. And that doesn't really solve the problem. You see how GitLab Geo does things is they make full replications of your GitLab instance across servers. It assumes that each machine of your cluster has enough file system storage to hold all the content of your Git repositories and they make 100% copies across them. It's officially supported, but it really didn't solve our problem at Alibaba because the overall size of all repositories are big. We don't want to store them on one single machine. They are not enough disk space to hold them. So from a distributed system point of view, GitLab Geo is a one master enslave full replication design and in CEP theorem which says consistency, availability and partition tolerant cannot be achieved at the same time. You can only achieve two of them. So GitLab Geo achieves a NP of those three parts and there are no disaster recovery supported and absolutely no sharding because it's full replicated. And the other option that we could use, it's seemingly a very perfect way to solve the problem. Well, first of all, we eliminated SSH by that gem written by me called SSH to HTTP so that we could forget about the problem SSH and focus solely on HTTP. Seemingly, there is something we could take use of. It is the, you know, every repository stored on GitLab could be routed using namespace slash repo name and that part appears in almost every UI or of every request. Like when you see the repository commit history on page, the road format contains that part and when you clone it, when you push it, they all contain that part. So why not use that part as a routing key and make some routing logic into Nginx to make a sharded GitLab. And by doing that, every request after coming to Nginx will be sharded. For example, if we are going to have a cluster of size three, we could invent some hash algorithm that distributes, that hash the namespace slash repo name into the cluster, into any one of those three machines.  Actually, one problem is, SideKick does not have sharding. Actually it does, but you have to dig into it and see how you could do that. You know, each shard of those three GitLab shards could spawn some SideKick tasks which needs to be consumed by corresponding SideKick shards as well. So when you start the SideKick shards, you have to start it with special queue names as well. There's one complication and there are others. Changes have to be made on application level as well, because it's not every page on GitLab falls into a single shard. Like in the admin page, you could see a list of all the repos with their sizes. Well, if that request falls down into only one single shard, you will not get that information, because some repos reside in some other shards. So major changes will be introduced to the application level as well. And also you need super user authentication, because the SSH requests are not designed to access all repos. Their user authentication layers in front of them is also another application layer, logic layer change that you have to be introduced. This is actually not ideal. Every way of solving this comes with a cost. So let's then think about how to deal with the file system storage. Well, we got a lot of options. Well, first, we could make it a 12-factor app by making the file system attachable. There are some vendors who provide such solutions, like hardware network attachable storage. They usually call it NAS, and there are software NAS as well, like Google has GFS. And also we could use remote procedure calls to only make shards on the FS level instead of on the application level of the entire GitLab. And also we might consider killing it. We could maybe use Amazon S3 to replace DFS as the backend for Git stores. Well, we evaluated all those options. It turned out that NAS is not for Alibaba. Hard NAS, well, Alibaba do not buy those things because it has no IOU policies. And soft NAS, Alibaba does not have that yet. Like Google have GFS, but Alibaba does not have AFS. But I have to remind you that those two options might be good options for your organization. If you want to scale GitLab, you know, they are really good means to solve the problem because it introduces very little change to your application level because all the change are confined on the lower service that got attached to GitLab. But I did not try them, and they surely come with a cost as well because software NAS tends to be very complicated. As far as I know, there are some good solutions called CFFS, which just came to stable about a month ago or days ago. And if something happens on that layer, you need to have some very talented operation or dev-op engineers to solve those problems. And also by attaching a NAS, soft NAS, you will also lose performance because each IOU to the FS is now networked, and they are added latency to each network IOU. And you are replacing the thing on a very lower level, so the added cost will be much. So those two operations, if you have a chance, could dig into it. And RPC? Well, that is a good solution. I looked up how GitHub solved their problem. It seems like they are doing RPCs. They are dispatching access to some RPC calls into Git shards instead of Git lab shards. It's a shard on a different level. They does that. It surely looks like a good solution. And what we did at Alibaba is use the fourth option. We kill the FS and use the cloud. What clouds we use? Well, it's called Alibaba OSS. Well, it's something not that well-known, but you could thought of it as the same thing as Amazon S3. It's object storage in the cloud. And how we did that? Well, the rest of this talk will become a little bit technical. It turned out that GitLab has three ways to access Git repositories, namely Libg2, Git, and Git. Grit is a very old gem. It's written in Ruby. Well, we found that it could be eliminated making the whole problem easier because it's only used in the Viki part of GitLab. And it's used in a gem called Golem. And Golem was designed to have Git access part plugable. So we unplug Grit and we plug Rocket, which uses Libg2. So that makes this only Git and Libg2. And we compare those two projects, Git and Libg2. Well, Git was pretty old. It's probably written by, started by Linus Torwas. And it did not consider the problem of backend to plug and unplug. So it's backend is hard to replace. All of the code are written to access content from the file system. But Libg2 is very modern. I don't know how their creators think about the problem, but they designed the backend as replaceable. You could write your own backends. So the basic idea is we write our own backends. We write the backends that actually stores the content on the cloud storage. And also, well, the grid has been eliminated. Also, we have to implement Git on top of Libg2 because Git cannot easily replace its backend storage, but Libg2 could. So cloud-based backend, what's that backend looks like? Well, that emotes some details about Git. Git has two parts to store its content. One is called ODB, and the other is called RefDB. ODB is for the chunks of data that you put inside the repositories. And RefDB is the branches, tags that you put in the repositories. And for the ODB, there are also two parts, two kinds of ODBs. The first is loose ODBs.  The content-addressable being the SHA1 value of the object that they are trying to fetch. So loose storage actually stores each SHA1 values. I will open up an example. Maybe I will .js. That's a Git repository. And if you go into the .git directory and you see tree, you could see there are some, like those files, those are called loose stored files. And there are also packed stored files. Those are the packed stored files. That's what I mean. So we wrote a cloud-based backend to store both types of those files. The basic idea is for the loose files, it's pretty straightforward. When you read, you make an HTTP request to read it from the cloud. Oh, I forgot to explain the refDB. It's very similar to loose files, where you can see it's under the refs directory. All of your branches are inside the refs has master. And master will tell you a SHA1 value. So it's basically K-value store. And that translates to HTTP requests pretty straightforwardly. You see, each refDB read, we made it an HTTP read. And each refDB write, we made it an HTTP write. Each loose ODB store, we make it an HTTP port. And each loose OSS store, we made it a HTTP read. So that's the simple part. The complicated part is the packed content of it. Because if you only store those loose content, it will be as slow as SVN. The very reason why I get it so fast is because it has a very good design of packs. Pack files are used both as a way to transfer content from between server and the client and as a way to store the content you are repository on disk. It's both a transfer file format and a storage file format. The way we write those packs are easy. We just translate them to port request to HTTP. But the way we read it is complicated. You see, every pack came with the index file. And that index file tells you about if you are looking for some object in the pack, where to start. So each request will be translated into a lot of ranged HTTP requests. First, it will read the IDX file to find the next range to read in the pack. And then it read only that small portion of that file using the range header from the object store. So as an example, if Git need to read this content, then first byte will be blah, blah, blah. And it will binary search in the index file. And it will get an offset to begin in the pack file. And in the pack file, it will see if this content stored is a delta or not. If it were a delta, then it has to continue looking for the base of that delta. And the whole chain continues, continues, until you find the root. And by combining all the deltas with the base, you get the object that you're reading. And here's an example. It's a reward example. The chain is as long as file. You have to jump inside the pack file to actually got the thing that you want to read. Because each time you read it, it's actually only a delta. So that is a real problem to us. Because if the IO pattern inside that pack file is not good enough, then you will end up having a lot of range requests on the HTTP. That will make the thing awfully slow. But the good news is the inventors of Git, they made some very good heuristic algorithms to when the pack files are generated so that those IO patterns are not that bad. So when we make a range request, we could actually make the range bigger than we needed. Therefore, we could fetch bigger content from each range request. And that content would be sufficient to fetch all the way to the root of that object. And by this good characteristics, we reduce many HTTP requests to make this whole solution not that slow. That's one part of it. And the other part of it, as I said, you have to make Git talk to libGit2 because Git does not have the backhand replaceable. It turned out that this is pretty easy. Actually, the inventors of Git, they're pretty smart folks. They write Git in a very unique way. All of the commands, they call each other. Like in Git fetch and Git clone, on the server side, they first got called was Git uploadPack. And Git uploadPack will then call another command called GitpackObjects. And for the commands that deal with the transmission protocols, we will not touch it. And it's complicated. And we do not touch it. We only touch the thing that does IO from the disk. So we only need to replace the GitPackObjects. And in the GitPort scenario, we only need to replace or re-implement Git unpack objects. And implemented on top of libGit2 is very easy. It's no big task. And also, there are other scenarios where there are two scenarios when doing GitPort. There are small data got unpacked right away and got written to the loose storage. And their big data didn't got unpacked because unpack consumes time. And they directly create the index for it and write those pack. So in this case, we need to re-implement Git index pack, which is pretty easy task turned out to be. All right. So after all of those changes, let's see how the performance looks like. It definitely going to be slow because you're still changing a fast file system IO to some slow HTTP IO. So let's see how it looks like. Well, the text fixture we use is a repository called GitLab CE. It has more than 200,000 objects. And when packed, it weighs more than 100 megabytes. And GitPort, well, about same performance because file system, we write it directly to FS. But on cloud, we write them directly to HTTP. And there are not too many new operations created. It's just only added a small amount of time to each of those two operations. And GitPort's delta also, like I said, there are two scenarios. When you push large content, it only stores the pack. So this is a large content scenario. And if you only push a little content, it got unpacked and stored loosely. And this is the delta case, also not too much time added. And GitClone, well, it is actually 100% slower because when you do clone, the range operations happen. And that's what makes it slow. And also GitFetch, it got way more slower because this is a delta fetch. This usually happens when you do GitPool, when your coworkers updated the wrapper. And it also has to go through the whole process of the range operations that I mentioned. So it's really slower. But the good news is it's not that slow. The user has to wait longer, but it's not as something that they cannot wait. And also on the page, it got way slower. All of the Rails operations were affected because we are operating on a deeper level. And the Rails we'll call rugged, rugged called libgit2. Libgit2 is slow, so Rails is slow. Like on this page, we're listing a file list. And the show actions now take five seconds to run. Well, let me see. All of those benchmarks are all without cache. So the real world scenario will be better because we have cache. And like this, this is another Rails operation. And before the change is 50 milliseconds and after is about five seconds. So that's the reason why we have to add much cache to it. We added cache on multiple layers. Like those Rails layers, we added them. I'm not going to elaborate on all the cache that we add. But for some interesting aspect of this, this is something interesting. Well, libgit2 was designed in a way that it could have more than one ODB backend. And you could even set a priority to it. So we basically made a hamburger structure of that backend. We added two new backends to it, which is the cache backend. You know, the servers that we deploy those things to still got a file system to use. And we use that as an on-disk cache. If we read some content once, we'll store it on the file system so that the next request hit it could just read the content from the file system instead of making remote HTTP calls. And the good news is the ODB of git never change. You can only put data into it, but you can never modify data. So we are free from the problem of cache expiring. And also, the refDB could cache that we already is. But that's way more complicated. That might not worth the effort. I might remove it in the future because you have to expire the cache. RefDBs got updated all the time. When you commit a new commit to master, say, the refs slash heads slash master got updated. And you have to expire the cache. So not going to go into details of when the cache got updated. And lastly, I want to say something about the future work. For right now, it seems like this idea works more or less acceptable. And if you guys love it, I will try to do a AWS S3 version of it because it's currently working on OSS, which is not so widely used. And there is some need for this. The reason why there may be some need for this is because GitLab cannot be deployed to Heroku at this moment. And if we could make this backend for AWS S3, then the users of GitLab could have a chance to deploy it to Heroku. And also, GitLab still has many direct calls to Git for the history page of the commit history page of a repository. It actually spawns another Git instance to fetch the result. So we could eliminate some direct calls to Git. And if we develop that backend for AWS S3, we could add settings for the user to choose which backend he wants to use. It could be either file system or AWS S3. That would be perfect. And Gollum, we could do some work to make them use rugged as the default. In Libgit True itself, we found it slower in many scenarios compared to Git. So we could improve its performance in the future. And I will be actively do those jobs on my GitHub account. So if you're interested, you could look into my account and see how it goes. Thank you very much.


For those of you who are watching this little segment of current events live stream that we are doing on June 20th 2020 on our beautiful garden patio, if you're watching this segment and I can guarantee you that this full live stream will not be loaded onto sensor tube then if you want to consume all the content that we are creating on this channel, the work that we are doing, you want to be watching our videos on Bichute, Rumble and Odyssey because that's where you get everything that we create. Okay so if you're watching this little segment which is about a minute or two minutes on sensor tube, the link to the full live stream which is a current events live stream talking about news politics, economics and everything will be in the description of this video including the link to the audio as a podcast on SoundCloud. Aside from that, those of you who've been watching these live streams, you know that's my little intro to sensor tube for this current events live stream. I'm going to go back to the current events and continue our discussion and remember gang, if you're on sensor tube, thank you for the support, thank you for being here, but the sensors will not allow us to upload this live stream to sensor tube. We have already been shadow banned and every time we try to share any type of discussion that goes against the technocrats in Silicon Valley and the capitalist power in Washington DC and the oligarchs in Wall Street and the propagandist in Hollywood and obviously corporate propagandists that share that spread lies from centralized power. We lose subscribers, we get demoted, we get warnings from sensor tube. So we're not even going to bother loading this stuff on sensor tube. The links to the full live stream will be in the description of this video. Snip.