Sorry. Hi, my name is Guida. I'm going to introduce Ganetti, which is an open source cluster visualization management tool We've wrote at Google for use in our corporate environment So first of all, I'm going to do an overview of how to manage a genetic cluster and what is it Then I'll go a bit into the details of how failover internal to Ganetti works Just to show it and then we'll talk what we're doing on Ganetti I'll have convinced you by the time to use Ganetti in your production network So I want you to know what we're working on and what the next features will be so What's gonna be basically you have Xen as a building block you Might want to use it rather than on a single box in a cluster possibly using failover But you don't have or you don't want to buy a Sun to store your instances on and things like that What we do is tie it with the RBD and build a purely open source based Cluster management technology without any need for special hardware without any need for raid on the machines you can just use your desktops and put them somewhere and start using them and We want support for different type of host systems because we have different customers that have different needs And we want to manage at the same time tens of nodes and hundreds of instances because We don't want to manage each node at one every after after the other, but we want to have a unique cluster system So what do we do basically we have this is a genetic cluster These are the physical nodes The DOM zero Xen terminology and then we have a virtual systems This is one virtual system. It has a virtual disk which is actually the RBD device between System one and system two so what you can do after this is you migrate the system between System one and system two right now This is a non-live migration Because we needed some features in the RBD 8 in order to do live migration now We have the RBD 8 in the latest version and we will implement live migration later But for now you have to switch off the system and restart it on the other system even if Xen Support live migration, but it does only if you have the sun and not with the RB this the RBD things without some changes So this is how to use genetic. It's very simple. It's very straightforward. First of all you In it the cluster and say this is my new genetic cluster then you can go ahead and add your nodes. So This was not zero. It's already added to the cluster when you add one two and three and then you have one cluster So you can for example install a system on all nodes or do things like that and This is the package that contains a genetic instance, which actually will be needed in order to Set up our cluster. So this is the cluster setup after we've set it up. We can just go ahead and Use it so we can create instances. This is an example command to create instances There are a lot more options around swap space disk space But for now, let's keep it simple You can create a normal instances with default options and then you can for example after a node crash You can fail over the instance To its secondary node. So it was a node to you fail it over to node one And then you can replace its disk and move them to node three. So the instance will be Rated again while before it was only on one node if the node had crashed Of course, it was if it was just a downtime when you can bring the node up again And you don't need to do that. It all depends how bad the node crash was So and then you can basically query your cluster status query, which instances you have What are they doing? How much memory? How much disk space? This is an example list when you have other fields You can ask for various information about instance configuration And this is the status of all the nodes in the cluster so you can decide where to allocate your instances and things like that now What can you do? This is the whole gannety like in a short version You have a gnt node command that lets you manage your nodes gnt instance. Let's you manage the instances the OS command basically lists and queries the operating systems you can install on the instances for example, you could install gannety on Debian, but have a red hot gen 2 and Debian images for your customers or whoever to use and You have global cluster commands and an instance import-export tool in order to backup an instance to Some other disk and then re-import it back or re-import it back with some changes So let's go a bit into the detail of an instance failover First, this is the normal cluster status as we've seen before Then what happens is that system one has failed So as you could imagine in Xen if the DOM zero fails the third system is down to and it's crashed But we have luckily a copy of the data So if system one's motherboard has failed for example, and we don't have a system one anymore or all its disk Have crashed because you didn't have a raid on system one. It doesn't matter We can just run the failover and then the system will be rebooted on system two which Will have a copy of the data as of a crash So it will basically be as bad as a power unplug but not as bad as having lost of your all your data Of course, this supports only a one node Crash out of the whole cluster or more if you're lucky But we can't guarantee more because if you have instances like node one system one and system two crash Then you've lost actually the virtual system one and you can't do anything about it So then what we do is replace discs So this of course is the wrong arrow somehow the real copies from system two to system three because we have no system one But theoretically what you have is you move your secondary discs from system one from system two And then you have your virtual system redundant again and ready for another failure So after you've done this and you've reduced the basically your cluster size You can support another one node failure as long as you have enough RAM and enough resources to host all your instances So this is the basic genetic what it does right now We have some optional features which are considered advanced for various reasons for example the replic the rbd replication Normally runs on the same network as the instance traffic is you might want to put it on a separate network for speed reasons Like if you have two network cards on the nodes why not use the second one for that or you could Have different instances Indifferent bridges for example rather than creating only one exam bridge you create five of them Connected to different route different for rolling policies instances that have not to talk with each other and then you connect them This is an example of it like green instances all talk to each other Magenta instances talk to each other in the blue instances on a network by itself and doesn't talk to anybody and Here the discs are connected to a special Interface a special switch which is just for replication which increases your performance during replication time And then you have tagging you can Tag instances or nodes with labels and then look for them If you need to make them special for any reason all the instances belonging to one customer or something you tag them So this is basically what we have till now then That's what we're working on for the next version of guanetti Which we don't know yet when it will be released, but we're aiming at sometimes in the second half of this year First of all what we're doing is a job queue So right now you have to create instances or do things One after the other and people of course want to create big number of instances So just script it and run it serially We want them to be able to say submit a job and then query the job status later And we want to be able to run these jobs in parallel also right now as every project starts We have one big lock and only one guanetti command can run at a time We call it the big guanetti lock because of historical reasons of course And then we want to have granular locking and for example If you want to create instances on two different nodes or set of nodes There should be no reason why this wouldn't work. So we're working on that Another thing we're doing is a remote cluster API. So right now you have to SSH to the cluster and Actually type your commands, but what if you have a web interface to control the cluster or something like that? We don't provide that now, but some people might want to build one So we want to build an API that allows you to control the cluster and maybe in the future even single Customers to control single sets of machines, but that's not in the first 1.3 release as a release target at least Then if someone has patches, of course that might happen Another thing that was asked is file-based storage What has what guanetti does now is it requires you to have an LVM disk and Allocates the instance disks on LVM So this works pretty well for the RBD, but if you actually do have a son it was reported to us that It becomes tricky because you would need either to have a cluster file system Anyway, it becomes more tricky than just storing the instances on files and Letting the son deal about everything else So what we are going to do is allow you to say put them in this directory and then you will manage your son Configure it and take care that it's visible from the other device for failover We want but at least it will allow you to use granted together with that then We'll have a possibility to Customize your instance a bit more as we said now you can install different operating systems But instances are kind of limited you have one disk One swap one network interface. That's all you can hack in the genetic config file manually to have more people have successfully done that because they needed it, but it's not quite intuitive and Basically, you have all the support inside genetic but not in the interface So we're planning to polish the interface and letting you having special instances instances with two network config connected to two bridges if you want to do like VPN And central points or things like that on genetic you'll be able to do that after This change Then we want of course to have live failover Now we don't have it failover because the RBD 07 which we were using before works in master slave slave master mode So you couldn't fail over the instance without tearing down the dRB device and re enabling with a master change to the other node The RBD8 can work in master master node But we don't want to do that because people will log into a node and just mount the disk for debugging While the instance is running on the other node They don't have cluster file system and everything will break So we want to keep with master slave nodes. So the disk on the slave node is just read only But what we want to do is to add the possibility to change it at runtime. So Well, when you will failover live lay an instance We will change this because we don't need to tear off the year the whole dRB disk stack anymore And so we can do live failover in the next version and then we want multiple coexisting hypervisors Right now only Xen in the latest versions. You can have Xen or Xen HVM so running windows or other operating systems all together and not just the versions of Linux with Paravir tops or anyway parallelized So with multiple coexisting hypervisors, you can run those together but you could also be able to run KVM or Virtual box if we will have a module for that on the same box or on the same cluster Now we have a module for the hypervisors. We are only supporting these two and we don't allow you to Basically Use more than one at the same time. So one cluster is tied with hypervisor We want to implement more and we want to make them coexist So also instances can have different parameters depending on which hypervisors and things like that And then what we want is a multi cluster management, but that's later after 1.3 We want to use the remote cluster API to have a multi cluster manager in order to decide which cluster to allocate your instances on In a central point external to Gennady, but that's also something we plan for a lot later And we don't know when it will be released And that's basically it. Thank you very much for your attention and if you have questions we have Two minutes now or you can follow me outside for Other follow-ups or just send a mail. We have a mailing list I didn't publish the website on this slide. Sorry, but it's code.google.com Slash P slash Gennady or just look for Gennady on Google and join the mailing list talk with the development team Send issues try it and let us know what you think about it. Thank you very much Do you have any questions? Yes Yes, you have to do it manually you could install some monitoring system that fails over for you But right now we have no like integration with heartbeat and checking the node uptime and so on what we do is basically If some for node is down and instances are down someone gets page and Fails over the machines, but at least you don't lose anything It's better than a failover like failing for a physical hardware, but it's not Automatized for now You could plug it to some automatic system that just runs the command Like plug it to a heartbeat or something Thanks Yes Yes, the cluster knows all nodes know about the whole status of the cluster So if you lose the cluster master, you can fail over that one To another node and you will have all the status Thanks other follow-ups outside


Not until the darkest days of the Siege of Terra did a planetary invasion of a sole geographic region rival the force that plummeted towards Istvan 5. Its passage tore the atmosphere of the planet like a skin. Over a dozen battleships and heavy cruisers spoke with the fury of drop pods and gunships, and as landing craft powered downwards, the batteries of the fleet spoke. No sustained orbital bombardment, this, the concept having already been dismissed by Ferris Manus as unnecessary and wasteful of time. This punishing fuselage was designed to tear through as much of the traitors as it could in the time it took for a drop pod to fall from orbit to ground. The Urgal Depression became a sea of light and fire, as the screaming air spoke of a lance beam scouring the ash wastes. The fortresses of the traitors were partially void shielded, barely enduring the sky-spoken cannonade. But beyond them, mobile orbital defense batteries, those that had previously thrown an impotent bombardment of their own at the arriving fleet were annihilated, their macro munitions torching off and sending mushroom clouds pluming into the tortured air. The orbital strike's true purpose succeeded impeccably. It covered the planetfall of the Loyalists. Occupied in seeking cover from the punishment of the retribution fleet, the traitors were capable of mounting nothing in the way of air defenses as thousands of drop pods smashed into the black sands of Istvan 5. In the center of this steel rain, cold fury marking his every movement, was Ferris Manus himself. Apocryphally, his were the first Loyalist boots on the black volcanic soil, thundering out of an iron hands drop pod with the rage of an Imperium spurned.  Such was the deluge of Loyalist steel plunging planetside that their batteries could not possibly miss, but for each drop pod plucked in fire from the skies a dozen more threw up impact craters in the Ergal depression. The killing began immediately. There was no need for anything even approaching Parley. Loyalist Astartes came out of their drop pods firing. Traitor Astartes rose from their trenches, firing. The precision of the Gorgon's attack vectors had placed the bulk of the forward assault right at the edge of the fortress's shield overlays. It was little in the way of No Man's land for Loyalists to cover before they were blade to blade with their former kin. The battle began as it would continue, as savage and pitiless as anything the Great Crusade had ever accomplished. An Astartes is a weapon built for a lentless conquest and extermination. Whereas for 200 years they had been committing unspeakable atrocities on Xenos empires and deviant human polities, the raw, genhanced power was now, as it had been on East Van 3, pitted against each other. Bolters fired at point blank range shattered Ceramite armor. Lady I plunged into bodies with a force that would have turned an unaugmented human into pulp. Chain swords tore at limbs and armor joints, teeth splintering and flying like shrapnel. Esoteric weaponry reduced transhuman behemoths to ash and slag. Hundreds died in minutes, the hammer of the Loyalists upon the anvil of the traitors, but the shock assault had achieved what it had intended to. Behind the lines drawn up more and more drop pods arrived, heralding the arrival of the bulk of the Salamanders and Ravenguard legions, with the bulkier fliers bearing heavier armor and artillery pieces right behind them. Full engagement took less than an hour. The Loyalists were advancing on the fortification walls, weapons fire pouring down from battlements, dugouts, pill boxes and gun nests. Super heavy tanks dueled or devastated infantry formations, and on the traitor side the legions herded forward Imperial Army regiments loyal to the war master to serve as human shields, to almost negligible effect. On the left flank of the battle the Salamanders were forced into trench by trench combat with the 14th Legion Deathguard. The combat zone was turned into a sea of flames, the fire of the Salamanders weaponry being answered in kind by the alchemic conflagrations of the Sons of Mortarian. Upon the right flank Corvus Corax and his Ravenguard dueled with companies of the Sons of Horus. The area was a maze of rock scree ravines and maces, where the crumbling faces of the valley blended with the sands of the Depression itself. The 16th Legion had turned it into a killing field packed with razor wire and hastily assembled ambush dugouts, turning the battle with their cousins into a blizzard of hit-and-run attacks and counter maneuvers, almost anarchic in its fury and rapidity. Finally, the center was held by the rage of Ferris Manus and his Avernii clan, the Terminator armored elite of his 10th Legion. Over 1,000 of the clan were present from the outset, as well as six whole maniples of the Iron Hand's battle automata. The Primarch was a hurricane of violence. Traitors not crushed by his hammer were torn apart bodily by his liquid metal hands. The Emperor's children and Sons of Horus that faced him could only mount momentary resistance before perishing. The Gorgon simply swept them aside. The embodiment of the storm his 10th Legion had once been named for. It was, by what records could be obtained from Iron Hand survivors, upon this front line that Ferris Manus became the first loyalist in the invasion to witness the beginnings of the horrors wrought upon the legions that allied themselves to Horus. Amongst the Emperor's children ranks, it was suddenly palpable that significant changes had been wrought upon the sons of the Phoenician. Many of the Legion's elite now bedecked their armor with what appeared to be flayed human skin. Their once perfect Ceramite defaced with strange symbols or riotous colors not part of their typical panoply. Stranger and more horrid still were stardes whose unhelmed visages appeared swollen, augmented beyond anything remotely human. These twisted individuals howled in registers, impossible even for an astartes, and did so in apparent ecstasy even as they were shredded by bolt of fire or hacked apart by the weapons of the Iron Hands. Many of them bore esoteric sonic weaponry capable of penetrating even the tactical dreadnought armor worn by the Avernii. Manus crushed them as he did all others that stood before him. Even these debased once loyal unable to stay his wrath. The toll being reaped amongst the Iron Hands was becoming more apparent now as the sons of the Gorgon attempted to keep up with their nearly rage-blind Primarch. That being said, the lives that the Clan had sold were being well spent. The massive lie in the Iron Hands had pushed forward was being followed through by the Legion's formidable armor corps, and its mobile artillery was picking up their affrain previously handled by orbital weaponry. Legion whirlwinds and basilisks poured rocket and shell fire onto traitor lines, denying the astartes manning the battlements easy return fire as they were relentlessly pounded from afar. Super heavy tanks and support vehicles poured fire on the massive looming figures lurking behind the lines. The Imperator-class Titan, D.S. Irei, one of the most powerful god engines of the Legio Mortis. The Titan had reaped a heavy toll upon the Loyalists during the opening hour of the battle. Its void shield impregnable, its weapons capable of reaching even the armor drop zones behind Loyalist lines. Now, finally, the volume of fire it was taking was increasing rapidly, wavering its precious void bubble and claiming the life of one of its warhound Titan escorts. Such was the degree of punishment Loyalist armor placed upon it, but D.S. Irei was simply forced to retreat into the alien fortifications, cycling its void generators to permit them a sole recharge. The momentum the Loyalist forces had accrued demanded a response, coming in the form of a traitor counter assault led, finally, by the treacherous Primarchs of Horus's legions. At the fore, a cyclone of slaughter, Angron, twelfth Primarch, world eaters. Whereas before the Ironhands line had been resolutely implacable, the Red Angel hit it like a meteor sundering a planet. The wedge-driven forcing open a gulf between the Tenth Legion and the Ravenguard.  Elsewhere, his brother Mortarion's arrival was heralded by the blanketing of salamander forward elements with a poisonous fog of stunning lethality, a noxious miasma that exploited even the tiniest crack in a legionary's power armor to scald the body within into a dissolving slurry. Only by the efforts of the 18th Legion's fire-drake terminators was the Deathlord's fury stalled, but at a terrible cost to their numbers. First Captain Abaddon of the Sons of Horus likewise now appeared, at the vanguard of the just-air and first company elites, plunging into the Loyalists with the savagery of the Cthonian mine gangs. The only response currently possible for the Loyalists was an increase in attack runs made by their Legion Flyers, freed now from the role of orbital transporters. These sallies were often nearly suicidal, plunging in under traitor air defense flak to target their super-heavy armor or heavy weaponry squads, even be it at the cost of their machines and the lives of the pilots. By 2.2 Terran hours sidereal time, the battle had devolved into a horrendously bloody stalemate. With the entirety of Astarty's forces committed, the Loyalists had now managed to land the bulk of their Imperial Army auxiliaries, although these troops were encountering severe difficulties in bringing what they could to bear upon the foe. The black sands before the drop site rechoked mass of power-armored corpses and the wreckage of Legion vehicles, constraining the movement of Imperial Armor across the already tight Orgal depression. Worth still for the unaugmented auxilia, terror squads from the traitor legions had infiltrated their landing zones, sowing discord and horror with near impunity. Whatever the experience and veterancy of the troops Manus had seconded to the execution fleet, they were mere baseline humans pitted against traitor Astarty's genetic ferocity. Wherever these terror troops encountered Imperial Army regiments, little existed but wanton butchery. Just over two hours beyond planetfall, an Istvan 5 had already seen the deaths of tens of thousands of Astartys, on both sides of the conflict, and yet tens of thousands more still fought on at a pace that had not slowed one Iota. Where ammunition had run dry and supplies rendered impossible, both sides fought with the weapons of the dead, scavenging what they could until they were resorting to ceramite clad fists and basic enhanced physiology. The last of the Loyalist reinforcements from Manus' fleet were finally permitted to land, a massive red bark bearing the cog symbols of the Mechanicum of Mars, and the flaming sword of the Firebrands. A battle-manable of the Ligio Artaris. In response, Imperial Ospexes detected massive heat blooms that had previously slumbered. Knowing their former kin would almost certainly be part of the Emperor's retribution, the Titans of the Ligio Mortis had waited until this moment to rise. It was immediately apparent to the Firebrands, and their attached Mechanicum Tagmata, that they were woefully outclassed by the dead's heads. The Loyalists possessed only two Reavers and three Warhounds, ranged against a Warlord, Twin Reavers, and a Nightcaught mainline Titans, with two Warhound Scouts. DS E. Ray was still cycling her reactors and shield generators, but even without the Imperator, Ataris was on the back foot, as soon as its Titans disembarked. Odds of survival were calculated at a mere 13% by Mechanicum adepts. These were transmitted to Ferris Manus who ordered the attack, regardless. Mortis was to be brought to Imperial Justice just as the traitor legions were, and Ataris would merely have to inflict what they could upon the dead's heads. This they needed little in the way of encouragement to do. The Firebrands had long borne a furious hatred for the Ligio Mortis, and plunged towards the traitor Titans, heedless of the cost to their own engines. The Suicidal Charge was catatlysmic to behold, and achieved in its heedlessness all that Ataris could have hoped for. The lead Reaver, Red Naga, closed the distance between herself and Mortis' Warlord Breath of Thunder, the Ataris engine's power fist shattering the Warlord's head cockpit and killing its entire crew. Punitive Fire from its Warder engines breached the Naga's reactor core, however, with the ensuing explosion claiming dozens of traitor tanks in recompense for its demise. Approximately 2.8 Terran hours. The sheer weight of death and destruction was finally having a toll upon the pace of the battle. As much as 40% of the dispositions of all combatants three hours ago had now perished or been destroyed. Loyalist heavy artillery and armor had kept up their punishing salvos on the traitor lines unremittingly, forcing the withdrawal of Mortarian and Angron. But the sheer tempo of the assault had drained the ammunition and weapon reserves of the Ravenguard and the Salamanders, enforcing a series of regrouping actions as Vulcan and Korak sought to take advantage of the relative lull in fighting. Ferris Manus, however, continued his wrathful advance, unwilling to surrender an inch of ground or a scintilla of pace to his enemies. Tactics were at this point meaningless. Orders of battle dissolved into Ersatz's reactions to impossible situational exigencies. How, though, could anything resembling strategy survive contact with the reality of what had been unleashed? Powers that had a mere centuries earlier been enough to conquer half a galaxy had been unleashed over a couple of hundred square kilometers in the space of three Terran hours. They had achieved nothing except slaughter undreamed and destruction untrammeled. Ground had been taken but the traitor's walls still held. Loyalists had been bloodied but were still unbowed. Titans had clashed but neither side had achieved anything of note. The demigod Primarchs had torn bloody chunks out of their brother's legions but had not clashed themselves. Horus and Fulgrim, indeed, had seemingly not even deigned to enter the Frey personally. Their absences as inexplicable as they were notable. The future of the battle was perched upon a precipice. Not had been truly won and neither side could claim to have the upper hand just yet. But as Ferris Manus and his iron hands continued inflicting the chill ferocity of Medusa upon their enemies, a cry went up amongst the Avernii clan. Dropships. Reinforcements. Salvation. Justice. Retribution. All in the form of the four promised legions. They had come at 3.4 hours post initial planet fall. Thousands more orbital landards streamed from the heavens towards the Orgal depression. Touching down at the initial drop site of the Retributory Vanguard and in several key locations surrounding the valley. The Alpha Legion streamed towards the left flank, while the word bearers took up the right. Divisions from the Night Lords reinforced both their cousins, as well as seeding squads and drop pods amongst forward iron hands positions. The Iron Warriors landed massive orbital craft amongst the rearmost echelons, per their standard tactics. The custom craft of the 4th Legion entrenching themselves and acting as loci for prefabricated fortifications and artillery emplacements. It was as if a mirror to the traitors own castellum was suddenly appearing, a citadel in opposition to Horace's own, but built with frightening speed. At the mouth of the valley, its armored cheats and walls were expanding ever outwards. More and more reinforcements poured from orbit, tertiary drop sites being created with every minute as four whole legions now joined the battle. Korax and Vulcan immediately hailed Ferris Manus on the Vox networks, suggesting immediate retreat from their respective front lines. Never the largest of the legions, the 18th and the 19th had suffered vicious casualties in the fighting, the Ravenguard at the hands of the world eaters and sons of Horace, the Salamanders to the weaponry of the Deathguard. The Gorgon would hear naught of the concept. Unthinkable to him was retreat, not now, not with renewed strength that was to be had from recently arrived allies, but the accounts of survivors. It is at this juncture that Fulgrim emerged before the lines of the Iron Hands, the Phoenician taunting the Lord of the 10th Legion, bating him attack and reap in vengeance what his honor demanded. The response was immediate. Ferris Manus broke from his own ranks, oblivious to all but the sight of his erstwhile brother, fury blind, blood mad, berserk from the pain of betrayal. The laughter of Fulgrim was said to have rung around the Urgal depression with the clarity impossible. The primarch of the third, artfully trading blows with the meteoric attacks of the Gorgon in a duel unlike anything the galaxy had ever beheld. As the first elements of the retreating Ravenguard and Salamanders approached the new drop sites of the reinforcing legions, hails were sounded over interleagent vox channels. Thanks were given for sorely needed reprieve. They were calls of welcome, of gratitude, of fraternal relief in a world turned upside down. They were ignorant. They did not know. They could not possibly have known. A flare brighter than the sun in the ash choked skies. Shot skyward from the warmasters lines. As red as blood spilled. And then treachery. It was the word bearers that opened fire first. The first traitors, although such a thing would only be revealed in the years of pain and death that were to follow. The legion of Lorgar had waited for this moment with a religious fervor. Their act of purest malignant betrayal, one of, they believed, cosmic significance. Worship upon the altar of duplicity and deceit. The Ravenguard who had approached their lines had barely a chance to register what was occurring before being scythed down by hundreds of legionary bolters. The remaining traitors did not waste any time. The guns of the iron warriors opened up, launching an artillery barrage that blanketed the entire Urgal depression in death. The Alpha Legion and the Night Lords unsheathed daggers and plunged them into loyalist necks. The bloodshed was immediate. It was total. It was abominable. The loyalists took fire at point blank range. There was no cover. There was no defense. None was believed needed. A startys that had survived three hours of the most hellish combat in the history of the Imperium simply dropped dead. Broken in body and in mind. Flesh detonated by the bolts of the traitor legions. The gunships of the Night Lords sowed phosphax munitions amongst those that had survived the initial salvos.  The crawling death. In the rearward lines, the Alpha Legion struck apothecarian stations in overwhelming force, slaughtering the wounded and those that sought to tend them, before turning its coils towards the Imperial Army. Never being one to strike unless from a position of overwhelming superiority, the harrowing of the 20th Legion was slaughter at its most base. Nothing able to halt the Hydra's rampage. The world collapsed. All was betrayal. All was massacre. Apocalyptic, vicious, sadistic. Impossible, but so terribly real. Amidst this, Ferris Manus and Fulgrim yet fought. The Phoenician's laughter had risen to a hysterical torrent at the moment of treason. His taunts to his brother now carrying the poison of what was happening all across the battlefield. It is impossible to know if this could have deepened Manus's rage yet further. The propensity for fury had been with him all his life. The chilling collar present in his soul, only tempered by his equally frosty reason. He was at this point seemingly drowning in anger. The mental conditioning of his creation, broken utterly, are the pain of losing his most treasured brother to apparent madness. The blows traded by the Primarchs were thunderclaps, producing sonic waves that broke earth and shattered even Astarty's eardrums. Each scored the body of the other with a dozen wounds that would have killed any being lesser than they. Fulgrim with his sword, Fireblade, Manus with the hammer, Forge Breaker. The brothers were as unalike as any two within the Emperor's progeny, yet they had forged a kinship closer than almost any other pairing. Neither would yield, neither could yield. The world around them burned and broke. The iron hands desperately fighting to reach their Primarch were overwhelmed. Broken into tiny pockets of resistance by the Emperor's children, the Sons of Horus and the World Eaters freed as these legions now were by the arrival of the newly revealed traitor armies. Yet on the Primarchs fought, and from this point history is a blur of myth and legend. None of Manus's Avernii escorts survived to tell of what had occurred. Blasphemously, the only sources that we have upon the incident were torn from the minds of captured traitors, or whispered throughout the years in the way that all such tales of pivotal events inevitably evolve. The most common telling states that the Gorgon finally landed a near perfect blow upon his brother, casting the Fiannician to the Black Sands and tossing Fireblade from his grip. Laid low, Fulgrim was no longer possessed of laughter, but now with a toxic viciousness. He unsheathed a blade he had claimed once long ago from the Xenos Lair. Loyalist psychic sensitives reported towards the end of a battle a spike in etheric energy somewhere in the Orgolic Depression. Scholars have posited this weapon. This event was its source, although they are only capable of making this assertion, given all that had transpired. With the cursed sword, Fulgrim, third son of the Emperor, beheaded Ferris Manus, claiming the life of his brother. The first amongst Kinslayers, and far from the last, darker powers awakened now as the symphony of violence reached greater and greater heights. Amongst the word-bearers, the so-called Galvorback, the Blessed Sons, began to reveal themselves as blood sprayed and bodies were torn. These were the twins sold, the possessed, first of the traitor legions to sell their selves to the never-born creatures of the Dark Gods in exchange for worldly power. Revealed to their loyalist foes at last, the Galvorback were even more monstrous than the debased Emperor's children. Their bodies mutating visibly into bestial, warped war forms. Hidious jaws and protruding claws formed from the melding of ceramite and flesh through pure demonic power. Observing his sons, observing the slaughter, was Lorgar himself. The thrice-cursed arch-traitor was recorded at having wept at the bloodshed, whispering to himself that these were once his kin. If this was grief, if this was guilt, if this was religious rapture, it is impossible to know for the mind of so debased a being as the Yurizen is unknowable to any. His reverie was however broken, but the darkening of the skies with pinions of sable, arriving like a lightning bolt, a blur of gray black armor slammed into the 17th Primarch faster than even Astartes physiology could track. Korax, half mad with sorrow and fury, tore through the Galvorback, slaughtering the demon things and forcing Lorgar to act. Despite the supposed tumult in the Primarch's soul, despite his distinct lack of physical or combat prowess, Lorgar Aurelian hurled himself at Korax, swinging his crosious mace.  His overpowering of Lorgar was a relatively simple affair for the Raven Lord. The Yurizen was no warrior, and Korax was one of the Imperium's finest. One of the Primarch's great talons speared into the 17th Sun's belly, just as its twin was about to reach for the beheading, Lorgar's salvation arrived from a wholly unexpected turn. Conrad Curse, the Night Haunter, Grimm Primarch of the Eighth Legion Night Lords, had intervened, had deigned to intervene. The Raven Lord, unwilling to fight a second of his brothers, having just overcome a first, retreated, his true objective having ever been to buy time for the evacuation of his sons. Curse is reported to have expressed his extreme disgust for Lorgar, before returning to reveling in slaughter. Ferris Manus lay dead. His head torn from his demigod body. Korax's face would slip from the pages of history for some time, following his confrontation with Lorgar. The remaining loyalist Primarch, Vulcan, was at this point last seen by his salamanders holding a desperate rear guard alone against an impossible number of traitors, until a volley of atomic fire consumed him. It is difficult to tell when the battle ended there, in the Urgal Depression. For once the revealed traitors had opened fire, it no longer deserved the name. It was now, as it would ever be known, the Dropsite Massacre. An act of breathtaking brutality unrivaled in its duplicity and inhumanity in the annals of history. Pockets of resistance were surrounded and eviscerated, winnowed, bled. Survivors fleeing a fighting retreat for what drop ships remained operational, reaped what tolls they could from their betrayers, before gunning engines for orbit. But this was no recompense. Nothing could come close. By all accounts, the Alpha Legion were the first to withdraw from active combat operations. Removing themselves from the valley so as not to spend any more lives in an already one battle. And to avoid frothing bans of world eaters that were as much a danger to friend as they were to foe. When the guns of the Iron Warriors stopped firing, the only truly active Astartes left belonged to the most sadistic and twisted of the Emperor's children, sons of Horus, and Night Lords, gleefully taking bloody trophies from the shattered remains of the fallen loyal. In orbit, the war lasted for several more hours than it did on the ground. The ships of the Loyalists had been fully shielded at the moment of the trail, and were thus far less easy prey for the traitors than the Astartes' planet side. Many remained stubbornly in geosynchronicity with the drop site, receiving in their holds any and all gunships and landers that made it to safety, forced to retreat only when they had sustained damage that threatened their warp engines, or when the outer system all speck sweeps detected the immaterial wakes of the approaching fleets of the original traitor legions. Of these ships and their sundry fates, history had yet to dictate. Many made it out system, but with conveyance bays painfully, dreadfully empty. The drop site massacre thus concluded, not with a howl of victory, but with the type of silence that can only fall when those that had filled it have simply perished. Punctuated only by the screaming of baseline humans being flayed alive by the Night Lords, the odd explosion of burning ammunition reserves, the rage-filled impotency of mortally wounded with still alive Loyalists. The Urgal Depression was a choked mass of wreckage and bodies. Initial death toll estimates, laterally revised by historians, placed the tally at nearly 200,000 Astartes alone, but numbers of unaugmented troops genuinely unaccountable. One Primarch lay dead, two more were missing and presumed so. While the losses of the Loyalist legions were not total, their efficacy as combat operative legions was essentially ended. None had gathered in their totality. Such a thing was impossible at this point in the Great Crusade, given the deployment of the legion as Astartes to all corners of the galaxy. But the iron hands, the salamanders and the Ravenguard, had been mortally wounded by the massacre. Bands of survivors, having fled out system or into the ravines and canyons surrounding the Urgal Depression, would of course go on to lead storied histories throughout the coming years. Many would reap fearsome tallies from the traitors in revenge, but it is not for nothing that these warbands are collectively known as the Shattered Legions. The iron hands in particular fared especially poorly. The first Astartes legion in recorded history to witness and suffer the death of their Primarch, they were, bluntly, mentally incapable of processing the trauma this loss inflicted. This would express itself in many forms, suicidal rage in some, complete detachment in others, rampant, endless nihilism in yet more. Shattered is the only word applicable to their Astartes-meaned psychology. They were weapons in the forms of men, retaining some of their programming, but unable to reconcile the reality they now occupied. The roles these loyalist legions would later play in the heresy. All of their tales will be discussed in later records. But it is without a doubt that their part in the greater conflict ended before it had even begun. The Dropsite Massacre is one of the darkest acts of the traitor Horace Lupacal, an enduring cancer upon the history of the galaxy. Yet it was, in so many ways, a mere prelude of what was to come. The great crusade was finished. Now had begun the age of darkness, Ave Imperator, Gloria in Excelsis Terra. Very kind donations and support from my Patreon subscribers. If you'd like to help support the channel, head on over to Patreon.com slash Oculus Imperial. If you'd like to receive more updates about the channel and any future videos, you can contact me or follow me on Twitter at Oculus Imperial. Otherwise, please like, subscribe, comment, let me know your feedback, and as ever, thank you very much for watching.


Great, so I Was in a talk earlier today in the go dev room from Andre where he was talking about advanced techniques of debugging go and he asked the audience, how do you debug and everybody shouted out print line and It's quite clear that debugging, you know kind of lost its way over time But at the same time it's not because we've got this great dev room and there's been amazing talks from people talking about the debugging tools that are out there and I'm gonna go through this is mainly a tool talk. I'll do some demos, but the tools are still in their infancy I'd say and Debugging isn't gonna be your first problem when you get into kubernetes. Your first problem is deployment You want to focus on your CI CD process ideally with some GitOps techniques and then once you have a good flow of getting code into production you then need monitoring and There is a whole dev room on monitoring and tracing open tracing is a fantastic technique for finding out what's happening in your application Once you got all that going then on your day three if you're that fast you can start debugging and Just so we're all on the same page. I want to just do some basic kubernetes talk Hopefully, it's the end of the conference and you've been able to pick up at least enough to where you don't need this But just so we're all on the same page while I keep going through The container is the core. Well the pod itself I guess I think of as the core component in kubernetes and in a pod you have a container running You can have multiple containers running Ideally, it's preferably just one container, but there's no restrictions. You can have containers You can have volumes with data in them or not as you like. There's networking capabilities that kubernetes provides you and This is kind of your your basic concept of what you want to do with kubernetes is run your docker image Run your application in a docker image doesn't have to be docker either, but primarily it is and so those pods then run inside a node and the nodes would be Typically an actual machine and you can have multiple nodes and distributed across all sorts of Areas and each pod can then be in different nodes and distribute out in all sorts of different ways kubernetes is there to provide The orchestration you tell it what you want you give it the configuration This is how I want my environment to be like and kubernetes does its best effort to give you that So if a pod crashes if your application has a crash and it shuts down Kubernetes will do its best effort to spin that up if the node gets lost Yeah, if you're in the cloud and they just take that VM away from you kubernetes will do its best effort to shift those pods That were there to a new node or other nodes you do have so again. That's it's a best effort technique Based on what you want so Hopefully that's kubernetes in a minute and now into debugging and Everything that I'm going to show you today. I do not recommend to do in production all these tools Have huge security risks let alone just taking your whole system down So I gave a talk yesterday about ephemeral environments. I'd love to plug that There's a lot of tools on how you can get set up in a non-production environment and I mentioned Andre's talk earlier today, and there was a great talk from Lucas about One of the tools I'll talk about later that you can check out and Yeah, I mentioned that like There's this one technique of just logging out what you want to find in your application But the debugging has so much more power that when you can utilize it you can really get the best development process available So with the tools I want to start with two that I particularly find more like they're my favorites They're the first ones that I got these to work and when I was able to set breakpoints in my IDE on something running in kubernetes I thought this is great stuff, and I did it with the use of one tool was ksync and the the concept is that you have your file that runs in your pod and you know it's just an html file and you build that docker image and You put that file in it then you run your pod and that file sitting there in the clouds or in your cluster and You have that file in that docker image and you want to change that without having to recompile a whole docker image and everything And so ksync will sync a directory that's on your local machine with the Directory that's in the pod and so then on your file and your local machine You can change the file and it'll get synced into that container your docker won't restart It just updates the file and you now that have the change that you want it Available to you in your running system Once you have this going you could then connect a tool called squash Which is basically a wrapper around debugging for kubernetes It'll spin up another container in its own namespace which opens up the port connections And so then your IDE locally will have that port connection configured So then when you're running the application and hit the point where you would have a breakpoint It would then trigger in your IDE and stop the pause the application while you develop squash is limited to delve for go and Java and GDP They had Plans to support Python and Node.js this year this past year and didn't reach those goals So of course they're open for contribute contributors to help get there But yeah, so as I said this was when I got these two tools working and they each do their own thing So they don't have that extra complexity when something goes wrong if something goes wrong with one of them You know, this is where I need to focus my attention You don't have one tool trying to do 70 things and you're like wait. What is it this? No, it's there. Yeah, so So that's again the benefit of having these two separations There's another tool called telepresence which when this works, this is even better this when this works This is probably the best The concept is that you have traffic coming into your cluster into your pods You know communicating with other pods in your network and coming back out and you know You want to debug something in that one of these pods? so with telepresence you create this tunnel into your local machine and your local machine then has access to everything else that's in the cluster and so as I mentioned about not doing this in production if you do this in production and Traffic comes in here and then goes to your local machine and you're not running the app on your local machine There there's that's the end of the traffic, right? Or if you have a break point sitting on your IDE and somebody hits that break point Yeah, that's the end of it. Like nobody's gonna get back the results that they're waiting for Telepresence because it creates this interesting networking at least when we develop it or we use it with our go applications We have to create a new VPN for this network and You can't use another VPN while you have telepresence running So if you have as we have VPN for all our outside developers to be able to access the internal networking They're not able to use the VPN and develop with telepresence. It's not compatible for two VPNs at the same time at least with that technique of creating the network that uses Mac OS and Go and so there's the documentation on telepresence gets into its Limitations and what you can do and how you can do it with different IDs as well and integrate it with your ID directly So check it out if you can get it to work. You're gonna love it If you can't then try one of the other tools There's also an important aspect with telepresence is when you start the application locally, you know If you have the settings in your pod that say, you know, which ports something connects to like all the configuration that's Available for the Docker image that's running in the pod You need to know when you execute the command locally Yeah, if you need to know that this here is called, you know DB on port 306 when you execute locally You have to know that it's DB 306 and so forth Okay, so the next tool to talk about is scaffold which basically combines the two of case-sync and squash into one tool and It's available and IntelliJ and VS code as a plug-in and It works fantastic when you have just this smaller like the one application with less dependency structure and It'll manage the syncing of the file into the pod If it it'll know that if you have a compiled language that it'll need to spin up the new docker image It'll spin that up for you put it in the place that you tell it to and pull it down and start the pod again and have everything Running locally in your IDE and you get a lot of fantastic usability when it works and It was interesting because I have this demo that I'm going to go through with this application And I was trying to get scaffold to work with it because I thought oh this is you know a great tool This is how you should use it, but I couldn't actually get it to work on something that had this Pre-built helm charts that are I already had my application. It was already there It you then have to do everything manually and once you start doing things manually some things don't work And then you got to figure out how I did so I need to get there Maybe I didn't spend enough time with it all these tools there as I said they're in their infancy They're gonna take some time to get rolling. Yeah, so don't expect to just flip it on and ah, this is wonderful so the the last one I wanted to talk about was the new IDE from Eclipse that actually runs in the Kubernetes cluster so all that networking and mismatch and trying to figure out how to sync files you don't have to do because your ID is actually running there with the pods Started up with the code that you are developing right there. So the ID is a browser based interface, but you just interact directly with it and get all the all the magic there Lucas had given it gave a great talk on this a little while ago and It helped answer why when I tried to set it up two days ago. I still was like, hmm, how's this gonna work? What's going on here? It's brand new. I think it's just been out a few weeks. So it'll be good pretty soon, I think based on what I'm hearing and the the one limitation now is that if you're doing it in like your local machine if you're spinning up a mini cube as an example and then Installing it that install process takes a little bit of time as well And so it's not a quick up and running experience at least if you're doing it locally Yeah, if you have a cluster in the cloud where it's running You don't need that extra time of getting up and running But if you have that cluster in the cloud other developers can then interact with the same code base I believe so demo time I Learned a lesson and recorded my demos But what I haven't done was figured out how to actually Give a demo that based on a recording. So we'll see how this goes and yeah, it's not going good yet because Share the screen that should Okay, so here I'm gonna be demoing k-sync and I start out by running mini cube on the machine and I'm using a slightly older version of Kubernetes and not very older Google only released it a like a few weeks ago. So Made it available in GKE I mean a few weeks ago, but that's because the The application that I'm going to be installing there doesn't have the new API versions configured So the app only works with one dot one five and before So now everything's up. We have our our cluster. We have a default namespace and everything the kube system namespace and I Got a helm installed locally, but not in the help in mini cubes or install that I Have armador installed which again was in the talk I gave yesterday on how that works So now I'm going to need to pull some code down and there's this example voting app that I went into that has these different components and The idea is that I'm gonna clone this repo. I'm gonna be a developer of the voting app I'm gonna be you know this front-end developer working on a Python application and so I need this set of code here and So I clone that code copy it into another directory and then I Also want to have the helm configuration for that code and of course docker doesn't give that in their sample file, but Code fresh heads originally wrote it and then I added it into armador and the documentation there so We can easily pull down the The helm chart with the armador configuration and merge that into the voting code So as a developer of the voting app, this is kind of the this would be the Code I would have locally With you know my app.pyton my docker file my helm configuration In the helm configuration I have the armador file with the dependencies that I need and The docker file that adds this app and Here we can see that there's you know cats and dogs that are available for voting and In the example of using ksync I'm gonna want to switch from being able to vote for cats because I'm just not a big fan of cats But I love bunnies, so I'm gonna work on making that happen So first we're gonna install the whole environment Of all the components we need so there's the five charts to be installed one of them is the voting app and The others are the Postgres the Redis the results app and the worker those got installed Then we can see they're installed but not yet running So we'll wait for them to get running which 39 seconds 50 seconds whoo that went quite fast time's flying so everything should be running the Database is still initializing Postgres is still a bit slow, but we should be able to now load up the front end So we'll use mini cube to open the front end There's the cats and dogs We'll load up the results so we can see yeah, I voted for a dog and Here we go in and change cats to bunnies Maybe nope not yet first we want to set up ksync so It's already installed, but it's not initialized into my cluster so that runs and then I'd start watching and so ksync will now just watch what's happening in the cluster and Then I'll have to create an actual instance to I Don't know to be watched I guess and from that I'll need to know the oh Maybe I'm showing you something else first Yeah, so here you can see that there's in the coupe system So ksync has to have these extra privileges to which you again wouldn't want to include in your production environment so here it's just sitting there waiting and listening and It's gonna need the The app selector for the pods so the voting app has the the label that Will be needed during the create Yep, so first I show you some documentation on ksync show you how I Google to get documentation and Quickly read through it all. Yep. Yep. Yep. Yep. Yep. There we go get that selector and so We need to take a look What our labels are in our pod? There's a lot to read through here, but there's the voting app and there's the selector and Let's speed this up because we might be low on time Hopefully I can talk fast enough where I make a directory. So I have this theory where With ksync, you know, it syncs the folder from your local machine to the folder. That's in the docker image and and the docker image You know, we have hopefully just a small subset of what what you need to run Yeah, so in the actual code base you have your docker file you have your application you have all your configuration you have your ID settings and Hopefully your docker image is compiled and built without all that extra stuff and just the application you need And so when I do a ksync, I don't want to sync my directory of my code I want to sync just an empty directory and get all the stuff from the docker image first done to my machine and Then copy things from my local environment into that and this is maybe because we work primarily in go language is a compiled language and You need to compile it anyways. I'm not going to have it in my local in the directory of my code base So I'm gonna have to copy it over anyways And so here I'm going in and I'm showing you what's in the folder that syncs and then what's in the actual docker image there Now, of course, they built the docker image with the docker file My old ideal of you know separating out what's there is lost on this technique with this example But you got to do things right if you want to be right now baby steps There we go. Now I changed some bunnies and I saved the file again. It doesn't sync anything yet because I only saved it into that local directory but then I copy it into the ksync directory and We see that There it goes. It copied the file Reloaded the pod didn't actually stop, but there's our bunnies we can vote for bunnies and Well, we voted for cats that didn't work so I think that's the end of this demo and then Oops So let's try to fix the the bunnies on the back end and My theory is that it's a problem in the worker and so we have the the voting app in the front end in the worker that converts it into then show in the result app and The worker is a Java application. So in theory in this I could have used to squash. However I'm not a Java developer. So I ran into a whole bunch of problems getting that to work so in this demo, I'm going to use telepresence and Be able to spin up the worker On my local machine while it's connecting to the Redis and the Postgres and in the cloud So I start up IntelliJ go into the worker code And again, I took it from the example like I took their code You know, so this is me as a developer of the worker I want to debug the worker and I go into their source code. Yeah, there's just one file and It does the connection to the Redis and to the database and then it just waits for things to happen and In a moment, I'm going to set up the configuration to be able to debug Don't know what I'm doing there There we go set a break point and then Go into the menu and Okay, so debugging it's waiting for Redis right because I started the application But I don't have the connection to Redis. I'm just running an app on my machine so now I need to turn tell the presence on and What I need to do that or so I have this Redis It's actually interesting so I already have the telepresence command set up I'm going to swap out the deployment of the worker to be a tell it presents instance and so I'm going to do that. I have to get my password right and then telepresence starts up Then I was going to show you about how there's you know Ways of doing this with minicube. There's plugins for development with IntelliJ There's a whole lot of different options here and here you can see that I'm using the TCP connection with the VPN So that has the limitation that only tell one tell the presence and So I'm still waiting for Redis What actually happens with this instance is that the Redis DNS name doesn't Resolve so I have to change the Redis to be the port I think this is just a Setting on my local minicube thing, but the DB Resolves so there we go. We're connected. We got Redis and the DB running and Then we're going to set a breakpoint to see what happens when we vote and we're going to go and we're going to vote we're going to vote for a dog and It doesn't change because it hit the breakpoint here. Yep, there it goes the votes in the queue and to speed things up Yep, so we vote for B. We can see that there's It's actually coming in just as the letter a and not as bunnies or cats or anything. So It's not even a bug in the worker. It's a bug in the front end application. That's I didn't change what I needed changing Yeah, so that was a nice demo. I have five minutes left to try to do a demo of squash that It's not here. I Uploaded a zip file with the this code that I've got So VS code I've got just the simple application with a kubernetes configuration of just a service and a pod Running on port 880 the docker file is really straightforward. I uploaded the the default docker file into quay And the application is really I was looking at this I must have taken this from somewhere because it doesn't make any sense, but it gets to the point across and so if I first run it could fly and So and then I got a port forward so we can see that His ad is true. So I'm trying to add option one to option two So I'm trying to add two plus two and it equals zero. All right, so clearly there's a bug there and so now I need to first set up the casing because again go as a compiled language so I need to compile the Changes and have it with the debug available. And so let's do a case sync watch and a case sync Create Nope, that's not it So it's in that name call a calc. Yeah, this should be good. So now I've got this thing Yep, there we go running case sync get yeah, so now that's We're in the code so if I do make Case sync this is just going to build the application and save it into the casing folder which I think should have worked and The port forwarding still there and so in the code in here I can do I already installed the squash plug-in into VS code I Yeah, that's basically all you need to do with that and delve needs to be installed and then you start up squash and It squash is running in another namespace So I'll get namespace so there's the squash debugger runs there and As I started up it says hey, you know, I've got all these namespaces. Where do you want to debug something? I got it running an sq and here's the example app and which debugger do you want to use delve and So now it should just start up there we go and I can set breakpoint somewhere in the code and around here after this print calculating and Load the app It didn't work. What if I change? This and do another make that's probably why it's not actually updating the binary I think I'm a little low on time here to find out why this isn't working But again, I uploaded this code base and so based on part of what I said you might be able to piece this together on your own one last thing to show you is kubectl exact minus IT minus n sq spin H L s minus all so here we can see that the The application was updated Well, you know an hour ago, but no real time based on time zone The file got synced. It just didn't change Times up Thank you


My name is Maxine and I am a farmer in Makwini County, Kenya and climate change has been real for us because we've had long spells of drought within the county and in most parts of the country one mitigating factor we've had to figure out for our kettle is a source of water to drink. We have managed to do a borehole as you can see behind me at least to provide water for the cows and for the life within the farm to be more manageable.